FS#2698 - Many plugins cannot compile with new PluginConfigure.cmake requiring to run from a git repo

[dominig]

FS#2698 - Many plugins cannot compile with new PluginConfigure.cmake requiring to run from a git repo

The new version of PluginConfigure.cmake file written by John Gaugh and reused for several plugins (see list bellow) requires the build to be run fro ma git valid repo. When building packages in the distro system (my case Open Build System from OpenSUSE), the source used for the build is a copy and not a valid git repo, so the cmake fails miserably.

Having at least an option to deactivate that feature to allow and off git build is required.

Could we have a clean solution to by overcome that feature when not needded ?

Thanks.

Dominig

[jongough]

The design of the new process is to use git. To stay current it is required that a git file hierarchy be in place and that git is installed and used. Testplugin_pi is the repository used to build frontend2, ODraw just uses this. So this issue should have been raised on that git repository.

This is the first indication of anyone trying to build plugins for Plugin Manager without using git. For ease of use just clone the required plugin git repository and follow the instructions on how to build. The new process is already complex enough without having to cater for 'hack' builds. As far as I am aware git is supported on all platforms being used, so what is the real issue? There are far more onerous requirements to allow the build to occur, i.e. wxWidgets, etc..

If you want you could add a check for git being available, then change the bit looking for local git and add an if statement to provide default values instead of working them out.

[dominig]

Hello,

the plugins in OpenSUSE build system are also built from git but,
the build system makes a local cpio which is then automatically
copied on a compiling worker in the cloud.
There is no way that CMake configure can "see" the original git in
such automated build configuration.
I guess that any built done in distro build system should have
similar issue.
In that case the distro provides the repo and the keeping updated
process is done by the distro build Cloud system automatically.
Any idea how to also support in distro build system ?
Regards
Dominig

Le 27/08/2020 à 00:12, jongough a
  écrit :

  The design of the new process is to use git. To stay current it
    is required that a git file hierarchy be in place and that git
    is installed and used. Testplugin_pi is the repository used to
    build frontend2, ODraw just uses this. So this issue should have
    been raised on that git repository.
  This is the first indication of anyone trying to build plugins
    for Plugin Manager without using git. For ease of use just clone
    the required plugin git repository and follow the instructions
    on how to build. The new process is already complex enough
    without having to cater for 'hack' builds. As far as I am aware
    git is supported on all platforms being used, so what is the
    real issue? There are far more onerous requirements to allow the
    build to occur, i.e. wxWidgets, etc..
  —
    You are receiving this because you authored the thread.
    Reply to this email directly, view it on GitHub, or unsubscribe.
  [

{ "@context": "http://schema.org", "@type": "EmailMessage", "potentialAction": { "@type": "ViewAction", "target": "https://github.com/jongough/ocpn_draw_pi/issues/455#issuecomment-681150397", "url": "https://github.com/jongough/ocpn_draw_pi/issues/455#issuecomment-681150397", "name": "View Issue" }, "description": "View this Issue on GitHub", "publisher": { "@type": "Organization", "name": "GitHub", "url": "https://github.com" } } ]

-- 

Dominig ar Foll Senior Software Architect

[rgleason]

Dominig, The ci scripts do exactly the same thing and then build. Perhaps you can analyze what paths are being used by using a verbose mode?

the build system makes a local cpio which is then automatically
    copied on a compiling worker in the cloud.

[jongough]

All the current build processes use the full directory tree including the '.git' branch. The process uses the git command, when being built locally, to determine the git repository name, branch and tag to workout what type of build to do and where to put the results.

The new process works when using local developer builds on any supported platform and by using appveyor, circleci or travis for 'deployable' builds. For testing purposes the requisite build script can be used to do the build if needed for testing.

Why do you need to do a different process? If you want to use the cloud, then use one of the cloud build environments customised to your needs. If it requires changes to frontend2 to make it work submit these as a pull request to testplugin_pi and I will incorporate them if appropriate.

[dominig]

Jon,

I maintain OpenCPN for OpenSUSE. The build is done in the cloud as
for all OpenSUSE packages with the Open Build System or OBS.
That system allows to keep uptodate all the distro packages for the
supported multiple release of OpenSUSE and SUSE.
It triggers automatic rebuild as soon as one dependency package has
changes, republish automatically and push the new packages to all
OpenSUSE user via the standard update system (Yast, Zypper or
Package manager depending of user preferences).
Packages are pointing to the git repo or use release tar files, for
OpenCPN I only user git repo.

I do not have control of the internal of the build system which does
a cpio from the git repo and then "sub contract" the build to a
build worker server (~2000 server instances). If the resulting build
creates a binary which is different than the previous one, the new
package is published and push as an update.

That solution allows to guarantee that the entire system on the
final user is consistent (eg all required packages are compatible
between them).
My strategy has been from the beginning of OpenCPN on OpenSUSE to
download an install all the valid packages by default (manual
selection is still possible for debug or test), leaving the user to
activate the desired one.

That model of operation has allowed to keep OpenCPN running on
mutiple version of OpenSUSE (often old PC are used in boats) for the
past few years and I just try to keep the same quality of service
for my users.

I understand that the package manager make sense for Windows and Mac
users, it's far less obvious for Linux distros in particular when
OpenCPN is maintained as an integral part of the distro as in
openSUSE.

That why I want to keep building in OBS which does not allows me to
build in a git repo.

   https://build.opensuse.org/
   https://openbuildservice.org/help/

Regards
Dominig  

Le 27/08/2020 à 23:14, jongough a
  écrit :

  All the current build processes use the full directory tree
    including the '.git' branch. The process uses the git command,
    when being built locally, to determine the git repository name,
    branch and tag to workout what type of build to do and where to
    put the results.
  The new process works when using local developer builds on any
    supported platform and by using appveyor, circleci or travis for
    'deployable' builds. For testing purposes the requisite build
    script can be used to do the build if needed for testing.
  Why do you need to do a different process? If you want to use
    the cloud, then use one of the cloud build environments
    customised to your needs. If it requires changes to frontend2 to
    make it work submit these as a pull request to testplugin_pi and
    I will incorporate them if appropriate.
  —
    You are receiving this because you authored the thread.
    Reply to this email directly, view it on GitHub, or unsubscribe.
  [

{ "@context": "http://schema.org", "@type": "EmailMessage", "potentialAction": { "@type": "ViewAction", "target": "https://github.com/jongough/ocpn_draw_pi/issues/455#issuecomment-682194692", "url": "https://github.com/jongough/ocpn_draw_pi/issues/455#issuecomment-682194692", "name": "View Issue" }, "description": "View this Issue on GitHub", "publisher": { "@type": "Organization", "name": "GitHub", "url": "https://github.com" } } ]

-- 

Dominig ar Foll Senior Software Architect

[rgleason]

Dominig The OpenSuSE system sounds like it is a good packager and it has a good responsible maintainer! It would be great if we could get it working with frontend2 somehow, perhaps with a few changes. Can you make some suggestions how to make a simple change that can be made by Jon to frontend2 that will make it compatible with

    the plugins in OpenSUSE build system are also built from git but,
    the build system makes a local cpio which is then automatically
    copied on a compiling worker in the cloud.
    There is no way that CMake configure can "see" the original git in
    such automated build configuration.
    I guess that any built done in distro build system should have
    similar issue.

I expect that these changes would have to be tested by you somehow anyway, as Jon does not have that distro, and then added as a PR to frontend2. Maybe you need a cmake switch to set it into USE_OPENSUSE mode, (but then we would need that switch in the metadata file too perhaps. However you say the distro has already installed the plugins....)

It would be worthwhile to get it working, as there are a lot of plugins that use frontend2. Would another alternative be to manual install all the plugins test them and then make your distro?

[dominig]

Rick,

I have been maintaining OpenCPN for OpenSUSE since I did the initial
port a few years ago. So far it has worked quite well.
The OBS is a very powerful build system and can build for many
distro or model (including Docker or AppImage) as well as providing 
a signed publication service. So far I have used it only to build
for OpenSUSE and to help some friends for embedded Redpesk distro 
https://redpesk.bzh/, I also created a set or rpm for Fedora 32 last
year.
I would be happy to collaborate to define a system that fits all,
for that it would help if I could find somewhere the description of
plugin package model that you have selected. It would be easier than
to second guess the selected structure from the code. Is there any
documentation somewhere ? Have you selected a standardised format ?
Is there any security (signature, hash, ...) added ?

Regards
Dominig

Le 28/08/2020 à 14:32, Rick Gleason a
  écrit :

  Dominig
    The OpenSuSE system sounds like it is a good packager and it has
    a good responsible maintainer!
    It would be great if we could get it working with frontend2
    somehow, perhaps with a few changes.
    Can you make some suggestions how to make a simple change that
    can be made by Jon to frontend2 that will make it compatible
    with
      the plugins in OpenSUSE build system are also built from git but,
the build system makes a local cpio which is then automatically
copied on a compiling worker in the cloud.
There is no way that CMake configure can "see" the original git in
such automated build configuration.
I guess that any built done in distro build system should have
similar issue.

  I expect that these changes would have to be tested by you
    somehow anyway, as Jon does not have that distro, and then added
    as a PR to frontend2. Maybe you need a cmake switch to set it
    into USE_OPENSUSE mode.
  —
    You are receiving this because you authored the thread.
    Reply to this email directly, view it on GitHub, or unsubscribe.
  [

{ "@context": "http://schema.org", "@type": "EmailMessage", "potentialAction": { "@type": "ViewAction", "target": "https://github.com/jongough/ocpn_draw_pi/issues/455#issuecomment-682499543", "url": "https://github.com/jongough/ocpn_draw_pi/issues/455#issuecomment-682499543", "name": "View Issue" }, "description": "View this Issue on GitHub", "publisher": { "@type": "Organization", "name": "GitHub", "url": "https://github.com" } } ]

-- 

Dominig ar Foll Senior Software Architect

[rgleason]

Dear Dominig, That's great. Your skills are far superior to mine and I am just Jon's helper. Jon is taking a well earned break from Opencpn after an intense period of development and now that he can reach his boat, is able to do the needed maintenance and go sailing! He does have email available but is away from his main computer.

To answer, we don't have documentation yet, really, but there are some early attempts that need updating and Leamas has provided documentation for the Opencpn side. I will try to summarize and provide relevant links about the new Plugin Manager system.

1. Opencpn Plugin Manager (leamas prime dev with bdbcat oversight ) This is the brains behind using the individual plugin metadata files that have been gathered together (using https://github.com/OpenCPN/plugins/commits/master commits to master catalog ) into opcn_plugins.xml to install the tarballs in the Cloudsmith repositories. - Leamas wiki is https://github.com/leamas/OpenCPN/wiki

2. Plugin Fronted. The plugin frontend uses CMake and cloud build services (circleci, appveyor and travis) and scripts (in the ci folder) to create installation tarballs for each type of OS build along with a corresponding metadata file. The metadata file is also added to the tarball to allow the "Import Plugin" feature. There are two Plugin frontends being used:

• Frontend1 as shown by the example plugins: oesenc_pi and vdr_pi . This system is comprised of the the folders circleci, ci, api-16, cmake, mingw, buildosx, and the CMakeLists.txt and appveyor.yml and .travis.yml. The vdr_pi example is simpler to look at because I have tried to simplify the folders, also there is a "frontend1.md" readme.

• Frontend2 as shown by Jon's primary example plugin: testplugin_pi with similar folders and a CMakeLists.txt file that has all the primary settings for a plugin (rather than being sprinkled throughout the files) to make it easier for developers to use. There are some more complete readme's in my fork of Jon's testplugin_pi located here Frontend2.md and Read-Build.md.

There are also early documentation in the Dev Manual here (The pages that are "Plugin Manager" or "PI")

I hope this helps. I am trying to get my head around how to make this compatible.

[jongough]

To make frontend2 work in your environment you need to make a change to pluginconfigure.cmake. at the top where it is trying to find the local git put another ifelse and use an Opensuse build environment variable (there should be one available just like there is for circleci, appveyor and Travis). Inside this section set the three variables needed to whatever works for you. If you set the branch to master and a tag you will get a production build if not you will get a debugging build.

Once the build is complete the output files will be in the build directory. Now you will need to package these as you see if.

I would suggest that you fork testplugin_pi, make the changes and test it to make sure it all works (may take a few iterations as you may need to tweak other cmake files). Testplugin_pi is a simple plugin used to test the ocpn_draw_pi API so it is real, does work and has the simplest setup of the plugins. It is why I used it for developing frontend2. If you get it working and Rick can test that all the other builds still work I can then accept a pull into the repository. This will then allow all the other plugins to be updated easily.

An alternative is to use circleci, I believe it will do what you want, and add opensuse to the standard build process. Then cloudsmith would contain the built files. This would require changes/additions to the .circleci/config.yml to add a new build environment and a new build script to do what you want. Then opensuse could be incorporated into the plugin manager process.

[jongough]

Can this be closed now?

[rgleason]

Jon I think you should close it now. Dominig hasnt replied.