flatpak and cmake

[jongough]

Hi, I have been playing with this in my own version of squiddio. I now have a build process for flatpak that works with the full cmake process. All it required was to ensure that these two lines were added to 'docker-build-flatpak.sh': su -c "dnf install -y sudo dnf-plugins-core" sudo dnf builddep -y ci/opencpn-fedora.spec

I have tested this by starting the travis-build-flatpak.sh, but I would assume that 'circleci-buil-flatpak.sh' will work as it seems to run the same docker commands.

The full cmake will now run quite happily. So in my testing the flatpak stuff is in the PluginPackage.cmake file and I no longer require the cmake process to work around the incompatability.

Do you want me to submit a patch for this?

[rgleason]

Good Morning Jon.

This is a great achievement. I think Mauro has given Alec the "Push Token" until he advises that he has the major deployment issues all handled and hands the "Push Token" back to Mauro. If you could hold those changes it would be good. Or since you are a collaborator on my repository, you can make a push and merge to a new branch and it will compile on my cloudsmith. That way we can test the changes.

Let me clean up my squiddio repository (I won't remove any existing branches) with the commands that Alec suggested. I tested them on another repository and they work well (also I have written github support). Then I will make a clean new branch that is identical to the current master and you could push to that and merge (as collaborator)?

Then we will see if flatpak builds.

[rgleason]

Now from the github interface, you can make a pull request to rgleason:flatpak-test Then merge it and see if it builds again. https://github.com/rgleason/squiddio_pi/tree/flatpak-test

You might even be able to make a PR with --tag in it to make the whole thing deploy to cloudsmith

Here is what was done:

C:\Users\Rick\Documents\GitHub\squiddio_pi>git checkout master
Switched to branch 'master'
Your branch is up to date with 'origin/master'.

C:\Users\Rick\Documents\GitHub\squiddio_pi>git branch -m master master.old

C:\Users\Rick\Documents\GitHub\squiddio_pi>git fetch upstream master:master
remote: Enumerating objects: 36, done.
remote: Counting objects: 100% (36/36), done.
remote: Compressing objects: 100% (23/23), done.
remote: Total 31 (delta 20), reused 17 (delta 8), pack-reused 0
Unpacking objects: 100% (31/31), done.
From https://github.com/mauroc/squiddio_pi
 * [new branch]      master     -> master
 * [new tag]         v1.0.17.5  -> v1.0.17.5
   178dae4..f4e5103  master     -> upstream/master
 * [new tag]         v1.0.17.2  -> v1.0.17.2

C:\Users\Rick\Documents\GitHub\squiddio_pi>git checkout master
Switched to branch 'master'

C:\Users\Rick\Documents\GitHub\squiddio_pi>git push origin +master
Total 0 (delta 0), reused 0 (delta 0)
To https://github.com/rgleason/squiddio_pi.git
 + 331084d...f4e5103 master -> master (forced update)

C:\Users\Rick\Documents\GitHub\squiddio_pi>git checkout -b flatpak-test
Switched to a new branch 'flatpak-test'

C:\Users\Rick\Documents\GitHub\squiddio_pi>git push -u origin flatpak-test
Total 0 (delta 0), reused 0 (delta 0)
remote:
remote: Create a pull request for 'flatpak-test' on GitHub by visiting:
remote:      https://github.com/rgleason/squiddio_pi/pull/new/flatpak-test
remote:
To https://github.com/rgleason/squiddio_pi.git
 * [new branch]      flatpak-test -> flatpak-test
Branch 'flatpak-test' set up to track remote branch 'flatpak-test' from 'origin'.

[rgleason]

Jon, I found I had not changed various files necessary to have flatpak-test branch commits build and deploy.

This flatpak-test branch does now build and deploy. https://github.com/rgleason/squiddio_pi/commits/flatpak-test These files deploy to my squiddio_pi directory.

What command would I use (? --tab) so that they deploy to squiddio-stable?

[leamas]

All it required was to ensure that these two lines were added to 'docker-build-flatpak.sh': su -c "dnf install -y sudo dnf-plugins-core" sudo dnf builddep -y ci/opencpn-fedora.specopencpn-fedora.spec

Hm... OK, that builds. However, this might be maintenance problem in the long run. What you do here is that you first load the dependencies required to build a fedora plugin -- this is the deps pulled in by opencpn-fedora.spec. This makes cmake happy when looking for OpenGL, wxWidgets and friends.

However, the point is that none of the libraries pulled in by opencpn-fedora.spec are actually used to build anything. The libraries used are loaded by flatpak install --user -y flathub org.freedesktop.Sdk//18.08 and flatpak install --user -y http://opencpn.duckdns.org/opencpn/opencpn.flatpakref in docker-build-flatpak.sh. So, you are basically cheating, pulling in lot's of stuff not used in the actual build. This of course also creates a lot of needless dependencies that sooner or later will bite us.

This becomes even more complicated when the need to drop support for fedora plugins is considered (fedora is my native platform, but it is covered by flatpak. The fedora plugin support is a left-over).

Bottom line: I'd not recommend this kind of fixes.

[jongough]

I think this build process needs some careful thought. We are now building in travis to build a docker enviornment to build a flatpak environment which takes its instructions from a location on the internet. This seems overly complex and we should attempt to reduce the number of levels used. We should also attempt to make all configuration items for a plugin contained within the git repository of the plugin.

If the flatpak process does everything inside a container why do you have to install cmake, gcc-c++, make and tar in the owning docker? Surely this would be done in the flatpak-builder environment and that would be controlled by a file in the git hierarchy.

For Alec this may be very simple and normal, for the rest of us (me at least) it is very confusing, complicated and anything but normal. The idea, I think, behind this process is to simplify the generation of installable plugins and to make the developer life easier by making the places where changes are needed as few as possible. I am not sure this process is currently doing that. In any case this will need some very comprehensive and clear documentation to explain it and it will require someone with lots of knowledge of the process to maintain it for all plugins when things don't go quite as expected.

Using the new flatpak process I have no idea where I would make changes to include other packages or files, I just keep on disappearing down rabbit holes.

[jongough]

I cannot see how the flatpak process uses the information in the CMakeLists.txt to build the environment particularly as in your version you only do the flatpak process then exit. There seems to be a file, org.opencpn.OpenCPN.Plugin.squiddio.yaml, which has a commit number in it but this should change dynamically with new commits, but I don't see where this is done. I also cannot figure out where the Makefile in the flatpak directory gets generated.

[leamas]

Hi Jon,

Thanks for good questions! I'll try to answer to the best of my ability. That said, your questions are good...

We are now building in travis to build a docker enviornment to build a flatpak environment which takes its instructions from a location on the internet. This seems overly complex.

The current setup is like it is because the docker container running flatpak-builder needs to run in privileged mode (that is started using the --privileged option). Why? Don't remember. But in the end, starting a new container this way became necessary.

If the flatpak process does everything inside a container why do you have to install cmake, gcc-c++, make and tar in the owning docker?

I'm on thin ice here. But as I understand it, gcc and cmake are used in the container, running in the flatpak sandbox. This is different fron the libraries which are present in the sandbox both in runtime and build time (tools like gcc are not available in runtime).

The idea, I think, behind this process is to simplify the generation of installable plugins and to make the developer life easier by making the places where changes are needed as few as possible.

flatpak is something new, and as such adds complexity. The basic question then becomes why should we build a flatpacked plugin?

The reason is that the linux users are divided between different distros such as debian, ubuntu, fedora and arch. Each distro has it own unique set of runtime deps, which also varies as the distro version evolves. Flatpak offers a way to create a plugin which can be used on all these distros, and we need something like that to handle linux.

There are alternatives (snap, appimage...) but their complexity is about the same. I have concluded that using flatpak + debian is a reasonable way of handling the linux users.

Using the new flatpak process I have no idea where I would make changes to include other packages or files, I just keep on disappearing down rabbit holes.

Here, I don't follow you...? The dependencies used when building the flatpak plugin is listed in in the yaml file. Most of them are present either in the sdk or in org.opencpn.OpenCPN.

I cannot see how the flatpak process uses the information in the CMakeLists.txt to build the environment particularly as in your version you only do the flatpak process then exit.

The flatpak directory is basically a self-contained environment for building the plugin. However, the cmake environment is used when deploying it. The Makefile there is not generated, it's hand-made and checked in the git tree. I see three way to build the plugin:

• Invoke the Makefile directly. This will probably end up in defining things otherwise defined in CMakeLists in the Makefile, in particular for deployment.
• Create a separate CMakeLists in the flatpak dir. This would make it possible to reuse some parts of the cmake environment, but still contradicts the idea that everything is defined and driven from one central CMakeLists.txt.
• Or the current setup which basically invokes the self-contained flatpak/ dir but uses common routines to deploy etc.

TBH, I would just handle the situation by refactoring the current code to something like

if (OCPN_FLATPAK)
  include(OcpnBuildFlatpak)
  return ()
endif ()

and be done with it. But then again, that's me.

[leamas]

have no idea where I would make changes to include other packages or files,

If you want a more complete example how flatpak dependecies are handled you could have a look at https://github.com/leamas/oesenc_pi/blob/master/flatpak/org.opencpn.OpenCPN.Plugin.oesenc.yaml. Here we have a dependency (ip-route) togetrher with a much more complicated setup involving a helper binary.

[jongough]

For the build process surely you would use cmake to generate the Makefile, if you don't you are building two different manual build processes and the likelihood of failure increases. With the new process we now have configurations in:

• CMakeLists.txt,
• .travis.yml, appveyor.yml,
• ci/control,
• ci/appveyor-upload.sh,
• ci/circleci-upload.sh,
• flatpak/*
• and possibly ci/opencpn-fedora.spec Then the repositories that go along with this need to be maintained:
• git
• Travis
• Appveyor
• Circleci
• launchpad
• Others??

All of which have to be maintained, tested and understood by a developer.

It would appear that a major simplification process needs to be undertaken to reduce the places where configurations have to be maintained. For me who is trying to build and support a couple of plugins and mods to others this is daunting. It takes a lot of time to try and get my head around this stuff which seem rather fragile. I understand the need for automation, but this current process is rather like an alpha attempt and needs to be refined considerably before we try and get others to use it.

The squiddio plugin is quite simple in its structure and requirements, my main plugin, ODraw, is quite a bit more complicated to configure and I would find this current process difficult to use. If you want a 'play' plugin that is meant to be used by developers then you could try my 'testplugin'. I developed this to help with the OD API to see how to make the JSON and Binary interfaces work. It is also meant to help other plugin developers understand how to interact with the OD API. It does work and looks like a standard plugin but is designed for testing purposes. In this sense it does not matter if it breaks for a while when building this deployment process. Whereas Squiddio is a real plugin used by real users (not developers).

So, I think the following need to be done to the process:

• Make one main configuration file
• Make the process robust and simple
• Do not use 'temporary' web addresses for containing build information
• Document this such that a developer knows what to change and how
• For testing reduce the downloads needed i.e.
  • don't restart docker every time
  • don't update dnf every time
  • make use of caches to speed up the process
  • etc
• don't restart docker every time

To do the testing here I have now got virtual machines for:

• Windows (7 & 10)
• Ubuntu (mint 19)
• Fedora 29 & 30
• Macos 10.4 as well as my normal development machine. This is getting too much for a normal plugin developer and requires good internet access (which I don't have, at a good time I get 3Mbps down and 300kbs up).

[rgleason]

Agreed, Jon, however I think it is very remarkable what Leamas has accomplished. It is a huge step forward, if it could be simplified somehow that would be great.

My father was an inventor, with many patents. He would build and prototype 3 versions of assembly line automation machines, test for failure rate and select one design to be perfected and built by a subcontractor. I think we are near that last important step, Alec knows this too.

[jongough]

Sorry Rick, just updated the comment as I posted it by mistake (clicked the wrong thing!!).

Prototyping is not a bad thing, it is what I did for many of the changes in OD, but I tried to protect most people from the effects. As I said this seems to be an alpha proceess and it may be better to use a non-production plugin for this testing, hence my suggestion of my testpluin plugin.

[rgleason]

Jon, I'm no expert, but Alec has been using the structure he was given (git, travis and appveyor) to that was added circleci and cloudsmith. Circleci and cloudsmith seem to work hand in hand nicely and work well with git. https://circleci.com/integrations/github/?utm_source=bnb&utm_medium=SEM&utm_campaign=SEM-bnb-200-Eng-ni&utm_content=SEM-bnb-200-Eng-ni-GitHub Is it possible to convert the appveyor and travis scripts to run completely in circleci?

Also I hesitate to mention this but GitHub has been an innovator too, and they have created a program similar to circleci I believe. I am looking for the information now.

[jongough]

I have no idea if they could be converted. However the first step should be to reduce the number of places configurations take place. We should not be hand building Makefiles, etc., and let cmake do it for us. If we need to modify cmake to create the requisite Makefiles then that is what should be done.

I am at the very early stages of trying to understand docker and flatpak and do not know where to start. Someone with more experience in both should be working on this as it is probably quite simple if you know how. The same was true for Travis and Appveyor, it took quite a bit of work to get it going properly.

[leamas]

Is it possible to convert the appveyor and travis scripts to run completely in circleci?

For travis, yes. The other way around should also work. But in order to do this in a sane way we need to resolve #71 first. Besides this, this is a low hanging fruit.

Appveyor is the only working ci service used to build using mscv on win32, so it will be required in any case.

Another low hanging vegetable is the different *upload scripts. The should be easy to merge to one, with just some centralized configuration. But then again, this is also blocked on #71.

EDIT: Also, as soon we have resolved #71 it should be possible to clean up the repos involved.

[leamas]

So... if you discuss things in #71, please keep them focused on squiddio. As soon as the discussion is about anything else (like an overall strategy for all plugins) there won't be any conclusion. Which is a problematic blocker.

[rgleason]

Thanks Alec, I think I found the github resource Automating your workflow with GitHub Actions I have no idea if it would be useful here, but it uses yml It is in beta https://help.github.com/en/github/automating-your-workflow-with-github-actions/virtual-environments-for-github-actions supported environments https://help.github.com/en/github/automating-your-workflow-with-github-actions/virtual-environments-for-github-actions#supported-virtual-environments-and-hardware-resources

It is part of github I guess.

[leamas]

Quoting the initial message:

[...] Do you want me to submit a patch for this?

My first answer, basically a "please no", started a long discussion which left the subject of this issue in the dust.

Although the discussion has been interesting, I'd suggest that this issue is closed. This is based on the idea that we have a common understanding that the initial approach won't work and also that we havn't really been able to come up with any alternative.

[jongough]

I would suggest that this issue is not closed but we are ignoring issues that are not convenient. Whilst flatpak may seem like a good idea, it needs to work with all plugins and not just a few. So I think there is more work to be done on this. There needs to be a check that more complicated plugins will work, dismissing it as 'we will get around to it' is a head in the sand approach.

I am having major issues getting flatpak to even build plugins, sometimes it does and sometimes it doesn't and there appears to be no easy way to find reason. The logs certainly don't seem to show why.

[leamas]

am having major issues getting flatpak to even build plugins

Does this include building squiddio?

[jongough]

No, squiddio builds fine using cmake on the same machine and in the directory structure I run ./ci/travis-build-flatpak.sh from. When doing the flatpak build via the travis script it goes through the whole process but does not actually do a build. The following is the 'build' part of the process.

`+ rm -rf build

• mkdir build
• cd build
• cmake -DOCPN_FLATPAK=ON .. -- The C compiler identification is GNU 8.3.1 -- The CXX compiler identification is GNU 8.3.1 -- Check for working C compiler: /usr/bin/cc -- Check for working C compiler: /usr/bin/cc -- works -- Detecting C compiler ABI info -- Detecting C compiler ABI info - done -- Detecting C compile features -- Detecting C compile features - done -- Check for working CXX compiler: /usr/bin/c++ -- Check for working CXX compiler: /usr/bin/c++ -- works -- Detecting CXX compiler ABI info -- Detecting CXX compiler ABI info - done -- Detecting CXX compile features -- Detecting CXX compile features - done -- Squiddio Version: 1.0.17 -- OCPN API Verson: 1.16 -- OCPN Min Verson: ov50 -- Build type: Release -- Configuring done -- Generating done -- Build files have been written to: /opencpn-ci/build
• make flatpak-build Scanning dependencies of target flatpak-build Downloading sources Fetching full git repo https://github.com/mauroc/squiddio_pi.git Starting build of org.opencpn.OpenCPN.Plugin.squiddio Cache hit for squiddio, skipping build Cache hit for cleanup, skipping Cache hit for finish, skipping Everything cached, checking out from cache Pruning cache Built target flatpak-build
• make flatpak-pkg Scanning dependencies of target flatpak-pkg /usr/bin/gtar: Removing leading /' from member names /usr/bin/gtar: Removing leading/' from hard link targets Built target flatpak-pkg
• echo -ne '------\nEND OPENCPN-CI BUILD\n' ` This was done using the master branch from current squiddio git.

I am trying to find out why it is not doing the actual build, but as it is now taking 2+ hours per run this is slow progress.

[leamas]

First, this is not a problem with the squiddio code at all. I propose that you close the bug. It's really a cuortesy to @mauroc, here is nothing that needs to be fixed.

We could still continue the discussion here in a closed bug.

Secondly, the log says it all: Everything is already built and cached. flatpak-builder caches results of earlier runs in the .flatpak-builder directory, and will not rebuild things as long as the sources are identical.

[leamas]

BTW: the flatpak-builder manpage is worth reading, it fleshes out some of the details on this

[rgleason]

Maybe the discussion should continue on ocpndraw_pi Issues since it is obvious that Jon would like to get his PI running with the Plugin Installer?

Perhaps some flatpak issues will get solved there.

[leamas]

Indeed.

[rgleason]

I've created a new issue in OcpnDraw https://github.com/jongough/ocpn_draw_pi/issues/414 so that the flatpak / ocpndraw issues can get resolved. I leave it for someone else to close this issue.

I think it is pretty clear now that Cmake is going to be improved in increments now, after Jon's big set of improvements.

[rgleason]

Jon, you must be a Taurus.. very stubborn and persistant. Have you found that a rerun takes less time?

[jongough]

No, Scorpio.

No the runs all take quite some time. I have tried to create caching for the rpm's and sometimes it works and sometimes it doesn't. As I have a slow internet connection these large downloads take quite some time.

Currently I would have thought that flatpak would have been a new environment when a new docker client image is started. I am not sure how 'caching' would work as far the client is concerned it should be a new instance with clean directories. As flatpak is a sandbox and it is run in docker which is a sandbox I would have thought restarting docker and then the flatpak build would have resulted in a clean build. If not this may be an issue.

[rgleason]

I'm Taurus. Still so much to learn. Alec uses docker containers, very handy, and it appears I don't need to have a docker account or even touch it. It just works when asked. Very powerful. Wish I could advise.

Does Docker for Windows take the place of a Virtualbox + Linux?

[leamas]

I would have thought restarting docker and then the flatpak build would have resulted in a clean build. If not this may be an issue.

Restarting docker definitely creates a pristine environment.

Did you by accident include flatpak/.flatpak-builder in the git tree? That would explain things...

[jongough]

I found flatpak/.flatpak-builder in the directory structure. I removed it and issued the ./ci/travis-build-flatpak.sh and it built the plugin. The docker container was still around at the end so I checked and the build process created the .flatpak-builder file. Interesting is it created it in the directory structure from which I ran the command. This suggests that the docker and flatpak build environments are not quite as 'sandboxed' as you may expect.

To demonstrate the issue have a ubuntu machine (I am using qemu/kvm virtual machine for this), clone the squiddio repository to a directory on this machine then issue ./ci/travis-build-flatpak.sh from the root of the squiddio repository (I have mine in ~/development/squiddio). The build should progress if you have docker and its prerequisites installed. At the end of the process you will find the offending file in the squiddio directory structure on ubuntu. Either there is something wrong with the sandboxing or there needs to be a clean up done of any files generated to ensure they cannot impact a future build.

[leamas]

You have probkems wrapping your head around flatpak. Nothing wrong or strange in that. However, it has nothing to do with squiddio. Using the squiddio issue tracker as a message board for unrelated stuff is IMHO somewhat disrespectful to other parties involved in the squiddio project.

So, please close this bug. I will continue discussion in https://github.com/jongough/ocpn_draw_pi/issues/414

[jongough]

As Squiddio is trying to use a new process and it is when using this that the issues arise then it would seem sensible to report and discuss them here. Transferring the issue to another plugin does not make this issue go away. There are problems with the current Squiddio travis build for flatpak, this needs to be worked through and fixed.

[jongough]

In light of the above, if docker/flatpak is going to make 'build' changes outside of the sandbox (this goes against the whole point of a sandbox, think VirtualBox and snapshots) then these should either be cleaned up at the end of the build or handled correctly at the start of the build.

The process I am using is not to allow me to compile the plugin but to try and replicate what travis does and find out why it does not always work. Rather than having to 'play' in the real world I have setup an environment which is very similar to travis (as far as I can tell) and uses the travis scripts. It is these scripts that I am trying to understand and make more robust.

I know I am learning, so I cannot be across all the vagaries of docker and flatpak, but when I find issues I am raising them for the experts to try and help. The travis build process does not make use of caches, re-runs in current docker containers, or old copies of runs that have happened before, so that is what I am trying to do. The whole process seems very heavy handed with the amount of downloads needed to build the build environment. I am on a slow Internet link hence I notice it, but for the travis maintainers it will become obvious that certain processes are taking large amounts of bandwidth and that may become an issue. Perhaps a pre build docker/flatpak environment would be a better prospect that can be updated occasionally if needed. This would streamline the process and make it faster and more predictable.

[jongough]

An example in the current Squiddio docker-build-flatpak.sh is that there is a line to delete the build directory, create the build directory then change to it. Surely the simple solution would also be to ensure the flatpak/.flatpak-builder directory has been deleted prior to the start of the build. This would then insure a full rebuild of the plugin occurs.

[leamas]

Anything which actually doesn't work in squiddio is a problem in squiddio, and is probably then best handled in a separate bug describing the issue. Other things should really not be here.

[leamas]

Perhaps a pre build docker/flatpak environment would be a better prospect that can be updated occasionally if needed.

Perhaps. That said, current jobs on circleci completes in between 3 and 7 minutes, which actually isn't that bad. If anyone is motivated enough to create and maintain such images it would certainly be helpful. However, the maintenance of such a thing shouldn't be underestimated.

[jongough]

As you are using squiddio as you test bed where else should issues be raised? The only test platform we have at the moment is the squiddio plugin. I have offered the use of one of my non-user plugins for OCPN, but that has not been taken up. So the only place I can raise issues with the new build process is the one that is currently being used.

[leamas]

I have no problem to try to handle and reply to flatpak-related tings w r t squiddio. However, an issue should not be an endless discussion, it should be something which eventually could be resolved.

The issue here, as filed, was a simple question which we together, as I understand it, have answered with a simple no. With that, this issue should be closed.

If you have other issues, please file them separately. I think the discussion will benefit a lot from this. If about things which doesn't work in the squiddio code, please file them against squiddio. Otherwise, please file them "at home", so to speak (when it comes to pidraw you are the maintainer and thus makes the rules).

[jongough]

To try and allow a more generic process I am currently modifying my testplugin_pi with this code but generalising it along the way. When this is complete it can then be used as a template for all plugins. If you want to help or provide comments you will find the repository here: https://github.com/jongough/testplugin_pi

[rgleason]

I think this is a good idea.

[leamas]

If you want to help or provide comments

Done.

Can we close this bug now?

[rgleason]

Leamas, I can't close it, but Mauro and Jon can. Perhaps just don't worry about it?

[jongough]

The original issue with cmake and flatpak is currently solved, although not all agree that it was done the correct way. I think there are other issues in the cmake/flatpak environment that need fixing up, but I have moved to doing this on one of my plugins. If these changes work then they probably should be retrofitted to this plugin.

[leamas]

The original issue with cmake and flatpak is currently solved

Then please close this bug.

I think there are other issues in the cmake/flatpak environment that need fixing up

If so, please file as separate issues.