search

rgleason / weatherfax_pi

Weatherfax Plugin for Opencpn
GNU General Public License v3.0
5 stars 9 forks source link

cmake flatpak #96

Closed ozolli closed 2 years ago

ozolli commented 2 years ago

I have no idea what the cmake command line parameters have to be set to compile a flatpak plugin...

Rick, can you help me please?

rgleason commented 2 years ago

@leamas Good question! I think we need to have Alec Leamas advise.

ozolli commented 2 years ago

And do the Weatherfax 1.9.31 and 1.9.32 flatpak build fail on you? I don't see them on Cloudsmith.

rgleason commented 2 years ago

@ozolli @bdbcat was going to take a look at the 4 failing flatpak builds when he had a chance. I also asked @jongough about this because he might know how to fix it. There is some problem with finding the right libs for the ftp feature.

bdbcat commented 2 years ago

Working on this with Alec. Not trivial.... Hope for progress today. Dave

rgleason commented 2 years ago

Much appreciated. ..Did not realize it was a problem.

bdbcat commented 2 years ago

PR posted to your repo now. Fingers crossed....

leamas commented 2 years ago

hm... @rgleason: If you got a PR moving weatherfax to the shipdriver templates: would that be a good thing or not?

rgleason commented 2 years ago

Alec, that would be ok with me, I am a little nervous about it though.

There are a lot of moving parts right now. We are updating the data xml files (links to MET services), also trying to get the Weatherfax schedules updated and I would like to get the Kap function fixed and confirm that we can decode sound and fix that if necessary.

It would probably be better after we get this done, unless you think it would help.

rgleason commented 2 years ago

PR posted to your repo now. Fingers crossed....

It built the flatpaks! now finishing with MacOS. I had one user say MacOS is not working, but no detail. Going to put this up on Beta channel and get it tested by the crew.

Thank you Dave and Alec. BTW Ozolli did a super job on the WFIR xml files!

leamas commented 2 years ago

It would probably be better after we get this done, unless you think it would help.

let's wait then. That said, I actually think it helps in the long run.

rgleason commented 2 years ago

Understood Alex.

ozolli commented 2 years ago

It built the flatpaks!

Flatpak ftp download working fine on Rocky Linux 8.6. Well done guys !

jongough commented 2 years ago

Have extended the ability to add extra libs to flatpak, armhf and arm64 builds in frontend2 1.0.205.0 . The method implemented here breaks the ability to use a template build process.

bdbcat commented 2 years ago

Jon... I took a pragmatic approach to get the weatherfax_pi to build, so that it might be released. I'd be happy to follow your ideas so that the template stays sound. Please describe your design, and how it might be used to include libcurl on flatpak.

jongough commented 2 years ago

This is part of the testplugin_pi that I used for OD development and now for frontend2. If you go to "https://github.com/jongough/testplugin_pi" you will find an extras directory under the ci directory. In this directory there is a README.txt file and an extra_libs.txt file. I created the second form of file name in the readme for the missing ssllib and added the requisite code to the ci script. This is the code that does it:

# Install extra libs
ME=$(echo ${0##*/} | sed 's/\.sh//g')
EXTRA_LIBS=../ci/extras/extra_libs.txt
if test -f "$EXTRA_LIBS"; then
    while read line; do
        sudo apt-get install $line
    done < $EXTRA_LIBS
fi
EXTRA_LIBS=../ci/extras/${ME}_extra_libs.txt
if test -f "$EXTRA_LIBS"; then
    while read line; do
        sudo apt-get install $line
    done < $EXTRA_LIBS
fi

and the entry in the extras directory is:

circleci_build_flatpak.txt

containing:

libcurl4-openssl-dev

The idea being that a developer can use the template builds and add customisations to the installed packages without modifying the provided templates. The only file that needs to be maintained by the developer is the CMakeLists.txt as all the customisation for cmake is in there. The 'libs' directory contains the extra libs that have to be build with their own CMakeLists.txt file. As we now build for 18 different platforms it is much easier to have one person maintain the build templates and others just use them by upgrading (copying) the new/changed files. The build process can be customised during development to build only those environments required and, with the latest patches, use apt proxies and avoid downloading master.zip through the use of circleci local.

Hope this helps.

leamas commented 2 years ago

Here are really two things. The first is about using curl in Flatpak. The simple answer is that curl is included in the flatpak runtime, the flatpak OS so to speak. This means that only thing needed to use curl in flatpak is to let cmake find it and then use the cmake data to add include headers and library. Nothing special is required for this, cmake finds it out of the box.

jongough commented 2 years ago

I only added the file that Dave added to the flatpak build. The process to do so exists in frontend2, I just updated the process to include a few other scripts that use apt. I made the update to the file that could have been used rather than modifying a standard template, the script file, that would cause this plugin issues in future when changes are made. If it is not needed it is one file to remove, the template will stay the same.

leamas commented 2 years ago

The other is the use of EXTRA_LIBS in Flatpak. In general this does not work, but see below on -dev packages

Flatpak runs in a sandbox, and the only libs a flatpak application sees are those in the runtime and possible libraries built from source. In particular, libraries installed using apt-get are not visible in the sandbox.

Building from source could be done as part of the plugin build, or as a separate source in the .yaml flatpak manifest.

For a plugin, the runtime used is OpenCPN. This means that besides the standard libraries like for example curl also all libraries linked by OpenCPN are available. This is the reason for example wxWidgets is visible for plugins. Which means that if a plugin needs to link to a library, the Flatpak build has three alternatives:

  1. The library is available in the Flatpak runtime. All is fine, just let cmake do it's job.
  2. The library is available in the OpenCPN runtime. In this case, the headers are missing which is pretty complicated. The simple solution is to avoid it. The alternative is to somehow provide correct headers (and to maintain them)
  3. Build the library from source. This could be done as part of the plugin build or by adding a new module to the Flatpak .yaml manifest

Added to this is of course MacOS and Windows. We cannot use any external libraries here either, we cannot assume users are using package managers like choco or Homebrew. This basically boils down to that libraries not available in OpenCPN are better off built from source, this is a solution which works everywhere.

For Debian, there is a special case with a library like curl which is available in the runtime because it's linked to OpenCPN. To link against such libraries the build needs the corresponding -dev pack (for curl, it needs libcurl-dev). This is OK to add to the build environment since it does not add a requirement to the plugin runtime. libcurl4-openssl-dev is another example on this. But then again, this does not affect a Flatpak build

As I see it, these are hard facts that a templating system needs to adapt to.

rgleason commented 2 years ago

Hmm. It's not simple, but it is. I guess we have to build the curl library in weatherfax?

rgleason commented 2 years ago

JG has kindly provided a PR for weatherfax which uses his method. I am going to accept it as that will conform with his scripts in TP. Improvement PR's very welcome. @bdbcat Thank you for getting it working, testers are working with it now.

rgleason commented 2 years ago

Well now the 4 flatpaks are failing (android usually fails anyway, should comment it out). JG's config has a problem with flatpak as Alec noted. Perhaps I should revert that PR?

rgleason commented 2 years ago

I reverted Jon's PR so weatherfax would work with flatpak.

leamas commented 2 years ago

Hmm. It's not simple, but it is. I guess we have to build the curl library in weatherfax?

I have yet not looked at the code.Or rather, read but not to the point I understand.

But as for Flatpak, no. It is included in the Flatpak runtime.

bdbcat commented 2 years ago

Alec... Just for my clarification: If we simply install "libcurl4-openssl-dev" in the host build system, how is it that we know that the same API is available for the libcurl.so which is included in the "sandbox" by virtue of being part of OCPN?

bdbcat commented 2 years ago

Note on Windows: libcurl.dll, and all required header files, is included in the downloaded build dependency package. So I see no need to rebuild from source in a plugin.

leamas commented 2 years ago

@bdbcat: It's the other way around (I think)

leamas commented 2 years ago

libcurl.dll, and all required header files, is included in the downloaded build dependency package. So I see no need to rebuild from source in a plugin.

In that case we should probably update the curl library I provided to you, have the Windows headers and dll file(s) there and use them on Windows. It's much easier to make this system modular if we group things in a library which covers all platforms rather than having all windows stuff on one place, macos in another , etc.

Will look into it, but now heading for wine and Friday dinner.

bdbcat commented 2 years ago

So, can we not get the curl headers from the flatpak SDK? I tried to do this, but cannot find the directory "/app...." in the build system, although flatpak-builder seems to know about it. I poked around using a CCI ssh login.

leamas commented 2 years ago

So, can we not get the curl headers from the flatpak SDK?

Yes, cmake finds them with default settings

bdbcat commented 2 years ago

No, if I do not install "libcurl4-openssl-dev" in the host build system, then the find_library, find_header find nothing. Added comments to cmake:

-- Adding libs/curl -- libs/curl/CMakeLists.txt processing flatpak -- libs/curl/CMakeLists.txt LIBCURL: LIBCURL-NOTFOUND -- libs/curl/CMakeLists.txt CURL_HEADERS: CURL_HEADERS-NOTFOUND

bdbcat commented 2 years ago

Rick... Jon's patch "almost" works. It fails to execute the extra_libs detection code on flatpak. This is an easy fix which I will leave to Jon tonight.

bdbcat commented 2 years ago

@leamas

In that case we should probably update the curl library I provided to you, have the Windows headers and dll file(s) there and use them on Windows. It's much easier to make this system modular if we group things in a library which covers all platforms rather than having all windows stuff on one place, macos in another , etc.

Agreed.

leamas commented 2 years ago

@bdbcat. It's there. "libcurl4-openssl-dev has nothing to do with in the Flatpak case:

$ flatpak run --command=bash --devel org.opencpn.OpenCPN 
[📦 org.opencpn.OpenCPN OpenCPN]$ ls /usr/include/curl/curl.h
/usr/include/curl/curl.h
[📦 org.opencpn.OpenCPN OpenCPN]$
bdbcat commented 2 years ago

The plugin build script does this (for a flatpak beta build):

 flatpak install --user -y flathub org.freedesktop.Sdk//20.08
 flatpak remote-add --user --if-not-exists flathub-beta https://dl.flathub.org/beta-repo/flathub-beta.flatpakrepo

 flatpak install --user -y flathub-beta org.opencpn.OpenCPN

Do we need something else?

leamas commented 2 years ago

No

bdbcat commented 2 years ago

Well, then, I cannot explain it. Using the supplied lib/curl CMakeLists.txt, the only pragmatic solution that actually works IRL is to install libcurl4-openssl-dev in the build-bot. (grumble)

leamas commented 2 years ago

There are two cases:

On a sidenote, there are different curl packages linked to different ssl libs. It might be better to install the virtual libcurl-dev package which picks the proper implementation.

bdbcat commented 2 years ago

Leamas... That all makes sense to me. However, I'm left with the bald facts:

  1. The cmake file does this: 
    
flatpak install --user -y flathub org.freedesktop.Sdk//20.08
flatpak remote-add --user --if-not-exists flathub-beta https://dl.flathub.org/beta-repo/flathub-beta.flatpakrepo
flatpak install --user -y flathub-beta org.opencpn.OpenCPN
2.  Then, it does:

message(STATUS "libs/curl/CMakeLists.txt processing flatpak")

On Linux and Flatpak: use the system library and be done with it.

find_library(LIBCURL NAMES libcurl curl REQUIRED) message(STATUS "libs/curl/CMakeLists.txt LIBCURL: ${LIBCURL}") if (${CMAKE_VERSION} VERSION_LESS 3.18) find_path(CURL_HEADERS NAMES curl.h PATH_SUFFIXES curl) else () find_path(CURL_HEADERS NAMES curl.h PATH_SUFFIXES curl REQUIRED) endif () message(STATUS "libs/curl/CMakeLists.txt CURL_HEADERS: ${CURL_HEADERS}")

3.  Result to console is this:

-- libs/curl/CMakeLists.txt processing flatpak -- libs/curl/CMakeLists.txt LIBCURL: LIBCURL-NOTFOUND -- libs/curl/CMakeLists.txt CURL_HEADERS: CURL_HEADERS-NOTFOUND


Logfile is here:
https://app.circleci.com/pipelines/github/bdbcat/weatherfax_pi/7/workflows/9a0c964b-ac5b-4181-af8a-966c9e23b962/jobs/110

???
rgleason commented 2 years ago

Jon, I accepted the PR and all the flatpak seem to be failing still. https://github.com/rgleason/weatherfax_pi

So maybe cmake isn't working as expected?
This is still a work in progress. So I am going to revert your PR to get it working again. Then try to give the volunteer testers some real files to use. This testing is not starting well. I am not using a non-master branch, so the CS beta repos has turned over too much. This is not good practice I guess, but this issue has gotten more complex daily. Something I've just realized, as the number of packages build goes up, the retainage in CS has to increase. I do not plan on redoing all of that anytime soon, so I hope the number of packages will go DOWN soon.

leamas commented 2 years ago

@bdbcat : I need to look into this. This obviously works in the shipdriver case. Give a day or two...

bdbcat commented 2 years ago

OK, thanks. Meanwhile, I'll compare with the shipdriver log.

jongough commented 2 years ago

I have created a new pull request for weatherfax_pi which corrects the issue (naming and typing issues on my behalf) of not installing the library that Dave identified. Now the flatpak builds all work. What I did notice is that to install the dev files it also had to install curl and libssl. I thought these should have been in the base flatpak image, but....

rgleason commented 2 years ago

Thank you Jon, I will merge your new PR, but will first change the retainage on weatherfax-beta just to be sure I don't mess up the testers.

bdbcat commented 2 years ago

Jon... For flatpak build, there should be no need to install anything related to curl. They should be already in the flatpak base image. But they seem not accessible to the flatpak builder. That is why I patched to install them explicitly. There is definitely something fishy going on here, and Alec and I are debugging. More as we know it. Dave

jongough commented 2 years ago

Dave, The change made to frontend2, from a build perspective, is just a text file that can be removed from the 'extras' folder. It allows the build to go at the moment, but if it is not needed it is a simple delete.

rgleason commented 2 years ago

Since eyes are on this Issue. Are there any ideas about armhf bullseye gtk3 rpi Beta Catalog Issue #105

rgleason commented 2 years ago

Sean has merged a big PR for Weatherfax 9 hours ago! Should I now make a PR from his repository to be "current"?

leamas commented 2 years ago

Trying to have a life this weekend, and to get our boat ready.

After a nights sleep I'm pretty convinced that this is about the build system. In particular, if a Flatpak build fails without a -dev host package, this is a bug. The build system should not enforce any host packages for building Flatpak, they are by definition useless for this purpose.

Furthermore, here are notes about installing the rtlsdr-dev package when building. This is a Bad Idea. If we build against this package it means that users will have to install the corresponding rtldsr package. Without it, plugin will not load.

rtlsdr hasn't many dependencies and should not be hard to rebuild from source. This is most likely the correct approach since it should fix the situation for Windows and MacOS as well (that is, if it is buildable).

In short, this is messy. Also, interesting, I actually don't find any references to curl here: 

 git grep curl ../src ../include/
../src/InternetRetrievalDialog.cpp:#include "wx/curl/ftp.h"
../src/InternetRetrievalDialog.cpp:#include "wx/curl/dialog.h"

Those refers to wxcurl, another package. What am I missing?

bdbcat commented 2 years ago

libs/wxcurl depends on curl.

from libs/wxcurl/CMakeLists.txt


  find_package(CURL REQUIRED)
  target_link_libraries(WXCURL_IF INTERFACE CURL::libcurl)
bdbcat commented 2 years ago

So, after comparing many build logs, trying different experiments, and thinking pure thoughts, I now understand the flatpak build process, as relates to dependent libraries, anyway. The key: For flatpak builds, cmake runs twice: once to configure the flatpak environment, and then again to actually find the dependents in the sandbox, and create the Makefile. In Jon's template, there is a cmake flag (OCPN_FLATPAK_CONFIG) that controls this staging. I had forgotten all about that. With this understanding, and appropriate changes to the various make script files, the build completes for flatpak, and confirms leamas' explanations regarding SDK sandbox logic. I'll post a PR to Rick's repo now.

Sorry for the run-around. Dave