Flatpak build files not consistently available

[jongough]

There is an issue with http://opencpn.duckdns.org/opencpn/opencpn.flatpakref, flatpak builds keep failing because this website does not respond and a timeout occurs. The file it is trying to get contains:

[Flatpak Ref]
Name=org.opencpn.OpenCPN
Title=OpenCPN - Concise ChartPLotter - Main package
Description=OpenCPN main package, loaded by Plugin.Base
Branch=master
Url=http://opencpn.duckdns.org/opencpn/repo
Homepage=https://opencpn.org
Icon=https://opencpn.org/OpenCPN/assets/img/logoOriginal.png
GPGKey=mQENBFyk1UIBCACwFaQLPtrmEhOddy7LYJ6wmRJs9MNBtSp7F3+3uWX+uycQSW3strOwbyYwiDsuymWwmjRfDDBClYvIrFlUAB/OPiuSimE7CHZVye0zbT9i9W4LMf2uRYtYxkZYAngF6NPye8qANRbwocqn3QPBjlNdjd7OjBSCpBUwy50EW3JtstxKeth3Fn7a6EI3sCkHZLaFxhO3A6uxk89Wpq1lBqWAkkR1zMHrIs1TKrvGP/Dfc0ShYdwWaOeuzkWaDYkF0SZiSQcV3JLkIIJh8KJpos4+izYdmrLkDFdpyXit9OByd42mjbWkHKicAkUKrsjQDL2G6FSXs1IdTBh3j1y88aVPABEBAAG0EmxlYW1hc0BvcGVuY3BuLm9yZ4kBVAQTAQgAPgIbAwULCQgHAgYVCgkICwIEFgIDAQIeAQIXgBYhBBcYLymPkdBZBDv7MPuc+iyUDyOBBQJeYh4bBQkCqpbZAAoJEPuc+iyUDyOB/IYH/Rk9UVwUoe+9kNkR/lFrxoMWr9nPCRVs/8Sa1cc/qjuyp1sHkF4fiOJtbOUIam3mWkz1Kikh52i8A+aWdTQSicFORR2Te8rbi3KMsXA2+KT6IKLw1oq7/uK935PKShIm7JPjCwiWpY87zYynmhsr34/7PbsYY4El9wxyNFBUOxXrJgQwUHAlg4JlMEgm8CTG0eNjCjMRc4EbAD3T4oDRFgtiBAG/cEBgnUYm+O5BSUSg8xPPUXx/1+n35+HMQ9Dd1WTL9o/ZEAqgQWW4kXVGd4oUKZiKW2tWo1mMlKMvCPykk9b8delXcZ/cb04dVeUfqL9j5LPeDLXUurxFd0AWl0S5AQ0EXKTVQgEIAKurHsSJyZCsi0MBYbPd9LFYgf6hQK61TwVq7bk0d9VJxbCBhHl2p+hFJB786cQrPEXOy76QrLvooM8dsP6f/qdUkIb4esBteq07bga+HQOc94KafZ54OWFMn/7g+abU3RAKbM7RwfDj20wvEQuqxMeA5JNzZaamAa84Uq9ZdMpDJL4qN/4VTST5XFgGpRYM+qTC54HRfRh2uOUy13nlYpbUKQU1PcBvuOU6lhRs1fEr+VBbNfVmS1zfagNQ8h0om+65AynWkU9eqCIstaH++6e7bobv1rLo9QZsyMZl14Qw5FQzGKobqy8TdB2/D62SyDJOcIVjef0eB55Ys30cSkkAEQEAAYkBNgQYAQgAIBYhBBcYLymPkdBZBDv7MPuc+iyUDyOBBQJcpNVCAhsMAAoJEPuc+iyUDyOBQa0IAKwnA2CF9V21VeWdhQxSrYgtL8cLcsV7wVAsR5dPvC6gDzbrG6kFd5GRAs28EdsqDJvyhqlGWqcMc1ldEUiE3OpPxCTLHfRoccBgTNYJd2yTPyKjDyqsiEYGtKXBMadUrS0NdOM/d4Yj7mtX9VDS/bl/KWo9W3PJc2IxKeXGEDhOdy8jppFStPCoIOeSTDWGzRAZ9UPMANojbx5cmYrP8SxCSneJGZKbdvDFMEmmmFwHut3PFPdmF8rwkpQQzjXrR5a59pereGU910jWpxaazwMaoJaI8NOOp06XZfufR6IGyMWvhvwtxKW6GG+WfLCta9hqtmzZQiwLeStn3dfkbbc=

Which refers to Url=http://opencpn.duckdns.org/opencpn/repo. The error that turns up in the flatpak build is:

+ flatpak install --user -y http://opencpn.duckdns.org/opencpn/opencpn.flatpakref
GLib-GIO-Message: 08:26:37.680: Using the 'memory' GSettings backend.  Your settings will not be saved or shared with other applications.
error: Can't load uri http://opencpn.duckdns.org/opencpn/opencpn.flatpakref: Could not connect: Socket I/O timed out

Is there some other way to install the required files for flatpak?

[rgleason]

So sorry, perehaps I should drop doing the rpi?

[leamas]

Could we just postpone it a little, say to after Xmas?

[rgleason]

Yes, of course.

[leamas]

I have started the attempts to set up a "cloud" for opencpn downloads, with focus on redundancy, availabilty and possibilities to handle stuff not easily hosted on traditional web services.

The first step is that that there is now two independent servers. End goal is more servers accessible in a transparent way using DNS which just picks a reandom, available server. However, we are not there. For now, the duplicated two servers are mumin.crabdance.com and gafsan.crabdance.com. Both offers http access:

• http://xxx.crabdance.com/opencpn-dl is the plugin downloader (https://github.com/OpenCPN/OpenCPN/issues/1865)
• http://xxx.crabdance.com/opencpn-flatpak-beta is, despite it's name, the flatpak nightly builds repository.

@jongough : It might make sense to add code in the build chain to use one of these server as fallback if the current fails. Eventually, this should be transparent, but setting up these things needs time.

@rgleason: We need more servers in the cloud, and after the holidays I will be able to talk more on this,.

mumin and gafsan are on the same MAN but have different ISPs and physical addresses around 1 km apart.

[rgleason]

Ok, I think I will order a rpi and start.

[leamas]

@rgleason: I you have a need for a rPI for you own needs, that's of course up to you. But when it comes to a role in the anticipated cloud nothing can be said until at earliest after the holidays.

[rgleason]

Alec, not a problem if you decide not to use it, however I will try to get it ready. Its a project I want to do anyway.

[lskillen]

hm...

was now time to create a more 'formal' process for access to these files

In short: yes, we need a better solution. But cloudsmith, or anything like that doesn't fit the bill. I'll try to explain why, but this will likely become a too long post.

The flatpak build process creates a repository. This is based on ostree, which in many ways is similar to git. In particular, each build doesn't create a new file as other builds. It just creates a commit and tag in the repo. When you as user "downloads" an application you basically clone a repo, similar to a git clone.

The repo is quite large, currently about 300M.

Building opencpn flatpak plugins means that you need to install the opencpn flatpak application so you can link to it during the build. This is done by dowloading an opencpn.flatpakref or opencpn.flatpakrepo file. These are just pointers to the repo which makes it easy for users to easily add an application or repo to their own software. The basic contract is that if you can access these files you can also access the repo they refer to.

Storing the flatpakref/flatpakrepo file on cloudsmith would easily break this contract: since the repo might be unavailable while the files isn't. There will also be a synchronization problem. In short: we should IMHO keep the flatpakrepo/flatpakref files in the same server as the repo to keep the deployment simple and sane. More to come...

You might be interested to know that Flatpak is on the roadmap for Cloudsmith. So that might help in the near future if we support it as a native format. We already support other formats that are "git-like" (e.g. Cargo for Rust, and CocoaPods). It's meant to be a semantic repository after all, with intelligent support for formats so they work with native tooling like apt and flatpak, and not just a file dump (which is probably the least useful way to use Cloudsmith imho). :)

[rgleason]

Thanks Lee!

Vote for flatpak support on the Cloudsmith development path. Only 2 votes so far. You need to register an account on https://trello.com/b/IhvGx4aN/cloudsmith-roadmap

Once you've voted you can "Watch"

https://trello.com/c/BXjGt1BO

Flatpak Packages

Also see Formats for inclusion roadmap https://help.cloudsmith.io/docs/supported-formats#format-roadmap

[leamas]

Hi, hope you've had a nice Christmas

According to the Trello kanban link above, work on a cloudsmith flatpak integration will not start until at earliest Q2 2021. Even if it eventually becomes a viable alternative we IMHO need a solution sooner than that. So, while an interesting piece of info, it probably doesn't affect us right now.

To be frank, I also see challenges when it comes to hosting flatpak repos on a service like Cloudsmith. The combination of missing push support in ostree, very large repositories and an open-source offer without fees is, well, interesting.

EDIT: Remove link inserted by my touchpad. It wasn't me, for sure...

[lskillen]

Merry Christmas and Happy Holidays to y'all too!

Sure, I get it. You'll appreciate that we drive development based on demand and that's why I mention it. In terms of the challenges, other than timeliness I don't see any issues with the others? Push would be supported (note: I realise you mean that a remote OSTree itself has issues, but we'd figure it out; since that's our job). Plenty of our repositories for others are well into the TBs range. OSS projects don't have fees. 😄

[rgleason]

See https://github.com/OpenCPN/OpenCPN/issues/2296 Alec says " In general, production plugins should be built against the flatpak version available in flathub, this is users actually use.' (...not his little development server). (:-)

It appeared to me that Jon, in Testplugin, was using some circleci resources with a docker image, so that is possibly why Alec's server is used. Flatpak has a history with plugins and they are not completely resolved.

I don't believe we have access to that image (is this in "master.zip" perhaps? - later: no that is for android), and Jon is being denied access to circleci by US Treasury right now.

Flatpak Builder And as Alec suggested Available Flatpak Runtimes

I would like to get weatherrouting_pi and other plugins built and done for the summer with Flatpak and RPI builds and it is more involved to do this piecemeal.

https://stackshare.io/stackups/docker-vs-flatpak

[rgleason]

The second try worked, but testplugin_pi FE Flatpak builds need to be changed to use the flatpak versions.

[leamas]

The core issue here is that my home servers are not fully reliable. However, to build flatpak plugins these servers are not required any more. In fact, using my servers should be considered deprecated. This is because my servers represents the development branch, not the stable branch used by users.

The stable branch is these days available on flathub (this was not the case when we started the flatpak builds). Thus, today the flathub servers should be used instead of my home server, see examples in the shipdriver repo. Given this, the original issue here is IMHO moot and should be closed

[rgleason]

Jon, perhaps this can be closed now?