htmlcxx dependency may be impossible to build from source on some distros

[ssokolow]

I decided to try making a Flatpak manifest for lgogdownloader without waiting for your comment on whether you'd be receptive, just as an exercise, but htmlcxx isn't in the shared runtime and attempting to build it fails with this linking error:

libfl.so: undefined reference to `yylex'

I tried switching to 0.85 as is depended on by the Ubuntu 20.04 LTS lgogdownloader package instead of 0.87 (the newest version), but no change.

According to a couple of StackOverflow answers [1] [2] that error is caused by an API compatibility break in Flex 2.6.x, which would make sense given that htmlcxx hasn't seen a release since December of 2018 while the ecosystem of things that necessitate flex compatibility hacks would be moving on around it.

The answers point out that Ubuntu provides a flex-old package for that, but I don't see how that's a viable solution for other distros, which Flatpak's runtime essentially is. Am I supposed to find a distro that's maintaining their own htmlcxx patchset and depend on their source deb/rpm/etc. instead? ...intentionally build an older Flex as part of the build process?

Can you provide more detailed instructions?

Failing that, would you be willing to consider vendoring a patched version and offering a build flag to use it?

I have no experience with Flex, Yacc, or Autotools (Just Rust, Python, PHP, JavaScript, Bourne Shell, some entry-level experience with Watcom C and WMake for a retro-hobby project, and fading memories of various other languages like Lua, Perl, QBasic, DOS Batch, etc.) so my expertise from my time spent typing ./configure && make && sudo make install in the early 2000s stops at reading ./configure's log to get more detailed error messages and googling for solutions.

(The StackOverflow answers suggest directions for patching the source, but blindly patching without understanding is how the Debian OpenSSL Fiasco came about. Hence my question about whether I'm supposed to find some Debian/Fedora/etc. downstream patchset to depend on.)

[FabioLolix]

Hello, you can find a patch to build htmlcxx here https://sourceforge.net/p/htmlcxx/patches/8/

https://sourceforge.net/p/htmlcxx/patches/8/attachment/htmlcxx-0.87-c%2B%2B17.patch

htmlcxx has been added to Arch repository on 2020-09-07 but has not been rebuilded since,

Trying to build today failed with:

In file included from tests.cc:6:
CharsetConverter.h:20:90: error: ISO C++17 does not allow dynamic exception specifications
   20 |                         CharsetConverter(const std::string &from, const std::string &to) throw (Exception);
      |                                                                                          ^~~~~
In file included from CharsetConverter.cc:5:
CharsetConverter.h:20:90: error: ISO C++17 does not allow dynamic exception specifications
   20 |                         CharsetConverter(const std::string &from, const std::string &to) throw (Exception);
      |                                                                                          ^~~~~
CharsetConverter.cc:10:74: error: ISO C++17 does not allow dynamic exception specifications
   10 | CharsetConverter::CharsetConverter(const string &from, const string &to) throw (Exception)
      |                                                                          ^~~~~
make[2]: *** [Makefile:728: libhtmlcxx_la-CharsetConverter.lo] Errore 1

The Arch pkgbuild don't use patches at the moment, however this error will be caught sooner or later during reproducibility rebuilds it use LDFLAGS="$LDFLAGS -Wl,--no-as-needed" which isn't a default

https://github.com/archlinux/svntogit-community/blob/packages/htmlcxx/trunk/PKGBUILD

https://github.com/archlinux/svntogit-community/tree/packages/htmlcxx/trunk

[shakeyourbunny]

HTMLCXX is buildable from source with CentOS 7,8 or a derivative of it, although you have to do a minor edit.

The current binary packages situation can be found here: https://pkgs.org/download/htmlcxx-devel

It looks like that - if you wanna have prebuilt packages - you have to resort to a BSD flavor, Fedora, OpenSUSE or Mandriva. The Debian family (alongside Ubuntu-flavored) are out of luck, there are no more maintainers more there.

[shakeyourbunny]

@ssokolow regarding building the flatpak: I'd suggest that you just use a fork of CentOS 8 (AlmaLinux, Rocky, Oracle Linux) for building the flatpak instead of Debian to save you headaches or a distro which still has packages and shows signs that it will still build the packages.

The options - taken this into consideration - from pkgs.org are:

• Fedora
• OpenSUSE (use Tumbleweed)

If you wanna compile the stuff with my instruction gist, don't use CentOS 7 any more (due to the ancient openssl libraries)

[shakeyourbunny]

or just adapt my lgogdownloader build script, where this emits a binary for lgodownloader from the most recent git source: https://github.com/shakeyourbunny/update-lgogdownloader

You can adapt it for building your flatpak, just include in the credits me.

[ssokolow]

Hello, you can find a patch to build htmlcxx here https://sourceforge.net/p/htmlcxx/patches/8/

Thanks. I'll try to find time to give it a try within the next day or two.

I'd suggest that you just use a fork of CentOS 8 (AlmaLinux, Rocky, Oracle Linux) for building the flatpak instead of Debian to save you headaches or a distro which still has packages and shows signs that it will still build the packages.

That's not how Flatpak works. Regardless of what distro you build on, it builds using and against the Freedesktop SDK build container or one of its derivatives like the GNOME and KDE SDK build containers... and if you upload to Flathub, you do it by uploading the build manifest and then they build the package.

I believe Fedora offers their own build SDKs for Silverblue, but those are specific to Silverblue and any derivatives and you can't use them to build Flatpaks for other distros unless you want to piss your users off by asking them to add an entire separate package source and duplicate your runtime downloads.

It's like how you're supposed to use one of the manylinux Docker containers if you're building a Python extension with compiled components and want it on PyPI.

(It's all part of how they guarantee that, if it compiles and runs on your end, it'll run on their end, regardless of what distro they're using... it just assumes dependencies maintained enough that the build process doesn't bomb out in a modern build environment... or that you're vendoring them and patches to achieve that.)

or just adapt my lgogdownloader build script, where this emits a binary for lgodownloader from the most recent git source:

Thanks. I'll take a look at it if the aforementioned patch doesn't work or is too much hassle.

[ssokolow]

Sorry for the wait. Some unexpected higher-priority stuff came up.

Hello, you can find a patch to build htmlcxx here https://sourceforge.net/p/htmlcxx/patches/8/

The patch applies cleanly but it's for a problem I wasn't having yet, since I hadn't upgraded from the 5.15 to 5.15-21.08 version of the KDE Flatpak runtime. It still dies at the linking stage with libfl.so: undefined reference toyylex'` if only that patch is applied.

it use LDFLAGS="$LDFLAGS -Wl,--no-as-needed" which isn't a default

Thanks for pointing that out. Adding "build-options" : { "ldflags": "-Wl,--no-as-needed" }, to the manifest allowed the htmlcxx module to successfully complete its build.

Also, huh. tinyxml2's README says "Generally speaking, the intent is that you simply include the tinyxml2.cpp and tinyxml2.h files in your project and build with your other source code." Am I correct in guessing that's not being done at the request of distro maintainers?

Anyway, it's late and I'm goint to bed, but I now have a Flatpak manifest that appears to work:

1. It gives the expected output when I run flatpak run com.github.Sude.lgogdownloader --help (UPDATE: and lgogdownloader --download --game "^jill_of_the_jungle_the_complete_trilogy$")
2. it produces a com.github.Sude.lgogdownloader launcher script in <system/user install prefix>/share/flatpak/exports/bin, meaning that it can be run as com.github.Sude.lgogdownloader if you've added that folder to your PATH

That leaves the following to be done:

1. Identify things like .a files that need to be added to the cleanup section of the manifest (Done.)
2. Verify that the 12.9MiB of /lib/debug files will automatically get split out into a supplementary .Debug package like the Flatpak documentation says they should. (Done as part of taking a screenshot.)
3. Change the placeholder App ID because they don't want you using com.github prefixes. Either io.github or a separate domain under your control. (Some apps do, but it's discouraged)
4. Test that it actually downloads games when I run more than --help (Done as part of taking a screenshot.)
5. Check that the Flatpak is obeying the XDG Base Directory Specification environment variables which Flatpak sets to redirect ~/.config/ and ~/.local/share into ~/.var/app/<app-id>/ for when filesystem=home or filesystem=host isn't specified. (Done as part of taking a screenshot.)
6. Find or create an Appstream XML definition for lgogdownloader to define what would appear on Flathub or in an app like GNOME Software or KDE Discover (Done, aside from taking a screenshot and adding an icon)
7. Add an icon to be used by the Appstream XML, since Flathub uses a file-browser-esque "grid of captioned icons" format for search results and KDE Discover uses a "Large icon with several rows of text beside it" format for software updates. (The latter provides a placeholder icon for things that don't define their own, but the former considers a custom icon to be mandatory, even for CLI applications where they waive the requirement for a .desktop file.)

8. Write a description that...

   1. ...explains that it's intended as an easy, cross-desktop way to get up-to-date versions of a reCAPTCHA-enabled (i.e. QtWebEngine-dependent) lgogdownloader build, but does not install a launcher and is meant to be used via something like alias lgogdownloader="flatpak run com.github.Sude.lgogdownloader" and a terminal. (Done)
   2. ...suggests that users who want to take advantage of sandboxing use Flatseal to narrow the filesystem=host permission to wherever they keep their downloaded GOG games. (Unless I decide to preconfigure it to download to the user's chosen xdg-download folder and use filesystem=xdg-download as the default sandbox instead.)

EDIT: Since I haven't gone to bed yet, a quick clarification:

It uses io.qt.qtwebengine.BaseApp, so the ~253MiB of QtWebEngine will be deduplicated with any other Flatpak packages also using the QtWebEngine BaseApp.

(Funny enough, ~118.2MiB of that is translation files and international dictionaries according to ncdu, with the dictionaries taking up ~97.8MiB of it. Exclude the QtWebEngine BaseApp, the debug files, and the stuff provided by the KDE Flatpak Runtime, and the total uncompressed size is ~8MiB. Not bad at all if the user already has something else pulling in QtWebEngine, given that everything but libcurl, zlib, and QtWebEngine has to be included in the Flatpak for being absent from the shared runtime... of course, there's still room for more savings since I'm not sure I counted all of QtWebEngine's dependencies in that estimation, which also come from its BaseApp, and the file-level deduplication could also deduplicate things like pieces of the bundled Boost.)

[ssokolow]

Verify that the 12.9MiB of /lib/debug files will automatically get split out into a supplementary .Debug package like the Flatpak documentation says they should.

I'm having trouble verifying this because I'm having trouble figuring out whether it's not happening or whether I'm not replicating far enough down the process that is involved in publishing a Flatpak package.

Never mind. It worked as expected when I did the full-blown "install to take a perfectly proper screenshot" process.

Change the placeholder App ID because they don't want you using com.github prefixes. Either io.github or a separate domain under your control. (Some apps do, but it's discouraged)

@Sude- Can I get your input on this? They prefer using at least a subdomain that you control.

Note that I have to use Sude rather than Sude- because otherwise you get something like this:

error: 'com.github.Sude-.lgogdownloader' is not a valid application name: Only last name segment can contain -

Also, which architectures do you consider to be supported for building LGOGDownloader? Flathub includes a build service and the metadata files account for that with support for specifying supported arches.

[ssokolow]

OK. Updated TODO now that I'm setting it aside for the evening:

• [ ] Get @Sude-'s feedback on what reverse-DNS App ID to use
• [ ] Decide where to host the screenshot (Flathub will cache it, but the Appstream XML still wants a publicly visible HTTP/HTTPS URL for the cache to retrieve it from. Note that the screenshot I made is Flatpak-specific, in that it shows an alias lgogdownloader="flatpak run com.github.Sude.lgogdownloader" line.)
• [ ] Get @Sude-'s feedback on the need for an icon to publish it in places like Flathub and, if necessary, start brainstorming to make some candidate submissions myself.
• [ ] Decide whether to incorporate a wrapper script and/or default config file for things like making --enable-login-gui Just Work™ and/or defaulting --directory to $(xdg-user-dir DOWNLOAD) so the default sandbox manifest can err on the side of security.
• [ ] Write one more paragraph for the Appstream XML description based on the decision of what default sandboxing policy to use. (i.e. Either advise security conscious users how to narrow the permissions or advise everyone else how to broaden them and change the default location.)
• [ ] Push the resulting set of files to GitHub Gist and get @Sude-'s opinion on if anything needs to be revised
• [ ] Re-read the Flatpak and Flathub guidelines to make sure I haven't overlooked anything
• [ ] Write a "preflight checklist" for anyone who wants to push a new release to Flathub once it's been published there.

[ssokolow]

Oh, to go back on the stated topic for this thread, it appears that LDFLAGS="$LDFLAGS -Wl,--no-as-needed" is how you get htmlcxx to build on toolchains that are too new to just work but not yet new enough to also need that patch, so mentioning that and the patch in the build docs should be a suitable solution to this issue.

Aside from that, it'd probably be best if I open another issue for the topic of a Flatpak release.

(I do actually have stuff ready to share. It's just that I was hoping to get an answer to "Get @Sude-'s feedback on what reverse-DNS App ID to use" before putting up a Gist because it's used for all the filenames as well as various IDs within the files.)

[ssokolow]

In order to make sure I don't forget, I've pushed the files I currently have to GitHub Gist and opened a specific feature request (#218) for the off-topic Flatpak stuff to finish pulling this back on topic.