Closed wvengen closed 3 months ago
Sure. Contributions like this are very welcome. However, we would need to integrate this into our current CI process to have those flatpacks generated automatically.
Note that creating installers for Artisan is not trivial. For example, Artisan uses Qt's QWebEngine to render the HTML reports internally and generate PDF exports. Correctly bundling QWebEngine is non-trivial.
We wish the builds would be smaller (the universal2 builds for macOS are larger than 1.2GB)! However, there are limits. Artisan uses almost anything from Qt. Not much to keep off there (the macOS build script does a lot removal and still the packages are large).
Also the translations/locales add considerable to the size of the packages. Not sure if it makes sense to split them off. How would the correct locale be installed per user? Besides that some users use several locales at the same time (I have my locale in english, but use software often set to German).
Regarding the app id: the github home of Artisan has a dash in the name as well. Maybe you can/have-to escape those dashes?
Thank you for your response, and happy to hear you're open to this.
Note that the building happens on Flathub infrastructure, ideally based on source code for all components. CI could be useful to ensure it builds correctly, but is not strictly necessary. There is a KDE+Qt + PyQt basis, I will have a look how QtWebEngine integrates into this.
I think it would be helpful to have a way to install the application, without bundling it (i.e. letting the system integrator handle dependencies). Something a bit cleaner than what I have now. This could also be a shellscript, a setup.py
, or whatever. Possibly related: #831.
Translations/locales can either be bundled in the Flatpak, or be present as a kind of language-add-on packages (there is infrastructure for this). There can be multiple language packages installed, so that should be fine. update There is also a mechanism to configure the languages a user wants. It takes the system as default, and one can add extra languages for Flatpak applications (see cli doc). It looks like this is integrated with desktop environments, so language packs for the the user's default language should be installed automatically (though I didn't test this specifically). (Note to self: if we have more than locale files, we may run into https://github.com/flatpak/flatpak/issues/4450).
I'll ask Flathub for direction on what to do with the dashes. dash becomes underscore, see here.
I think the Flatpak packaging is almost finished. There are some things I need your help with, before I can submit this to Flathub (the required points are marked with "(*)").
Inclusion of an appstream metadata file in the artisan repository (see Flathub on Required Metadata). This file is used to show the app in appstores, across different distributions. It contains description, links to screenshots, relevant URLs, and information about releases.
If the appstream file could be included in the Artisan repository, that would be great (and somewhat of a requirement for Flathub). Note that this file needs to be updated with each release (or the new release cannot be released on Flathub). If this is tedious, I can help to think about how to automate updating this, e.g. from Github releases and the release history.
Note that the appstream file may also be useful to include in other Linux packages. For a Debian package, see Debian AppStream Guidelines. Note that the id
is meant to be equal across distributions (used e.g. when Artisan is available as a system package and a Flatpak, the software store will show this as one entry, allowing a user to pick which version to install). So this file could be installed as /usr/share/metainfo/org.artisan_scope.artisan.metainfo.xml
. This also means the desktop file in /usr/share/applications
should be renamed to org.artisan_scope.artisan.desktop
.
Perhaps this is something to consider later. It is mostly relevant when the deb package is included in a repository that software stores can recognize. I would just like you to know about this, to decide where to put the appstream file.
OARS is about age rating metadata, and there is one question where I did not have the answer for: is any user tracking happening (e.g. a user counter, error reporting, or something similar)? If no, then the social-info
line can be removed.
Note that the money-purchasing
line is present because donations are requested (and artisan.plus integration may also count).
Artisan is an open-source software that helps coffee roasters record, analyze, and control roast profiles. This software can automate the creation of roasting metrics to help make decisions that influence the final coffee flavor.
Notable features
The Artisan project runs on donations from individuals and companies recognising the value of Artisan for their work or leisure. Consider supporting this project with your donation, or even better, purchase an artisan.plus membership!
This version is a bug fix release.
IMPORTANT NOTE
The following issues have found in this release and are being worked on:
The Flatpak currently contains some lines to install the package. It does not use pyinstaller, also because some dependencies are already installed in the runtime and base application (it wouldn't make sense to bundle those).
This is what I currently do. I would love to have most of these steps as an installer script in the Artisan source tree: something that installs the app in a directory from which it is runnable, without bundling or dependencies.
I could try my hand at that, but would need to refactor the build scripts. Please let me know if you think that's a good idea.
Sidenote: one thing I encountered is a Python recursion error on import of snap7
. Moving this import to earlier in the process made this error disappear, that is the sed
line on artisan.py
for. Could not pinpoint the underlying cause though.
Is the icon available as SVG? That is preferred nowadays.
I added tyour appstream metadata file under debian/usr/share/metainfo and the app icon as artisan.svg. I also renamed the desktop file according to your instructions. Regarding OARS, the information on the page you link is a little loose. The Artisan app is counting the number of roasts a user does to show a dialog asking for a donation on a regular basis. However, this information is not transmitted anywhere and stays local. Then there is a method to send detailed debug log information via email back to us. But that mechanism is somewhat hidden and has to be triggered by the user. The app is also not sending that email automatically but just generates an email with the log files attached on request by the user. This email then still has to be manually send by the user to somewhere. I assume this does not count as user tracking which we do not plan to introduce at any point. So I removed that social-info line assuming my interpretation and evaluation of those actions is correct.
Regarding point 3 I would like to suggest that you make a start with this and send us a PR. This is not totally trivial as we build for different platforms on CI and we would love to keep this going;).
This snap7 thing is puzzling. There is not much specific about that lib or our use (but for the fact that they just made a major version step and changed some API and we still have to use the old version on some platforms for reason and thus need to support both APIs). That lib has a platform specific binary part as it is only a Python thin wrapper around the multi-platform native Siemens S7 protocol implementation known as snap7. You could get around this loop by importing it in the artisan.py main script. This is not ideal as we tried (only with partial success) to keep all Artisan specifics in the main.py package. Did you try to move this import higher up in main.py instead of adding it to the artisan.py script? What was the loop exactly about? By copying it to artisan.py you made it escape your compileall action as the artisan.py script is not compiled, right. Hm.
Could those build scripts for flatpacks/flathub also be independent of what we have right now?
One more thing: we are about to release Artisan v3 (new major version) next week or latest the week after. This could be the first release on Flathub. We could go for a "manual" flatpack build for this one and it could also be submitted a little after the regular v3 release.
Let us know what is possible!
Thanks for your replies, and for adding the appstream metadata file, and icon (the icon still contains embedded PNGs for something, I'll have a look if I can make it fully SVG-based).
Regarding OARS, I would judge the optional debugging info sent manually by mail not as user tracking, good you removed the line.
I'll have a look at the build scripts. In the meantime, I feel confident enough to submit this in its current form to Flathub, then it can hopefully be simplified later.
The snap7 thing appeared to be caused by the importlib monkey patch for MacOS, that recently seems to have been disabled (but not in the latest release 2.10.4, which I'm working with).
The build scripts for the Flatpak can indeed be independent of the rest. But I do see quite some duplication across the build scripts, and I would think it is hard to maintain a growing number of them, especially e.g. what is copied to dist/
. I think it could help other packagers to have a script that installs all relevant files to run from a certain location, without dependencies. (And even pyinstaller could start from that, to keep the process clear across platforms.) I'll see what I can do in a PR (later, after the Flatpak is published).
Great to hear about v3, I'll be happy to get that submitted to Flathub. Agree that we can keep the Flathub building aproach as it is now, and gradually improve (by moving some parts to build scripts part of artisan), if that turns out to make sense.
I will update the SVG with a clean one in my next commit. Sorry.
Right, this monkey patch was necessary as packaging on macOS failed. At least this one is resolved now. We constantly have to work around those things. Currently the main Python packaging lib introduced several complex incompatibilities one version just with py2app another one with pip/pyinstaller/numpy/scipy. Hard to find a combination of versions per platform which allows to bundle builds together:(
Thank you! No problem :)
Ah, that's why there are all these version number specifics in requirements.txt
. I imagine the number of files explodes when not bundling. Thanks for all the hard work!
I'm trying out v2.10.5-beta now, and notice that pydantic
is now a required dependency, also for running. Could we make this optional? It is a dependency that brings in no run-time functionality (as far as I understand), and it requires rust to compile from source, making the build more complex. "No" is also ok, then I'll start integrating rust (or use an older version of pydantic).
I have it working with pydantic, if that is the course of action. The total Flatpak size is 564 MB without and 585 MB 578 MB with pydantic (incl. dependencies). Note that cryptography 43 has also started using rust, so it I guess this will stick. Now making pydantic optional could shave off ~10 MB. Not a priority, perhaps, but good to know.
Pydantic is needed and we plan to make use of pydantic within Artisan in more places in the future. Thanks for your work thus I don't understand why those python packages cannot be just downloaded from pypip via pip as the Artisan build scripts do. In that case there seems to be no need for use rust to compile from source, right?
Thanks for your reply! Good to hear more use of pydantic is planned (for the code), and to know what to aim for. My reasoning was, that often, checks done on development are omitted in production builds, I would have expected pydantic to allow this, some kind of 'dummy' mode where these checks are omitted (unless we want more checks in production builds). But guess that would be more something for upstream.
The Flatpak packaging scripts prefer building things from source, I think both because having the source is generally more flexible / debuggable / reproducible, but also just because that's what the packaging scripts do right now (it is not enforced, however). One important trait of the Flatpak approach is that no network access is permitted during build, so all possibly used dependencies (for all architectures) need to be declared beforehand.
In any case, the issue is tackled.
We use (and plan to use more) pydantic to validate input on read into the app during runtime. Those check we want during runtime not to have to handle such "broken" data within the app.
For development checks we use the type checkers mypy and pyright as well as the linters pylint and ruff. Those type annotations used by the type checkers are indeed ignore by the Python runtime.
I understand this idea of working from source. However, many packages Artisan is build upon are non-trivial to build from source correctly, like PyQt, numpy, scipy and some others. The chance is high that, without that expert knowledge of those package owners, your builds are somewhat broken, at least on some platforms. Therefore other Python packages and apps like Artisan using those packages load the corresponding binary builds from pypip. Those packages can be retrieved before the "build" (assemble) on Flatpack to avoid any further network communication during the build. This is how the CI Artisan builds work too.
Thanks for the background, good to know!
The idea that building from source might lead to a broken package is a something I need to think about. PyQt has a Flatpak BaseApp to work with, so should be fine. Numpy, scipy and matplotlib, as well as dependencies like cryptography are widely built from source in Flatpaks. There are some packages that need native libraries, which I have added where known (when not shipped in the Python package). I'll take a critical look at them again.
Isn't it very inefficiency (energy wise) if each install compiles every component from source, besides that the install takes way longer?
Ah, that would be cumbersome and wasteful, yes. What is installed by the user is in binary form (using OSTree) (served by Flathub), the build process - running on Flathub systems for each app release - builds from sources.
Question on secret storage (for the plus integration, afaik): keyring comes with both SecretStorage
and libsecret
backends out of the box (the latter is required for Flatpak, for the sandboxing of secret access to work properly). Artisan picks up a specific keyring backend in the code, is there a reason not to use keyring's default backend selection? That would make libsecret work automatically, and could clean up the Artisan code related to setting up the keyring (also giving users an option to override it in the environment, if needed).
If the backends need to be listed for pyinstaller, why not adding them to the hiddenimports in the spec file for each platform?
The automatic default backend selection did never work in builds (on some platforms not even on running from source). This seems to have changed. I just tested builds on macOS, Windows and Ubuntu and on all three platforms the backend selection seemed to worked without issues. Still a scary change as before this worked. I hope someone else can test this a bit. Note that the keyring is only used to manage the password for the access to artisan.plus. I did not test the selection of a different backend by using a configuration on Ubuntu to choose libsecret (whatever this is; I am not a Linux user) over the SecretStorage that seems to be used as default (if installed).
Ouch, that sounds like a bad situation. Good to hear it's working now, and yes, a little scray ... I will test on my Linux setup before the next release.
Even though documentation suggests that libsecret works with Flatpak for the sandboxed environment, I could not get it to work without adding permissions to access all secrets (it so happens that all applications I could find that work with passwords do this as well). And then the existing SecretStorage also works (the Python keyring packages actually prefers it). So no change needed here (it will probably at some point in the future, when sandboxing is improved).
I'll try to keep an eye on this in the future.
v3 released. What is the status on Flathub?
Thanks for the headsup! I'm on it, will do some testing, and create the PR on Flathub. Since a lot happens to actually install Artisan, I expect some questions, but think that in a week or so it could be available (depending a bit on how much work still is needed, according to the Flathub reviewer).
update I see the changes to the beta (which I've been testing) are not big, so I'll do minimal testing before submitting.
Ah, one thing: I'm missing the release info in the metainfo file. This is how the Flatpak finds out the version, so it is pretty important to add the release to this file for each release (sorted by release date, newest on top). Could you still add that? (Strictly speaking, this may be a new release, but I can also get the updated file from Github during the build, without doing a new release.)
Ah, I found a way to do this in the Flatpak repository, so no hurry I guess. You could add this diff, if you like.
ADDITIONS
+NEW HARDWARE SUPPORT
+CHANGES
+FIXES
+There's one thing I'm not fully happy with yet, which I noticed recently: when saving a file, the default directory you're saving to is not something the user recognizes, but a Flatpak-specific /run/user/...
directory, and the user needs to select a proper directory by himself. I'll need to look into how this can be made more user-friendly with Flatpak.
Related: https://github.com/flatpak/xdg-desktop-portal/issues/475, portal support in Qt, maybe https://github.com/flatpak/xdg-desktop-portal/issues/997.
update because of the autosave function, it could make sense to allow access to the user's homedir, and not only to files (and directories) specifically granted. It could be made to work with file sandboxing, but I'm not fully sure yet.
update the specific issue of the default directory being off, seems like it has been fixed by https://github.com/flatpak/xdg-desktop-portal/issues/982, and would work from xdg-desktop-portal 1.18 on.
update selecting directories should also work since xdg-desktop-portal 1.7.1 https://github.com/flatpak/xdg-desktop-portal/issues/200#issuecomment-604540237, and the autosave dialog does use the Qt dialog for selecting a directory.
update adding permission to access the Documents path, so autosave works now
update submitted to Flathub
Sorry for being late, I forgot to update that metainfo.xml. Did it now and corrected the release date (1.9 => 1.8).
Thanks for updating the metainfo file. I think the Flathub PR is ready to be merged, but is taking some time (perhaps because of the holiday season?). I'll refrain from making changes to the PR until merged (because it might need another review round if I do), and then I'll update the release date.
Something just happend. I got an invite to join flathub/org.artisan_scope.artisan and did so. Anything further I need to do?
The app is live on Flathub now! Correcting the release date in https://github.com/flathub/org.artisan_scope.artisan/pull/1. I'll merge that to correct the release date on Flathub.
Next step would be to get the app verified.
Working on verification...
I did add the verification token to the artisan-scope.org site and clicked "Verify" under the Flathub Developer Portal. It now shows "Unverify", but the Artisan entry on Flathub still shows Unverified. Maybe this needs a moment? There is also a node on the Artisan Flathub page which states "App data review pending". Some entries are ticked already, most are pending. Anything we have to do on this?
Thanks! The listing can take a few hours to propagate, sometimes. Let's see tomorrow.
And verified for me.
Cool! What a success. The linux download page already links to Flatpack.
Thanks a lot for all that time you spent on this! Wonderful contribution! We couldn't have done this on our own as we lack the required compentences. Hope you are also a user and maybe you'd like to join the development team or at least the Linux-dev/release team;)
Feel free to close this issue if you think this is done.
Thank you for your kind words. I'm a happy (though fairly fresh) user, and would be glad to be involved in Linux-dev/release areas of Artisan.
I think my next step would be looking at the installation and bundling process for the various platforms, for which I'll open a new issue.
We are releasing Artisan v3.0.2, a patch release, today. I did update the metainfo.xml in the repository and also added a screenshot of Artisan 3 running on linux to the repository.
Just received an email from the Flathub.Bot that the build fails. Not sure what triggered this build. Anyhow, I could not even figure out the reason why and what exactly failed from the provided log. Could you take a look?
Thanks!
Thank you for the new release, and adding a Linux screenshot. The Flathub package periodically checks if a new release has been made (with the external data checker information specified). This auto-created a PR in https://github.com/flathub/org.artisan_scope.artisan/pull/2 But we need more complex dependency handling, so auto-created PRs don't work here. I'll have a look at it this week.
Thanks for this information and taking care. There is no hurry on this, this release just fixes a build issue that breaks the new (non-essential) scheduler feature, makes some smaller adjustments and updates some libs. We could have gone with just a build update on Linux and Windows, but we decided that the performance increase of using the ruff-based bidi implementation and that of the new numpy/scipy major versions would be worse a minor release.
Seems to run fine. And nice to see the numpy major version bump (I'll do some more math-related testing then). The about screen is different, not sure why. Will look into it later.
Excellent news! I removed the list of Contributors from the About file as this one was not maintained and to simplify things. Financial contributions are listed in the corresponding release blog posts and code contributions are managed by GitHub.
In case you want or need to join the projects dev team on GitHub, let me know.
Numpy and scipy tests were fine, merged the Flathub PR. Would be happy to join the projects dev team. Thank you!
Artisan is packaged for Linux as AppImage, Debian package and RPM. It's great to have this. I would like to suggest to also offer a Flatpak package on Flathub, because:
I would be willing to work on this, and have a mostly working Flatpak already. Pending:
org.artisan_scope.artisan
(dash becomes underscore, see here)624MB539MBunpacked; note that the deb is 727MB unpacked including Qt) - remove build dependencies; symlinked fonts already present in base system; prune yoctupuce binary libraries; remove QtWebEngine devtools, webdriver and dictionaries;also removed babel and QtWebEngine locales for languages which Artisan is not translated intoOptionally split locales to separate packagesdoesn't do much yet for this app, skip for nowfinish-args
(do we need sounds? x11-fallback? ...)