Open Altonss opened 2 years ago
Sure, our small team would be happy to benefit from such contribution!
Personally I develop on Windows and my competency about Linux is close to zero. So I will make sure that Max, by far our most competent Linux guy, is aware of your proposal.
I know nothing about AppImages and Flatpaks. I think I remember that Oracle is to provide a general binary-packaging utility in some next Java release (perhaps based on jlink?). A track to explore. We just have a Windows-specific installer today, a more general approach would be interesting of course.
I made some progress to package Audiveris as a flatpak. For the moment it is using the build obtained using gradle and wraps this into a flatpak. That's not very clean, but at least it's a beginning. Should I share my recipe? I could also try something with the jar files, but that's more complicated as I would need gradle to run these...Or am I wrong?
Does Audiveris needs network access? Because I got some errors because of some lacking network access
Audiveris checks periodically for a potential new release. This is the only automatic network access I can think of (and you can change the polling period which is weekly by default). If there is no network, you should get just a warning like "Could not connect to Audiveris project".
Of course if you manually select, in the Help menu, items like Handbook, Wiki, Web Site, Check for Update, you will try to reach the information on the network.
Regarding flatpak, I can't help you. I will let Max answer your question if he can.
I've created a flatpak recipe here. I created a helper tool that allows downloading all Java dependencies and using them for a clean flatpak build using grade --offline
. I have created 4 branches for packaging 5.2.4, 5.2.5, and the current master
and development
branches, respectively. These branches contain all required dependencies, so they can be used for a simple flatpak-builder
invocation. The bootstrapping procedure for downloading the dependencies is explained in the README.
There are also two slim branches with no dependencies (scripts
and scripts-17
for openJDK11 and openJDK17, respectively).
Network access works in my package. But opening a browser to read the docs currently doesn't work, this seems to be a generic Java/flatpak issue.
@Altonss, I just discovered this issue here after I was already done with my work. I hope you're not upset. Perhaps we can make a joint effort to get this ready for prime-time?
@hbitteur, would you agree to submitting these packages to flathub?
Thanks a lot for your work @mwilck ! I already had tried to build the flatpak, and it kind of worked, but thanks for having made a clean version. Are all the functionnalities working? It would be sooo cool to be able to release some day on Flathub :)
This is still in it's early stages. I haven't checked if everything works (as I said, launching a browser does not). I've opened some sample music sheets and run OMR on it; I was quite satisfied with the results so far. I'd be very grateful if you could give my packages a try. You should be able to build them fairly easily (clone my repo, checkout branch b5.2.5
, and run flatpak-builder --repo /some/dir builddir org.audiveris.audivers.yml
).
I've created a short Wiki page explaining what the various branches are.
The web browser problem is solved now due to help I got in the openJDK17 issue.
@hbitteur: I'd appreciate a comment if you'd agree with submitting this to flathub, and if not, what you think I (we) should do beforehand. In particular, would you object me submitting this with the flatpak application ID org.audiveris.audiveris
?
@mwilck I have no objection to your submitting. And I'll be out for a couple of days so don't wait for other comments from my side, simply keep going as you intend.
I haven't submitted to flathub yet, but I've reworked the build scripts such that no dependencies need to be stored locally. Everything is now downloaded by flatpak-builder before gradle builds offline.
Any progress on this topic? We do need such Linux releases!
Sorry I haven't been able to follow up here. I suppose my previous packaging work could be brought back to work with relatively little effort. I will try to find some time to work on it soon.
FYI, back in 2022 when I worked on this, I got frustrated because Audiveris would always crash in unpredictable ways in the text recognition stage, typically with references to freed memory, which AFAICT should never happen in Java. I am certain that this wasn't caused by the flatpak, it also happened with native binaries, so maybe it was an issue with Tesseract, or possibly with my JRE (actually various different JREs I tried). I didn't want to submit something I knew was going to crash on non-trivial scan attempts.
I guess I should have opened in issue here, but I thought that this can't be a generic issue but must be caused in some ways by my Linux environment. I didn't make any progress with the bug analysis though, and eventually gave up, especially as the project for which I had intended to use Audiveris was finished.
So ... I had another look at my build environment for the flatpak, and to my surprise, I found that it still works smoothly on the lastest code. I have a branch for packaging Audiveris 5.3.1, and the development
branch has all the meta data for packaging current Audiveris development
. The other good news is that I couldn't reproduce the Tesseract crashes I'd previously seen.
So it seems that we're actually in quite decent shape. I'm sure there are still lots of issues (for example, the "About" page shows wrong versions, not idea why).
@hbitteur:
The flatpak organization prefers that the author of an application submits the package to Flathub. Maybe you could consider doing the submission yourself? The procedure is described here.
You need a fork and a clone of flathub itself, which must be populated with the necessary files to build the Audiveris package. I have created a sample flathub fork here ("audiveris" branch). It was created from my Audiveris-flatpak repo, branch b5.3.1
, using the script flathub-copy.sh
.[^gen]
AFAICS you could use my flathub fork for a first submission attempt of Audiveris 5.3.1 without changes. However, please review it again, in particular the org.audiveris.audiveris.metainfo.xml
file. If you find issues, or have any additional questions, contact me. Please also have a look at the Flathub submission requirements page, licensing etc.
Note that Flathub's requirements wrt licensing aren't overly strict. In my understanding, if we can package a Windows binary under some open-source license and distribute it, we should be able to publish on Flathub, too. But IANAL.
If you can't do this submission yourself for whatever reason, let me know, and I'll try to figure out how I can do it in your name.
[^gen]: The purpose of the mwilck/Audiveris-flatpak code is to create a build manifest for building Audiveris offline, as required by flatpak. The code that will eventually be submitted to Flathub is generated by the code in the mwilck/Audiveris-flatpak repo.
@mwilck
It's good news. Both for the Audiveris project and for all those who ask for easy installation on Linux.
We can add the fact that, thanks to recent PR from Peter Greth, Audiveris was able to replace JPodRenderer with Apache PDFBox. JPodRenderer still had a dependency on JAI (Java Advanced Imaging) which was not fully open-source. The road is now clear.
Today I went through the Flatpak and Flathub documentations that you pointed out in your post. I also read the 15 files contained in your Flathub fork. Should I say that I was far from understanding every line?
So, I think you, Martin, are the person who should take the lead on this Flatpak/Flathub submission. Indeed, I read that "some applications are distributed on Flathub without the involvement of their developers" and that Flathub "would prefer that these applications be controlled by their authors". But that's not our case. I think all Audiveris contributors, starting with me, would appreciate this submission.
This Audiveris project is open in all directions, all contributions are welcome. Baruch contributed with the Windows installer. This Linux distribution will be your contribution.
I will try to help you as much as I can. For example, I think we should aim for a new version (5.3.2), due to some bug fixes I'd like to include.
We'll get into more detail in the coming days, but I wanted to tell you that you should go for it. Thank you very much for your work. /Hervé
P.S. I need to resurrect my VirtualBox installation to get my hands on the Linux environment again
Flathub submission created. Let's see how it goes. @hbitteur please have a look and let me know if anything is not as you would like it.
@hbitteur, have you ever tried running Audiveris on other platforms than x86_64? The flatpak build fails for ARM because of missing tesseract libraries, and I wonder if it's worth the effort to try and make it build. For now I am just trying to disable the ARM build.
@mwilck No, I have never tried it myself. For what it's worth, have a look at issue #564
We made it! Audiveris is now on flathub. :rocket:
I suppose this issue can be closed.
@hbitteur, perhaps you want to add a note about this in your installation guide.
Sure, I need to do this! Let me try to run the installation on my PC.
Thanks a lot!
Thanks a lot @mwilck ! Once it is in the installation guide, maybe it would be a good thing to verify it on flathub https://docs.flathub.org/docs/for-app-authors/verification
I haven't double-checked, but I suppose verification can be only done for a single account. I can do this, but I'd leave the decision to @hbitteur whether he wants to link the app to his own account instead, which would increase credibility.
@Altonss, perhaps you could give the package a test?
About the verification: the process requires me create a file with an UUID-like token in a file on the audiveris.org web page (https://audiveris.org/.well-known/org.flathub.VerifiedApps.txt). I obviously can't do that myself. So @hbitteur, I'll need your cooperation for creating that file. Perhaps it's actually easier if you do this yourself. You can use github to identify to flathub.
(edited) Well it seems that audiveris.org doesn't currently do https at all. So I guess this needs more patience.
@Altonss, perhaps you could give the package a test?
Tried it, and so far everything worked fine!
Only issue is, by default Export book
exports to /home/username/.var/app/org.audiveris.audiveris/data/AudiverisLtd/audiveris/piece/piece.mvt1.mxl
, which isn't ideal for the end user I think (it would be better in documents for ex).
@hbitteur, perhaps you want to add a note about this in your installation guide.
Wrote first little documentation for the flatpak installation https://github.com/Audiveris/audiveris/pull/687.
Only issue is, by default
Export book
exports to/home/username/.var/app/org.audiveris.audiveris/data/AudiverisLtd/audiveris/piece/piece.mvt1.mxl
, which isn't ideal for the end user I think (it would be better in documents for ex).
I agree that this path isn't ideal. I have kept changes to Audiveris's own source code to an absolute minimum. The only commit affecting it is 1f326d4.
Let me try to clarify my role here: I will will work on keeping the flatpak up to date and improving the code for creating the flatpak package. I don't feel responsible for other issues such as this one, even if they only occur on Linux / in the flatpak package. I will be happy to help with such issues, but I don't know the sources well enough to commit myself to fixing them.
In the concrete case here, I guess we need some indicator to figure out whether the app is being compiled for a) Linux and b) a flatpak environment, and set some compilation defaults accordingly. I'll try to figure something out, but it may take a while. I hope @hbitteur will be willing to take patches to this effect if they are cleanly written to change defaults only for Linux/Flatpak.
That said, I set the permissions of the flatpak such that it has access to $HOME/Documents
and $HOME/Pictures
. You should be able to use these folders. The awkward paths under $HOME/.var/app
are part of the flatpak concept and should be familiar to most flatpak users.
About the verification: I would need @hbitteur's assistance.
I think we should stop here. Adding more side issues to the "make linux release" topic is confusing. Let's discuss the verification process here, all else should go into separate issues.
Hi guys,
I hadn't used VirtualBox since about 10 years... So, I spent half a day fighting with that beast: downloading VirtualBox 7, installing Ubuntu unattended, discovering that my (french) keyboard was stuck to Qwerty, that I could not open a terminal with CTRL+ALT+T... Then a lot of googling to read somewhere that I should NOT use the default unattended way if I wanted to have the opportunity to correctly manage my Azerty keyboard. So I reinstalled everything, this time keyboard was OK and I could open a terminal window. I then installed flatpak, and was able to install audiveris from flathub like a charm.
Now add a couple of hours after dinner to succeed in sharing folders between my Windows host and the Ubuntu guest. And voila, Audiveris now runs correctly.
As you can see, I have been able to store a resulting .mxl file. The target is similar to what you obtained.
I'll see tomorrow if and how we can target other folders. This is enough for today.
Only issue is, by default
Export book
exports to/home/username/.var/app/org.audiveris.audiveris/data/AudiverisLtd/audiveris/piece/piece.mvt1.mxl
, which isn't ideal for the end user I think (it would be better in documents for ex).
I agree with @mwilck: this has little to do with the main topic of this issue which is to provide a "Linux installer". I'd suggest we open a separate issue or discussion if needed on this question of target folder.
See the specific target folders article and the more general standard folders article in Audiveris handbook.
I then installed flatpak, and was able to install audiveris from flathub like a charm.
This is truly a significant success! Thank you all for this long-awaited achievement!
Just like we did with Windows installer, we now need to automate this publishing mechanism. Certainly by defining specific tasks in the Gradle script. We need to work with @mwilck on this.
I could not resist... Here below is Flathub web site today. Enjoy!
Just like we did with Windows installer, we now need to automate this publishing mechanism. Certainly by defining specific tasks in the Gradle script. We need to work with @mwilck on this.
Today, I tried to review and understand all the various pieces that Martin Wilck has worked upon to pave the way from Audiveris to Flathub. My motivation is double:
I first went through all the files, trying to understand their purpose together with the Flathub documentation in the other hand, then I stumbled on Martin's Audiveris-Flatpak README file which helped me a lot. I should have begun with it!
My understanding, please correct me if I am wrong, is that a submission to Flathub is implemented as a "pull request" on a forked Flathub dedicated project. And the PR must provide specifically populated files (a kind of manifest). The question is how to write these files and, even more importantly, how to keep them up-to-date with Audiveris evolutions. So that, for any new Audiveris release, we can easily publish the master source release on GitHub, the Windows installer, the documentation, etc ... and the Linux installer on Flathub.
Could we handle this (the process described in Martin's README file) via the build.gradle
file hosted in Audiveris GitHub project?
This is what we did for Windows installer, and for the schemas documentation as well (using a dedicated sub-project).
Problem: here we need to use tools like flatpak-builder
which, to my knowledge, need to be run on Linux.
Could someone infirm or confirm this possibility?
Minor suggestions for the Flatpak files:
desktop
:
lilypond
appears in the keywords.metainfo.xml
:
screenshots
element: we should provide better Audiveris images than these old ones.release
element for 5.4: we should remove this element (content is almost void and 5.4 is not released yet) and rewrite it when we are ready to release.* Instead of "Convert sheet music to **XML**", we should use "Convert sheet music to **MusicXML**". XML term is very broad and tells nothing about the contained information.
ok
* I don't understand why `lilypond` appears in the keywords.
Well ... because my original, subjective motivation to try out Audiveris had been to create lilypond files from printed sheet music (which requires an additional conversion step). I agree it's kind of misleading. At least other music typesetting programs like MuseScore should be mentioned, too.
I'm happy to change this to whatever you suggest.
* orp.audiveris.audiveris.`metainfo.xml`: * `screenshots` element: we should provide better Audiveris images than these old ones.
Yes, and it would be wonderful if we had reliable permalinks to up-to-date screen shots on audiveris.org.
* `release` element for 5.4: we should remove this element (content is almost void and 5.4 is not released yet) and rewrite it when we are ready to release.
Right, probably. This is one aspect of the flatpak submission / maintenance process I don't fully understand yet. In particular I am not sure if we could submit a 5.4 release to the flathub "beta" channel before it's officially released.
At least other music typesetting programs like MuseScore should be mentioned, too.
OK, let's mention Lilypond, Musescore and Finale
Yes, and it would be wonderful if we had reliable permalinks to up-to-date screen shots on audiveris.org.
I will take care of finding better images.
In particular I am not sure if we could submit a 5.4 release to the flathub "beta" channel before it's officially released.
I agree this is not very clear for me either.
In our case, we can consider that this very first submission to Flathub was a kind of beta-testing of our submission data and process.
I would suggest that the next Audiveris submission to Flathub will be tried only when the following conditions are both met:
Just like we did with Windows installer, we now need to automate this publishing mechanism. Certainly by defining specific tasks in the Gradle script. We need to work with @mwilck on this.
Could we handle this (the process described in Martin's README file) via the
build.gradle
file hosted in Audiveris GitHub project?
My plan is actually to try to merge my https://github.com/mwilck/Audiveris-flatpak stuff into the Audiveris code base by creating a PR moving all of it into the dev/flatpak
subdirectory. Perhaps after fixing some issues that have come up in the flathub review.
Integrating that into the gradle workflow would be the last step for me, because I (being a Linux person and not a Java person) have very little knowledge about gradle. So, help with gradle would be highly appreciated :grin:
This is what we did for Windows installer, and for the schemas documentation as well (using a dedicated sub-project). Problem: here we need to use tools like
flatpak-builder
which, to my knowledge, need to be run on Linux.
Right. I doubt flatpak-builder will ever run on Windows. But it might be possible to automate this using GitHub workflows, where using a Linux worker is no big issue. That should be the ultimate goal, once we have the manual flatpak creation working.
The current flatpak manifest for audiveris is generated from several parts. There is the (mostly static) .yml.in
file, the list of tesseract languages to include, and most importantly, the list of dependencies, which is generated by a script. In the flathub review, I was asked to move this list auto-generated sources into a separate file. I eventually understood how this works, this is the next step I intend to do. It's just a minor change to my scripting.
Note that the "static" part isn't completely static; for example the flatpak runtime version needs to be updated from time to time. Also, as @hbitteur already noted, some information in the appstream data (.metainfo.xml
), in particular the changelog, needs to be updated as well when a new release is attempted.
As you have seen, the current process runs the normal gradle build and collects the .jar
files and the meta data that gradle has downloaded, creating a source list for flatpak. Because flatpak will download all these sources into a "flat" :grin: directory, we also need the creategradlrepo.sh
script that will re-arrange them in a format the offline gradle build used by flatpak-builder
can use. I bet my left foot that someone could come up with a gradle recipe to create these two files directly, without this clumsy "run-gradle-and-examine-what-it-did" procedure. (The python script used this stackoverflow response as a starting point).
I would suggest that the next Audiveris submission to Flathub will be tried only when the following conditions are both met:
I think we should fix the keywords according to your suggestion (altough I don't think Finale supports Linux; thus unsure if it makes much sense mentioning it).
* We have converged between your ad-hoc repositories and Audiveris repository, * Audiveris 5.4 is officially released on Github (a matter of a couple of months, it is still in alpha status on development branch).
Sounds like a good plan. I am not sure about the full automation yet, but it's at least a worthwhile goal.
Right. I doubt flatpak-builder will ever run on Windows. But it might be possible to automate this using GitHub workflows, where using a Linux worker is no big issue.
Very good idea!
Note that the "static" part isn't completely static; for example the flatpak runtime version needs to be updated from time to time.
Yep. I noticed that flathub supports Java LTS versions 8, 11 and 17. But not Java 21 yet, even though it's the latest LTS Java version today. I had the intention to move to Java 21 very soon, at least on the development branch. We may have to wait.
I don't think Finale supports Linux
I'm afraid you are right. I just checked on the Internet.
@mwilck Hi Martin, I just redesigned the Gradle sub-project that builds the Windows installer. I also did something similar to automate the generation of Audiveris handbook as a PDF.
Now I feel brave enough to start a new Gradle sub-project dedicated to your Linux installer. I will certainly need your help here and there, but let's go! :-) /Hervé
@mwilck I would like to get all your items from https://github.com/mwilck/Audiveris-flatpak into a dedicated folder within Audiveris Linux installer sub-project.
I plan to use build.gradle
file
And your material (perhaps modified for Gradle use) would be put into
How could we do this? Should I manually copy every item. Or is there a more official way to proceed (I'd like to keep the link to your work, so that you get the credit).
@mwilck I have cloned your repository https://github.com/mwilck/Audiveris-flatpak. Now, I'm trying to proceed manually as described in the README.md file, before wrapping the actions as Gradle tasks. I have faced a few problems:
languages.sh
uses several "curl -s ..." which fail for some security reason. I modified these calls as "curl -sk ..." and the script now works correctly. The file lang_sources.yml
got appended with new languages, however I noticed the insertion of a blank line between the old ones and the new ones. I could not fix this, I simply hope this has no impact.dummy.yml
, by the command:
flatpak-builder --force-clean --disable-cache --keep-build-dirs dummy dummy.yml
I confess that I don't understand why we need to build some dummy stuff, perhaps you could explain this.
Anyway, I installed flatpak-builder (on my virtual Ubuntu machine), I then ran this dummy build, which failed for lack of freedesktop, which I had to install (the obsolete 21.08 was required, perhaps I could change it for 23.08 in the .yml file).
Then I could see the flatpak-builder downloading gradle-7.3-all.zip, before issuing this error message:
Failed to download sources: module dummy: Error renaming temporary file: Text file busyAt this point I do need some help. What is this text file? And why is it busy?
I apologize for these manually typed messages, I still can't make any cut and paste working with my Ubuntu virtual machine. Life can get hard sometimes... :-)
@hbitteur,
sorry I missed these updates because I was sick and there is some confusion with email delivery from github to my various email accounts.
I suppose you've seen my PRs #704 and #705 by now. The dummy.yml
stuff is history now, as you may have seen.
languages.sh uses several "curl -s ..." which fail for some security reason
That's strange. Were you using some proxy or the like? Or maybe your Linux installation was missing the standard set of trusted root certificates? The package is typically called ca-certificates
.
@mwilck Sorry for this late answer. I had to address a couple of bug-fixes. I'm back to the flatpak/flathub topic today.
I have pulled your PR 704 in a local folder of mine to study its content before merging it into the development branch. In particular, I'm trying to understand what you are doing and how this could be more of less automated in Gradle.
I have to say I'd never heard of git submodules before I read your README.md recipe file. So, please be kind with my naive remarks.
Once pulled, to initialize the submodule, the recipe says we have to run:
git submodule init --all
The --all
option is not recognized by git. So, I removed it.
I expected to see git-managed items in folders like "dev" and then the resulting (git-ignored) artefacts written somewhere under the "build" folder. This does not seem to be the case.
For instance, the dev/flathub folder seems to gather files like the manifest org.audiveris.audiveris.yml
file, or the appstream meta data org.audiveris.audiveris.metainfo.xml
file with other files that are auto-generated (lang_sources.yml
, dependencies.yml
and mkgradlerepo.sh
).
I checked the .gitignore
file in the same dev/flathub folder (submodule?). It does not ignore the auto-generated files. Why?
In short, I think we should more clearly separate the input files and the output files. Only the input files should be managed by git, and all the output files should be "cleanable" by gradle.
Focusing on the input files of Audiveris project as a whole (that is not only the dev/flathub portion),I think we should avoid any information duplication between them. Here are some obvious examples (as of this writing):
These constants should be defined only once, typically as symbolic constants in the build.gradle
file.
Then any other input file should refer to the symbol name, not to its value.
For instance, "openjdk17" appears several times in org.audiveris.audiveris.yml
file. Is there a way in such a YAML file to include symbolic names? I don't know yet.
I have to say I'd never heard of git submodules before I read your README.md recipy file. So, please be kind with my naive remarks.
I know, git submodules are somewhat difficult to get used to. But when you have overcome the first parts of the learning curve, they can be very useful. If you're interested, have a look at the chapter in the git book.
I understand very well that if you are uncomfortable with git submodules, you don't want to start messing with them. But' I'd like to ask you to give them a fair chance :smile: You can work just normally with the audiveris repos without checking out the submodule, as long as you don't work on the flatpak.
git submodule init --all
The
--all
option is not recognized by git. So, I removed it.
My bad, sorry. It's just git submodule init; git submodule update
. --all
is just for git submodule deinit
, which removes the contents of the submodule directory.
I checked the
.gitignore
file in the same dev/flathub folder (submodule?). It does not ignore the auto-generated files. Why?
Because for flathub, these files are part of the submission. We must have them under version control (a flathub PR is nothing but a github PR, so all files that are part of the flathub submission must be in git). It doesn't matter that they are autogenerated from the audiveris code.
IOW, these file are ignored in the audiveris git repo, but not in the flathub repo. git is smart enough to understand that and not show them when you work in the main repo.
In short, I think we should more clearly separate the input files and the output files. Only the input files should be managed by git, and all the output files should be "cleanable" by gradle.
I can do it like this if you prefer. We could have dev/flathub
(or some other path that you might want) simply as an output directory under audiveris version control, git-ignored. But it has disadvantages compared with the submodule approach.
git -C dev/flathub status
shows us if anything has changed and needs to be committed.org.audiveris.audiveris.yml
and org.audiveris.audiveris.metainfo.xml
. They are only strictly required for flathub, but it would be kind of awkward not to have them in the audiveris repo. I guess with the discussion below about using variable substitution, this argument is resolved; we'd keep the template in the audiveris repo and the generated output in the flathub repo.Anyway, maybe we should move this discussion into #704 ?
... These constants should be defined only once, typically as symbolic constants in the
build.gradle
file. Then any other input file should refer to the symbol name, not to its value.
Yes of course. Would it be a problem for gradle to simply update the files in the dev/flathub
submodule directory, using some sort of text substitution?
For instance, "openjdk17" appears several times in
org.audiveris.audiveris.yml
file. Is there a way in such a YAML file to include symbolic names? I don't know yet.
In this specific case, I think the answer is "no". YAML-based languages support variables, like GitHub actions with it's ${{ variable }}
syntax, but it isn't standardized, and YAML itself doesn't support it. Flatpak's YAML is just plain YAML, unless I am mistaken.
We need to create a template and generate the .yml
file from it. As a Unix guy, I'd think of some sed script, but I guess there are equivalent and probably more powerful means available in the gradle ecosystem.
Hello, I just discovered the project and am very interested! The project seems amazing and I want to help bringing the software on linux. I saw we can already build it from source and that's already great. But It would be even better provide builds, for example through AppImages or Flatpaks :) If I find the time to help with this issue, I would be happy to contribute!