Also make linux releases!

Altonss commented 2 years ago

Hello, I just discovered the project and am very interested! The project seems amazing and I want to help bringing the software on linux. I saw we can already build it from source and that's already great. But It would be even better provide builds, for example through AppImages or Flatpaks :) If I find the time to help with this issue, I would be happy to contribute!

hbitteur commented 9 months ago

Believe me, I had read the documentation about git submodules in git book. A really long and impressive page! Since you have the experience of a git submodule, and of its decoupling from the parent module, I trust you: let's use your approach to handle the flatpak/flathub work.

My contribution will be limited to the integration with gradle. The goal is to avoid as much as possible the duplication of information (because of the risk to miss some update). I agree that. for lack of official support of user variables, the idea of template files seems the way to go. More on this later.

hbitteur commented 9 months ago

If I understand correctly the various files in dev/flatpak and dev/flathub, the flatpak manifest (org.audiveris.audiveris.yml) would need variables instead of hard-coded constants for:

5.4-alpha (plus its commit ID -- where do we get this ID?)
openjdk17
gradle-7.3-all.zip (plus its sha256 -- where do we get it?)
What is not clear for me is the bunch of icon-xx.png files

The appstream meta data (org.audiveris.audiveris.metainfo.xml) contains one XML element per release. I suppose it was somehow copied from the Audiveris releases page on github. Manually? if so, it would not be a big deal to add a few lines for any new release.

I can't see anything in dev/flatpak/languages.sh and dev/flatpak/create-flatpak-dependencies.py files that should change from one release to the other. And both can be launched directly from a Gradle script.

So I think that, in the non-autogenerated files, only org.audiveris.audiveris.yml would need some template mechanism. Gradle is based on Groovy which already provides some template mechanism. To be investigated.

mwilck commented 9 months ago

If I understand correctly the various files in dev/flatpak and dev/flathub, the flatpak manifest (org.audiveris.audiveris.yml) would need variables instead of hard-coded constants for:
1. 5.4-alpha (plus its commit ID -- where do we get this ID?)

I made a hackish attempt in my flatpak GitHub workflow in #705. On GitHub, it's ${{ github.sha }}, in general we can use the output of git rev-list -n1 HEAD.

2. openjdk17

It would make sense to stick with this for the time being, IMO. Flathub's org.freedesktop.Sdk.Extension.openjdk package is openjdk 21 now, so we could try building against that. But if we do this, the openjdk version might silently change under us when FlatHub updates the package. In addition, FlatHub has org.freedesktop.Sdk.Extension.openjdk17 etc. for the LTS versions of OpenJDK:

OpenJDK SDK Extension   org.freedesktop.Sdk.Extension.openjdk   jdk-21.0.1+12   23.08
OpenJDK 11 SDK Extension    org.freedesktop.Sdk.Extension.openjdk11     23.08
OpenJDK 17 SDK Extension    org.freedesktop.Sdk.Extension.openjdk17     23.08
OpenJdk 8 Sdk extension org.freedesktop.Sdk.Extension.openjdk8      23.08

The Java version to use will be your decision. Not sure how you deal with this problem on Win/Mac – you will always have to make sure that the JRE matches what you built against, right? AFAICS, if you want to avoid nasty surprises, you have to stick with LTS versions for flatpak, meaning that this value will change vary rarely. The flathub readme for the non-versioined openjdk package says "OpenJDK 21 is the current latest version. This is not a long-term support (LTS) version and will be periodically updated as new JDKs are released.".

Note: we may also want to have the freedesktop runtime version (here, 23.08) in the template.

3. gradle-7.3-all.zip (plus its sha256 -- where do we get it?)

Simply using sha256sum I think. It might be available from the gradle download site as well (but AFAICS maven only supplies SHA1 hashes at this time).

4. What is not clear for me is the bunch of icon-xx.png files

Flatpak needs the application icon in certain sizes, that's all. Unless you remove some of the existing icons, you don't need to bother with this part. If we had an SVG icon, we can reduce the number of icons (see flathub requirements).

The appstream meta data (org.audiveris.audiveris.metainfo.xml) contains one XML element per release. I suppose it was somehow copied from the Audiveris releases page on github. Manually? if so, it would not be a big deal to add a few lines for any new release.

Yes, I did this manually so far.

I can't see anything in dev/flatpak/languages.sh and dev/flatpak/create-flatpak-dependencies.py files that should change from one release to the other. And both can be launched directly from a Gradle script.

I suppose both scripts will be slowly evolving over time. Note that languages.sh has an interactive mode. Btw, we may want to discuss which languages we want to ship by default – the current selection reflects my personal preferences. I suppose the selection should be the same as in the Win/Mac versions.

So I think that, in the non-autogenerated files, only org.audiveris.audiveris.yml would need some template mechanism. Gradle is based on Groovy which already provides some template mechanism. To be investigated.

Great!

IMO the end goal would be to replace create-flatpak-dependencies.py by some gradle-internal functionality. After all, the script simply scans the result of a gradle build run and gathers the dependencies that gradle has downloaded. It's certainly possible to simply make gradle print these dependencies and their sha1 hashes in some way, without acutally downloading them and running the build. If we can do that, it'd just be a matter of reformatting that output two times, for the dependencies.yml and the mkgradlerepo.sh output files.

hbitteur commented 9 months ago

Hi Martin,

Before we get into the improvements we are discussing here (variables, signatures, java 21, ...) I thought it was time to merge your two PRs (704 and 705) into the official Audiveris development branch. We could then proceed to small improvements on this starting base.

A few days ago I had already pulled your 704, using the commands indicated on the related PR page:

git checkout -b mwilck-flatpak-integration development git pull https://github.com/mwilck/audiveris.git flatpak-integration

I was thus able to review your files (in the 'mwilck-flatpak-integration' branch) and start this discussion.

This morning I decided to merge this PR, using the rest of the indicated commands as follows:

$ git checkout development
warning: unable to rmdir 'dev/flathub': Directory not empty           <== WARNING HERE ========
Switched to branch 'development'
Your branch is up to date with 'origin/development'.

>$ git merge --no-ff mwilck-flatpak-integration
Merge made by the 'ort' strategy.
 .gitignore                                       |   4 +
 .gitmodules                                      |   3 +
 dev/flathub                                      |   1 +
 dev/flatpak/README.md                            | 129 ++++++++++++++++++++++++++++++
 dev/flatpak/create-flatpak-dependencies.py       | 150 +++++++++++++++++++++++++++++++++++
 dev/flatpak/languages.sh                         | 182 +++++++++++++++++++++++++++++++++++++++++++
 dev/flatpak/org.audiveris.audiveris.desktop      |  10 ---
 dev/flatpak/org.audiveris.audiveris.metainfo.xml | 144 ----------------------------------
 8 files changed, 469 insertions(+), 154 deletions(-)
 create mode 100644 .gitmodules
 create mode 160000 dev/flathub
 create mode 100644 dev/flatpak/README.md
 create mode 100755 dev/flatpak/create-flatpak-dependencies.py
 create mode 100755 dev/flatpak/languages.sh
 delete mode 100644 dev/flatpak/org.audiveris.audiveris.desktop
 delete mode 100644 dev/flatpak/org.audiveris.audiveris.metainfo.xml

$ git status
On branch development
Your branch is ahead of 'origin/development' by 6 commits.
  (use "git push" to publish your local commits)

nothing to commit, working tree clean

$ git pull
remote: Enumerating objects: 214, done.
remote: Counting objects: 100% (214/214), done.
remote: Compressing objects: 100% (47/47), done.
remote: Total 214 (delta 160), reused 214 (delta 160), pack-reused 0
Receiving objects: 100% (214/214), 62.75 KiB | 845.00 KiB/s, done.
Resolving deltas: 100% (160/160), completed with 91 local objects.
From https://github.com/Audiveris/audiveris
   76a5f80f5..b7e02c25f  development -> origin/development
warning: unable to rmdir 'dev/flathub': Directory not empty          <== SAME WARNING HERE ========
Successfully rebased and updated refs/heads/development.

$ git status
On branch development
Your branch is ahead of 'origin/development' by 5 commits.
  (use "git push" to publish your local commits)

nothing to commit, working tree clean

$ git push origin development
Enumerating objects: 29, done.
Counting objects: 100% (29/29), done.
Delta compression using up to 8 threads
Compressing objects: 100% (25/25), done.
Writing objects: 100% (25/25), 8.92 KiB | 761.00 KiB/s, done.
Total 25 (delta 13), reused 0 (delta 0), pack-reused 0
remote: Resolving deltas: 100% (13/13), completed with 3 local objects.
To https://github.com/Audiveris/audiveris.git
   b7e02c25f..8280b604b  development -> development

$ git status
On branch development
Your branch is up to date with 'origin/development'.

nothing to commit, working tree clean

I think everything went fine, except the 2 warning lines about the dev/flathub directory not being empty. Perhaps because this directory is handled as a specific git 'submodule'? I don't know

Anyway, if we look at the PR #704 page, we can now read that:

This branch cannot be rebased due to conflicts

Which items are in conflict? If I open this project in my IDE, no conflict is shown. How can I safely complete this PR integration?

hbitteur commented 9 months ago

Also on PR #705 we can read:

This branch has conflicts that must be resolved Use the command line to resolve conflicts before continuing. Conflicting files dev/flathub

Perhaps I made a mistake by trying to integrate first PR #704 and then PR #705 Perhaps I should have tried to integrate PR #705 directly?

hbitteur commented 9 months ago

2. openjdk17
It would make sense to stick with this for the time being, IMO. Flathub's org.freedesktop.Sdk.Extension.openjdk package is openjdk 21 now, so we could try building against that. But if we do this, the openjdk version might silently change under us when FlatHub updates the package. In addition, FlatHub has org.freedesktop.Sdk.Extension.openjdk17 etc. for the LTS versions of OpenJDK:
OpenJDK SDK Extension org.freedesktop.Sdk.Extension.openjdk   jdk-21.0.1+12   23.08
OpenJDK 11 SDK Extension  org.freedesktop.Sdk.Extension.openjdk11     23.08
OpenJDK 17 SDK Extension  org.freedesktop.Sdk.Extension.openjdk17     23.08
OpenJdk 8 Sdk extension   org.freedesktop.Sdk.Extension.openjdk8      23.08
The Java version to use will be your decision. Not sure how you deal with this problem on Win/Mac – you will always have to make sure that the JRE matches what you built against, right? AFAICS, if you want to avoid nasty surprises, you have to stick with LTS versions for flatpak, meaning that this value will change vary rarely. The flathub readme for the non-versioined openjdk package says "OpenJDK 21 is the current latest version. This is not a long-term support (LTS) version and will be periodically updated as new JDKs are released.".

According to Oracle itself, Java 21 is a LTS version. So I think we should now jump on Java 21, at least for the "development" branch, so that the next release be based on it.

hbitteur commented 9 months ago

I can't see anything in dev/flatpak/languages.sh and dev/flatpak/create-flatpak-dependencies.py files that should change from one release to the other. And both can be launched directly from a Gradle script.

I suppose both scripts will be slowly evolving over time. Note that languages.sh has an interactive mode. Btw, we may want to discuss which languages we want to ship by default – the current selection reflects my personal preferences. I suppose the selection should be the same as in the Win/Mac versions.

The predefined languages for the Windows installer are now copied from the directory windows-installer/dev/tessdata. As of this writing, this means:

deu.traineddata
eng.traineddata
fra.traineddata
ita.traineddata

This should be enough for a start. I have on the TODO list the ability for the Audiveris end-user to dynamically download additional languages directly from the Audiveris GUI.

mwilck commented 9 months ago

@hbitteur

$ git checkout development
warning: unable to rmdir 'dev/flathub': Directory not empty           <== WARNING HERE ========

You currently have a checked-out submodule in dev/flathub. In order to avoid the warning, you need to run git submodule deinit first.

$ git submodule deinit dev/flathub
$ git switch development
$ git merge --no-ff mwilck/flatpak-integration
$ git submodule status
-38977b6adb819de3becb8f54e36f78151278a005 dev/flathub

At this point, dev/flathub is still empty. To populate it, you need to run

$ git submodule init
Submodule 'flathub' (https://github.com/flathub/org.audiveris.Audiveris) registered for path 'dev/flathub'
$ git submodule update
Submodule path 'dev/flathub': checked out '38977b6adb819de3becb8f54e36f78151278a005'

Now if you want to merge #705, you have two options. You can do with or without the submodule checked out. If you merge while the submodule is checked out, you need to run git submodule update after the merge. This is because the flathub code has changed in #705. Your local code under dev/flathub is unchanged by the merge though, thus dev/flathub is shown as "modified", because it differs from the version registered in your HEAD commit. You could commit dev/flathub now, reverting the changes made by the merge (not recommended!). By running git submodule update, you bring the submodule in a consistent state with your HEAD commit.

$ git merge --no-ff -m 'merged flatpak workflow from mwilck' origin/flatpak-workflow 
Merge made by the 'ort' strategy.
 .github/workflows/flatpak.yml | 80 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 dev/flathub                   |  2 +-
 2 files changed, 81 insertions(+), 1 deletion(-)
 create mode 100644 .github/workflows/flatpak.yml
$ git status -s
 M dev/flathub
$ git submodule status
+38977b6adb819de3becb8f54e36f78151278a005 dev/flathub (remotes/origin/beta)
$ git submodule update
Submodule path 'dev/flathub': checked out '4219985d4c3bf86b9f4000fd69ee5b430fd515fd'

Merging might be less confusing if the submodule is not checked out:

$ git submodule deinit dev/flathub
$ git merge --no-ff -m 'merged flatpak workflow from mwilck'  mwilck/flatpak-workflow
Merge made by the 'ort' strategy.
 .github/workflows/flatpak.yml | 80 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 dev/flathub                   |  2 +-
 2 files changed, 81 insertions(+), 1 deletion(-)
 create mode 100644 .github/workflows/flatpak.yml
$ git status -s        # no output
$ git submodule status 
$ git submodule status
-4219985d4c3bf86b9f4000fd69ee5b430fd515fd dev/flathub

Obviously now you need to run git submodule init; git submodule update again if you want to use the dev/flathub stuff.

mwilck commented 9 months ago

According to Oracle itself, Java 21 is a LTS version. So I think we should now jump on Java 21, at least for the "development" branch, so that the next release be based on it.

There's actually a flathub issue about this, only 2 weeks old.

Probably we'll see an openjdk21 fork on FlatHub soon. If you want / need to migrate to Java 21, we can use openjdk for the time being, and switch to the new LTS release when it's available.

mwilck commented 9 months ago

* deu.traineddata
* eng.traineddata
* fra.traineddata
* ita.traineddata
This should be enough for a start.

Off-topic: I was using almost the same, but I had added latin, and de-frak which I hoped would be able to recognize German Gothic letters[^lang]. The result for the Gothic letters was pretty distrastrous though, I ended up removing the text part completely. :smile: [^lang]: I am interested in scanning old choral sheet music.

hbitteur commented 9 months ago

I am afraid I'm trying to run beyond my competencies about git. The problem is that I finally pushed the 704 merge to github See my commands above, which ended with:

$ git push origin development
Enumerating objects: 29, done.
Counting objects: 100% (29/29), done.
Delta compression using up to 8 threads
Compressing objects: 100% (25/25), done.
Writing objects: 100% (25/25), 8.92 KiB | 761.00 KiB/s, done.
Total 25 (delta 13), reused 0 (delta 0), pack-reused 0
remote: Resolving deltas: 100% (13/13), completed with 3 local objects.
To https://github.com/Audiveris/audiveris.git
   b7e02c25f..8280b604b  development -> development

$ git status
On branch development
Your branch is up to date with 'origin/development'.

nothing to commit, working tree clean

Now, I would like to make sure that Audiveris project is still OK on github. For information, my local dev/flathub contains:

$ ls -l dev/flathub
total 123
-rwxrw-r--+ 1 herve herve   983 Dec  9 20:47 0001-add-local-dependencies-mirror.patch
-rwxrw-r--+ 1 herve herve    63 Dec  9 20:47 add-tessdata-prefix.sed
-rwxrw-r--+ 1 herve herve 51435 Dec  9 20:47 dependencies.yml
-rwxrw-r--+ 1 herve herve    33 Dec  9 20:47 flathub.json
-rwxrw-r--+ 1 herve herve   935 Dec  9 20:47 lang_sources.yml
-rwxrw-r--+ 1 herve herve 46540 Dec  9 20:47 mkgradlerepo.sh
-rwxrw-r--+ 1 herve herve   289 Dec  9 20:47 org.audiveris.audiveris.desktop
-rwxrw-r--+ 1 herve herve  4986 Dec  9 20:47 org.audiveris.audiveris.metainfo.xml
-rwxrw-r--+ 1 herve herve  3196 Dec  9 20:47 org.audiveris.audiveris.yml

What should I do precisely? I suppose I don't need to do again something like:

$ git merge --no-ff -m 'merged flatpak workflow from mwilck' origin/flatpak-workflow

ADDED: I don't really care about my local projects, since I can still clone/pull them from github. The question is about the project on github.

mwilck commented 9 months ago

The development branch looks good to me. You successfully merged #704 into it. What exactly is your concern?

I suggest you clone the GH repo into a new directory without doing anything wrt the submodule, and run your usual build workflow to make sure everything works as usual.

hbitteur commented 9 months ago

The development branch looks good to me. You successfully merged #704 into it. What exactly is your concern?

My concern is the page about this PR #704. I can read this at the bottom of the page:

hbitteur commented 9 months ago

I just cloned the repo into a new directory of mine. Went fine. Can I now integrate PR #705 on top of it?

mwilck commented 9 months ago

I just cloned the repo into a new directory of mine. Went fine. Can I now integrate PR #705 on top of it?

I guess so. Note that it enables aarch64.

hbitteur commented 9 months ago

OK. Let's go!

mwilck commented 9 months ago

I just made a fresh clone of development (before #705) and ran ev/flatpak/create-flatpak-dependencies.py successfully. As expected, the submodule state was unchanged after running the script.

hbitteur commented 9 months ago

FYI, the current view on PR #705 ( look by yourself) exhibits similar warnings:

mwilck commented 9 months ago

I just verified that you can run dev/flatpak/create-flatpak-dependencies.py even if you don't have the dev/flathub submodule checked out. It works nicely. The only problem is that if you later initialize the submodule, the results of the run might be overwritten. Therefore checkout out the submodule before running the script is recommended.

You should actually be able to run dev/flatpak/create-flatpak-dependencies.py under Windows if you have python (although there may be detail issues as I never tested it).

I'll double-check #705.

mwilck commented 9 months ago

The merge is actually non-trivial, because development is not an ancestor of my flatpak-workflow branch.

To do it manually, you must have the submodule checked-out before (well maybe there's a trick to make git submodule update work during a merge with conflicts, but I don't know it).

$ git merge mwilck/flatpak-workflow
Failed to merge submodule dev/flathub (no merge base)
CONFLICT (submodule): Merge conflict in dev/flathub
Recursive merging with submodules currently only supports trivial cases.
Please manually handle the merging of each conflicted submodule.
This can be accomplished with the following steps:
 - go to submodule (dev/flathub), and either merge commit 4219985
   or update to an existing commit which has merged those changes
 - come back to superproject and run:

      git add dev/flathub

   to record the above merge or update
 - resolve any other conflicts in the superproject
 - commit the resulting index in the superproject
Automatic merge failed; fix conflicts and then commit the result.
## 421998 is the HEAD of the flathub repo in #705 
$ git -C dev/flathub/ reset --hard  421998
## the following resolves the conflict
$ git add dev/flathub/
$ git merge --continue

But don't do this now, I'll rebase my branch; that's easier for both of us.

mwilck commented 9 months ago

Rebase done, you should be able to do a clean merge now.

mwilck commented 9 months ago

Don't worry, you won't face this kind of difficulty regularly in the future. The problem here was that you had made unrelated changes before merging #704. Therefore the result of the merge was not an ancestor of #705, and thus, for git, they were two independent branches that had both added the dev/flathub submodule, with different HEADs.

hbitteur commented 9 months ago

I just merged your PR #705 and pushed. The page on #705 looks great now.

However page on PR #704 still has a warning. Could you have a look?

hbitteur commented 9 months ago

A bit off-topic, but since your knowledge of git is obviously better than mine, could you help me on this.

I often need to tell people that Audiveris development (and thus pull requests and commits) is to be made on the "development" branch and that the default "master" branch is to be used only for releases. I suppose you've read the Audiveris Wiki article on git workflow.

I have in mind to configure Audiveris git repository so that its default branch is set to "development" rather than "master". See this related git documentation. What do you think?

hbitteur commented 9 months ago

I tried to run dev/flatpak/create-flatpak-dependencies.py. I modified just one line (number 82) from

subprocess.call(["./gradlew", "-q", "-g", temp_home, "build"])

to:

subprocess.call(["cmd", "/c", "gradlew.bat", "-q", "-g", temp_home, "build"])

to be able to run in under Windows. It seems to went fine (building the project and its dependencies). In dev/flathub, as expected, two files were modified: dependencies.yml and mkgradlerepo.sh This sound good.

We have normal differences between old (Linux) and new (Windows) versions of leptonica and tesseract files:

linux-arm64 and linux-x86_64 (in old)
windows-x86_64 (in new)

mwilck commented 9 months ago

However page on PR #704 still has a warning. Could you have a look?

I don't know. I just closed it. Let's observe if this pattern repeats. I don't expect that it will.

mwilck commented 9 months ago

I have in mind to configure Audiveris git repository so that its default branch is set to "development" rather than "master". See this related git documentation. What do you think?

Sure, why not. It's up to you. It depends on what you want people to use primarily. If "development" is stable enough for those who use git, make it the default branch. A word of warning: This may have the side effect that the "master" branch may become stale sooner or later.

mwilck commented 9 months ago

I tried to run dev/flatpak/create-flatpak-dependencies.py. I modified just one line (number 82) from

subprocess.call(["./gradlew", "-q", "-g", temp_home, "build"])

to:

subprocess.call(["cmd", "/c", "gradlew.bat", "-q", "-g", temp_home, "build"])

to be able to run in under Windows.

cmd /c won't work under Linux, but we could make this call conditional on sys.platform.

Why have these dependencies disappeared between old(Linux) and new (Windows)?

Honestly I have no idea. Can the dependencies be platform-specific? Can you check if these files have been downloaded under dev/flatpak/gradle ?

hbitteur commented 9 months ago

Honestly I have no idea. Can the dependencies be platform-specific? Can you check if these files have been downloaded under dev/flatpak/gradle ?

I checked under dev/flatpak/gradle/caches, none of them has been downloaded. After all, this is consistent with the fact that they don't appear in the resulting dependencies.yml file.

Perhaps you should check the cache on your (Linux) side. Do these items exist there?

mwilck commented 9 months ago

Do these items exist there?

Yes. Attached is the log file gradle.zip of gradle -i build. You can see the various apacheXY.pom files being downloaded for example:

Downloading https://repo.maven.apache.org/maven2/org/apache/directory/studio/org.apache.commons.io/2.4/org.apache.commons.io-2.4.pom to /mnt/git/audiveris/dev/flatpak/gradle/.tmp/gradle_download11109180900187897999bin
Downloading https://repo.maven.apache.org/maven2/org/apache/apache/10/apache-10.pom to /mnt/git/audiveris/dev/flatpak/gradle/.tmp/gradle_download5550538240804300346bin
Downloading https://repo.maven.apache.org/maven2/org/apache/apache/19/apache-19.pom to /mnt/git/audiveris/dev/flatpak/gradle/.tmp/gradle_download6352660500421014997bin
Downloading https://repo.maven.apache.org/maven2/org/apache/apache/21/apache-21.pom to /mnt/git/audiveris/dev/flatpak/gradle/.tmp/gradle_download6610376582969617185bin
Downloading https://repo.maven.apache.org/maven2/org/apache/apache/23/apache-23.pom to /mnt/git/audiveris/dev/flatpak/gradle/.tmp/gradle_download1005783574676517840bin
Downloading https://repo.maven.apache.org/maven2/org/apache/apache/13/apache-13.pom to /mnt/git/audiveris/dev/flatpak/gradle/.tmp/gradle_download14076722763108493102bin
Downloading https://repo.maven.apache.org/maven2/org/apache/apache/18/apache-18.pom to /mnt/git/audiveris/dev/flatpak/gradle/.tmp/gradle_download12179872011685715451bin

So gradle behaves differently on Windows and Linux?

mwilck commented 9 months ago

FTR: The Java 21 LTS Flatpak has now been added to Flathub.

hbitteur commented 9 months ago

Good to know that. Thanks

Audiveris / audiveris

Also make linux releases! #534