MorphoLibJ as a dependency versions conflict

romainGuiet commented 2 years ago

Hi @dlegland ,

I'm contacting you to know what is the best solution to add MorphoLibJ as a dependency to our EasyXT-Fiji.

I found in this repo releases the v1.4.3 , which I also found on maven.scijava.org but in Fiji MorphoLibJ (installed from the update site) is 1.4.3, which is only available as a SNAPSHOT on maven.scijava.org

Thank you for help,

Romain

dlegland commented 2 years ago

Hi,

both artifacts should correspond to the same release, I suspect there was some kind of automated generation of the artifacts (the "v" prefix is suggested by GitHub for tags, and we tag only releases). I would say that any of the two should give the same result.

I am not very aware of the full proces for automated generation (and there were recent changes in the CI system), maybe @iarganda has better insights?

We're planning a new release soon, so stay tuned....

David

NicoKiaru commented 2 years ago

What's weird is that it is not the same group: com.github.ijpb vs fr.inra.ijpb . This would cause some issues because the two jars can live together and have all their classes present as duplicates. If you can make a non-snaphot version of the same jar that is on the update site, we could also make a clean release for Fiji-EasyXT.

dlegland commented 2 years ago

Hi @romainGuiet and @NicoKiaru, I just released a version 1.5.0 for MorphoLibJ; I just checked that the release also exists on maven.scijava.org. So what I recommend is to switch the dependency of EasyXT-Fiji to this version, using the "fr.inra.ijpb" prefix.

The whole process is still not very clear to me, but at least there is a stable version you can refer to!

best, David

dlegland commented 2 years ago

I just added the dependency snippet to the readme file, this should make things more clear for other developers.

imagejan commented 2 years ago

@dlegland sorry for the late follow-up, but please always use the release-version.sh script to cut releases! This will create a version tag that is not part of the main branch, and will get picked up by the CI infrastructure to deploy a release version automatically to maven.scijava.org.

In addition, the script will warn you if 1) your license headers are not up to date, and 2) you depend on an outdated version of pom-scijava, which makes the whole release process and deployment into current Fiji a lot less error-prone.

/cc @ctrueden

dlegland commented 2 years ago

Thanks @imagejan for the info! It's fine for me to use the script;

I just have some (surely basic) usage questions: should I run on local machine (and if so, does it work also from Windows?), or is there some github system (maybe some kind of deployment workflow?) to make it part of the deployment process?

sorry, I'm conscious my skills in deployment automation are far from being up-to-date...

David

romainGuiet commented 2 years ago

@NicoKiaru made this documentation for us

dlegland commented 2 years ago

great, thank you!

imagejan commented 2 years ago

@romainGuiet thanks for pointing out your documentation. There's no way I would have found that without you pointing to it. The official documentation of the release process in SciJava is here: https://imagej.net/develop/releasing

So, @NicoKiaru, I suggest you update that page with everything you think it's missing, and try to link as much as possible from your own docs to this common SciJava docs. What do you think? It will be easier to maintain, and benefit a broader part of the community, in my opinion.

NicoKiaru commented 2 years ago

@imagejan yes I agree it would be nicer on the official doc, I wrote it on c4science because many parts are specific to the biop org (+ some windows specificities). It's also really hard to know the audience for such a documentation: do people know Maven ? Git ? Github ? Scijava ? Do they have a scijava pom on their project. Are they using intelliJ ? The pre-requisites are not obvious. For the biop, I know where people are and what they need.

ctrueden commented 2 years ago

@NicoKiaru Thanks for the list of SciJava-related BIOP repositories. I added them to my dotfiles mrconfig management (ctrueden/dotfiles@9d19c635d03f2dcb34c3275367b3bf0364cc5afb). I noticed that ijp-frc (by @lacan) is not on the list, probably because it does not use CI. Would it make sense to add CI for it? Or maybe include non-CI-enabled SciJava-related repositories as well on that page?

NicoKiaru commented 2 years ago

Thanks a lot @ctrueden! I've just CI-ed ijp-frc.

dlegland commented 1 year ago

Hello everyone (@romainGuiet, @NicoKiaru, @imagejan and @ctrueden)!

I just managed to release a new version of MorphoLibJ, but the process was rather complex, and I could not manage to release into maven central... Here are the workflow I followed, first for keeping track of it, but also to check if eveything is fine, and also to provide feedback from a "plugin developer" point of view.

I first tried to create a release "the old way" (by manually updating the pom, commit+push, and tag the version), but got a build error from the CI system. I therefore decided to try the release-version.sh script.
I followed the workflow provided by @NicoKiaru (thanks!!!). Except for the links specific to the facility, I found it more easy to understand than the official documentation... Maybe one reason is that I use Windows, and that official documentation seems to be Linux-oriented.
I realized I had to download an old jdk (namely a 1.8 version) in order to make the process run. Not complicated, but I found it strange it was not possible to use a "-version" tag with a more recent compiler?
I used the Git Bash and the release-version script. After several attempts (I realized I had to commit some changes generated by the script), I got a successful build, however I also got error message "The release preparation step failed -- look above for errors and fix them" while there was no error (the grep javadoc error resulted in empty output...). Snapshot just below. I suppose there is some warning bug in the script? I also used the "--skip-version-check" option, I left the pom update to a later release...
I pushed to GitHub, pushed also the tag for the new version, and made a release based on the newly created tag (yeah!)
I checked on maven: I could find the new "1.6.1-SNAPSHOT" version, but not the "1.6.0" one... I have no idea on how to push a release to maven central using the current settings.

My global feeling about this is that things are getting more (too much?) complex... I can understand the advantages of sharing conventions and uniformizing nexus repositories. But releasing a new version clearly requires additional skills and knowledge, meaning this requires time. And I fear this can refrain people for wanting to contribute. I will clearly not ask a student to try this.

Anyway, thanks all for the provided help and resources, and thanks for reading!

imagejan commented 1 year ago

Hi @dlegland,

I just ran into similar issues with the release-version.sh script on Git Bash just yesterday, but didn't have time to investigate (nor to report) yet. Similar to your case, I had no javadoc errors, but the release step seemed to have failed after pushing the tag, but without the release.properties file. That's the reason why the CI build for the release version failed (see here). I ended up trying with wsl (Windows subsystem for Linux) on my system, and the release (now with a new version number because of the existing GitHub tag for the previous version) succeeded.

Sorry to not be more helpful, I didn't manage to dig deeper yet.

(Maybe the useful parts of the workflow documentation by @NicoKiaru can be migrated to the imagej.net page to improve the official documentation?!)

NicoKiaru commented 1 year ago

Hi @imagejan and @dlegland ,

Maybe the useful parts of the workflow documentation by @NicoKiaru can be migrated to the imagej.net page to improve the official documentation?!

I can try, but it's a pretty big refactoring because what's in our documentation is split into several pages on the documentation wiki.

Quick question: did you 'git pull' the scijava-scripts repository ? Curtis made a commit in December that helps for debugging javadoc errors. If the build fail, you can see better the javadoc error message (at least in the github action build, but also maybe locally).

I realized I had to download an old jdk (namely a 1.8 version) in order to make the process run. Not complicated, but I found it strange it was not possible to use a "-version" tag with a more recent compiler?

Which Java is used in git bash is still a mystery to me. When typing which java in git bash I get this:

So jdk 14. I do not know how this one arrived there. But it does not cause any issue when releasing repositories targeting java 1.8.

Using a -version tag is complicated I think: if the right jdk is not there, what can you do ? Automatically download another one ?

That's the reason why the CI build for the release version failed (see here).

For @dlegland, I don't know if you know, but to get to the link provided by Jan, (the log of the github action build that failed), you need to click on the green badge, Build | passing on your github repo:

Then you can click on the failing build (the one used for the release, the snapshot went fine):

Click on build (2 times), and see you can see what went wrong:

Like @imagejan mentioned, it's a release.properties missing file.

I did not do any recent release from Windows, but I'll tell you if something fails on my side

My global feeling about this is that things are getting more (too much?) complex... I can understand the advantages of sharing conventions and uniformizing nexus repositories. But releasing a new version clearly requires additional skills and knowledge, meaning this requires time. And I fear this can refrain people for wanting to contribute. I will clearly not ask a student to try this.

Regarding maven central: I do not use it. I've heard it's notoriously difficult to get there. If it works with the release script, then it's fine, but if it does not work, then I usually don't care. The scijava maven repo is the one I use the most.

The cost of versioning is real, but it's really a strength of the scijava ecosystem. It's pretty easy to combine mavenified tools in a fiji plugin (and you really appreciate that once you've tried to combine a few python packages).

But that's right, the easier it gets, the wider the adoption. I'll try to contribute, share my windows experience, and try to fix issues or suggest improvements when possible.

dlegland commented 1 year ago

Thanks for quick replies!

I just ran into similar issues with the release-version.sh script on Git Bash just yesterday,

well, good to know I'm not the only one!

Quick question: did you 'git pull' the scijava-scripts repository ?

For me, yes, I forked the repository, and therefore used the last version.

For @dlegland, I don't know if you know, but to get to the link provided by Jan,

Yes, I start to get familiar with the results of GitHub builds (I started with MorphoLibJ!). For the build fail of the release, I could find there was a problem with a missing file, but assumed this had to do with the script, so I left at this point.

Well, it seems there is a release.properties file created during the process, but that is not pushed to GitHub. I will try to add it to repository and push it. But later...

imagejan commented 1 year ago

Well, it seems there is a release.properties file created during the process, but that is not pushed to GitHub. I will try to add it to repository and push it. But later...

Just to explain the logic of the steps that release-version.sh does (as far as I understand):

The script first runs a local maven build on your machine, ensuring that 1) the build passes, 2) the parent pom-scijava points to the latest release version, 3) the license headers are present and up to date in all files (and maybe a few other things)
Then, the target version is set, a release.properties file is created, a tag gets created with this state, and the tag gets pushed upstream.
Finally, the version gets bumped to the next SNAPSHOT in a new commit, and this commit (not containing the previously created tag) is pushed to the main branch. That's why the release tag is never part of the main branch history, but exists with its own release commit.

Here's an example from scijava-common (the master branch is the orange line on the left):

ctrueden commented 1 year ago

releasing a new version clearly requires additional skills and knowledge, meaning this requires time.

It's not supposed to require special skills or knowledge. I put in a lot of work to make it a one-liner:

release-version.sh

and that's it. It is supposed to be easier than the old double-push-to-master, not harder. If something goes wrong, the script tries very hard to tell you what went wrong, and how you can fix it. If it's still too complicated, I don't know how to make it simpler, sorry. [Edit: OK, I have one idea: improve the scijava-maven-plugin so that you can run mvn scijava:release and have it do everything currently done by release-version.sh, but in Java so it's platform-independent. But I think it would be a substantial amount of time and effort to implement, unfortunately.]

@dlegland You are not the first person to complain about this workflow. With every complaint, I try to improve the script to make the error messages better, but new ways of failure keep cropping up; it's amazing. This time: the MorphoLibJ_-1.6.0 tag somehow managed to not have the release.properties file committed, and I have no idea why not. I have done probably thousands of releases with this workflow and it's never happened to me. Maybe some difference in the Windows/git-bash behavior versus other platforms? I don't know. Therefore, I also don't know how to fix it. If someone who uses Windows is able to figure out why release.properties does not get committed to the tag when using git-bash, and can file a PR fixing it, I would appreciate it very much!

lacan commented 1 year ago

My two cents on this: Was there perhaps a change in IDE? Honestly since I moved to IntelliJ 2022.2.x, It tried to get creative with my .gitignore file sometimes. I'd maybe check there if some change to gitignore was done?

@ctrueden, I am constantly amazed at the release-version.sh file. While initally the learning curve was big, the amount of magic going on is amazing... It's also forced me to get serious with the way I do releases to avoid some common complaints. And the javadoc check! I am sure that without it, we'd still be claiming that 'the code writes itself''. So from an unusual channel, thank you so much!

dlegland commented 1 year ago

Hi,

Some (good) news: I could manage to create a tag release that was deployed to maven! I opened a Git Bash console, rewrote most commands one by one (I just skipped the controls, assuming they were OK from previous release...), and this time the missing "release.properties" file was correctly added. The new tag is 1.6.0.1, to distinguish it from the "real" release 1.6.0, but I think this is not a big trouble. I have no idea what went wrong yesterday. So everythings seems to be good!

Just to explain the logic of the steps that release-version.sh does (as far as I understand):

Thanks a lot, this helped understanding the whole process! and figuring out where to focus.

Just to clarify: I really appreciate all the efforts made to aggregate and simplify the use of the various plugins within the ecosystem. Making everything work within a single script, with many help messages, is a great advance, so many thanks for that! My purpose was to try pointing on what could be enhanced, from an occasional plugin developer perspective. Hopefully next release will be easier!

imagejan commented 1 year ago

@dlegland @ctrueden pinning to maven-release-plugin-3.0.0-M7 works around the issue on Git Bash.

See https://github.com/scijava/scijava-scripts/issues/49 and https://github.com/scijava/pom-scijava-base/pull/33.

ijpb / MorphoLibJ

MorphoLibJ as a dependency versions conflict #62