search

BIOP / biop-bash-scripts

Bash scripts to simplify your life (let's hope so!)
9 stars 8 forks source link

Make QuPath extension installer not depend on Fiji #2

Open NicoKiaru opened 2 years ago

NicoKiaru commented 2 years ago

Currently installing an extension on QuPath require running a script on Fiji -> that's not optimal

hedjour commented 1 year ago

You can install QuPath extension just by mv or untar the jar extension file in the QuPath Extension Path. No need of Fidji script

hedjour commented 1 year ago

@NicoKiaru Can help you with this Issue ?

NicoKiaru commented 1 year ago

If you find a way, that'd be nice (I'm also pinging @petebankhead), but I'm not sure it's super obvious.

I'll try to summarize the goal of the installer, what's being achieved here, and what's not being achieved.

Goal

Here's the goal of the installer scripts:

What's achieved

Regarding point 2:

All this logic is probably pretty annoying to do in bash, or at least it's far above what I can do.

What's not achieved

Now there are some issues:

petebankhead commented 1 year ago

The QuPath extension folder is stored using java prefs. It's easy to set in Java, but not in bash, because Java Prefs are stored differently depending on the OS (windows registry on win, some hidden file on Mac, and probably the same in Linux).

Would the ability to set it from the command line help?

I was recently thinking of adding the ability to set the extension directory as a potential command line argument, although I'm not entirely sure how straightforward that will be...

If, in a QuPath extension, a jar is not needed anymore and becomes obsolete, it won't be removed by the script logic: there's no way to know if a jar is needed or not. (However if there are several versions of the same jar, it's possible to remove the old ones). It's not a big deal I think, but that's in contrast with Fiji which has a more advanced updater.

If I remember correctly, QuPath looks for extensions in subdirectories. If you aren't doing it already, could you put all your jars into a subdirectory and then just remove that when it's not needed?

I guess conflicts could happen, but they probably could anyway...

In any case, QuPath's whole extension loading mechanism needs a rethink. Hopefully v0.5.x will have some improvements, but nothing very major or advanced.

hedjour commented 1 year ago

The QuPath extension folder is stored using java prefs. It's easy to set in Java, but not in bash, because Java Prefs are >>stored differently depending on the OS (windows registry on win, some hidden file on Mac, and probably the same in Linux).

Would the ability to set it from the command line help?

I was recently thinking of adding the ability to set the extension directory as a potential command line argument, although I'm not >entirely sure how straightforward that will be... It will be usefull for person who installed QuPath by script and which have a restrictive antivirus which need all information in a specific path.

Could the QuPath extension follow a guidelines ? like make a sub-directory by extension in the QuPath Extension directory if different jar are needed ? or have is version number in it's name.

So we can make a script which analyse the QuPath subdir and replace a version by another ?

petebankhead commented 1 year ago

Could you not already do that with your own extensions, without anything needing to check on QuPath's side?

QuPath's extension mechanism isn't very advanced - I'm expecting that most will just be a single jar. I know the BIOP ones are more complicated, but I think you're doing more advanced stuff with QuPath extensions than anyone else I know :)

I agree QuPath ought to look for version info and try to use it when available though. That's potentially a small enough change to try to get into v0.5.0 if I find time to work on it. I think proper extension support would require QuPath to make use of ClassLoaders in a more advanced way... but that's a much bigger, longer-term change.

NicoKiaru commented 1 year ago

Changing the QuPath updater mechanism goes way beyond this issue I believe. So far it's ok for us, we need a few extra dependencies that hold in a zip, and as of now, to my knowledge, no conflict showed up.

After, if some conflicting situation show up, I guess it will time to dig into this nightmare of resolving dependencies and conflicts, and probably try to re-use what's existing instead of re-inventing a dependency solver (on the Fiji side, the dependencies declared in pom and the use of maven is doing fine, with a pretty large codebase, but that's also probably working thanks to @ctrueden and team constant efforts). Jgo may be worth a look as well (https://github.com/scijava/jgo)

Anyway, as far as this issue is concerned, I would go for:

@petebankhead when you write a CLI to set the extension folder, is it for setting the extension 'permanently' or just during the script execution time ?

NicoKiaru commented 1 year ago

Could the QuPath extension follow a guidelines ? like make a sub-directory by extension in the QuPath Extension directory if different jar are needed ? or have is version number in it's name.

So we can make a script which analyse the QuPath subdir and replace a version by another ?

I think the problem here is that there are common dependencies for different extensions. imglib2 and imglib2-realtransform are used for both Warpy and and ABBA. Having a flat hierarchy for jars simplifies jars version checking

NicoKiaru commented 1 year ago

(side note I'm happy so far that the extension we provide are pretty stable, so far the Warpy extension dates from 2021 and is working great, the ABBA extension bugfix and API improvements did not lead to users complaints since its initial release in 2021. That's also thanks to Pete would did not break anything from v0.3 to v0.4 ;-)

petebankhead commented 1 year ago

@petebankhead when you write a CLI to set the extension folder, is it for setting the extension 'permanently' or just during the script execution time ?

That's the bit that prompted

I'm not entirely sure how straightforward that will be...

I think it makes most sense to do it just for the current execution and make no permanent changes. What do you think?

Also, do you think it would be possible to set an option to set the extension folder with Java system wide prefs ? Does such a thing exists (system-prefs and not user prefs)?

I think it exists in Java, and sounds like it could also be a command line option in QuPath (which could also be provided 'permanently' by modifying the QuPath.cfg file).

I think the problem here is that there are common dependencies for different extensions. imglib2 and imglib2-realtransform are used for both Warpy and and ABBA. Having a flat hierarchy for jars simplifies jars version checking

Would it make things better or worse if these were bundled with QuPath itself? We're just starting to explore the use of imglib2 elsewhere in the software.

NicoKiaru commented 1 year ago

I think it makes most sense to do it just for the current execution and make no permanent changes. What do you think?

Yes, makes sense. So far I did not use QuPath with a CLI (except maybe for this issue), but I can see how it could help to specify a subset of extension during a script execution.

which could also be provided 'permanently' by modifying the QuPath.cfg file

Is it possible ?? This would be awesome. But I could not find a way to set java preferences using Java options. It's not written on the Preferences API (https://docs.oracle.com/javase/8/docs/api/java/util/prefs/Preferences.html). And googling this question do not return any OS-independent solution. Maybe I missed it.

With the current QuPath code, I do not think it's possible to set the extension folder for all users (https://github.com/qupath/qupath/blob/b3a59e880a9feef1b0b449189d72ffab3890e38e/qupath-gui-fx/src/main/java/qupath/lib/gui/prefs/PathPrefs.java#L816-L821)

Would it make things better or worse if these were bundled with QuPath itself? We're just starting to explore the use of imglib2 elsewhere in the software.

I think it's a neutral change (except for a transition period where I should avoid duplications).

Regarding setting preferences in command line, @romainGuiet wrote this groovy script and pref file to set the extension folder (https://github.com/BIOP/renku-helpers/tree/main/qupath) but again, it's not system wide. Also, if I go for this direction for the bash scripts, this means creating an xml file with bash.

hedjour commented 1 year ago

which could also be provided 'permanently' by modifying the QuPath.cfg file

Is it possible ?? This would be awesome. But I could not find a way to set java preferences using Java options. It's not written on >the Preferences API (https://docs.oracle.com/javase/8/docs/api/java/util/prefs/Preferences.html). And googling this question do >not return any OS-independent solution. >Maybe I missed it.

Is awk and gsub function installed with git bash ? If yes you can setup the cfgfile in a single line of code.

petebankhead commented 1 year ago

which could also be provided 'permanently' by modifying the QuPath.cfg file Is it possible ?? This would be awesome.

I was imagining it would require a core QuPath change to optionally use https://docs.oracle.com/en/java/javase/17/docs/api/java.prefs/java/util/prefs/Preferences.html#systemRoot() Although I don't know what exactly that does...

For now, the most relevant think you can do is to provide a custom preference location. The relevant code is at https://github.com/qupath/qupath/blob/b3a59e880a9feef1b0b449189d72ffab3890e38e/qupath-gui-fx/src/main/java/qupath/lib/gui/prefs/PathPrefs.java#L111

I think it would look something like this:

-Dqupath.prefs.name=my.custom.qupath.prefs"

Not sure if that helps with anything, but I've found it useful during development.