Can I allocate less memory for imagej than it is initially requested?

[tornikepirveli]

I am trying to integrate ThunderSTORM plug-in to automate some image analysis logic, so I won't have to go through 100s of files. I am currently using imagej with scyjava.

When I run

from scyjava import config, jimport
import imagej
ij = imagej.init('C:/Users/Desktop/Thunderstorm/Fiji.app', mode='interactive')

def mem(gb):
    config.add_option(f"-Xmx{gb}g")
    Runtime = jimport("java.lang.Runtime")
    maxmem = Runtime.getRuntime().maxMemory()
    mb = maxmem // (2**20)
    mib = maxmem // (10**6)
    percent = 100 * mb // (1024 * gb)
    print(f"{mb} MB ({mib} MiB) available: {percent}% of requested")

mem(12)

(code provided by @ctrueden)

than it prints out: 3580 MB (3754 MiB) available: 29% of requested (I have 16 GB of RAM)

So I assume Fiji takes most of my RAM, since most of my memory for is available for scyjava when I don't use imagej. (There are files that are about 7GB or more in my case)

As I understand I need imagej for loading images and I need java runtime environment for its analysis. I tried lowering the RAM in fiji from 12GB -> 8GB but the result stayed the same.

Is there a way where I can maximize my JVM runtime memory in order to run analysis with ij.IJ.run(...) for big files?

[ctrueden]

@tornikepirveli Nope, something is wrong here. If you pass -Xmx12g, then the Runtime.getRuntime().maxMemory() should return a number close to that. The fact that you are seeing a percentage much less than 80% suggests that something is wrong with either your initialization steps, or your Java runtime. Based on your post above, I'm guessing the problem is that you are calling config.add_option("-Xmx12g") too late. You need to call this function before you do anything that starts up the JVM.

Here is how it looks on a macOS system of mine:

>>> from scyjava import config, jimport
>>> import imagej
>>> config.add_option(f"-Xmx12g")
>>> def mem(gb):
...     Runtime = jimport("java.lang.Runtime")
...     maxmem = Runtime.getRuntime().maxMemory()
...     mb = maxmem // (2**20)
...     mib = maxmem // (10**6)
...     percent = 100 * mb // (1024 * gb)
...     print(f"{mb} MB ({mib} MiB) available: {percent}% of requested")
...
>>> mem(12)
10923 MB (11453 MiB) available: 88% of requested

Try moving your config.add_option(f"-Xmx{gb}g") up to before you call imagej.init, replacing the {gb} with the actual number of gigabytes you want.

[tornikepirveli]

Thanks @ctrueden for replying in such a short notice! I actually tried that way, but after allocating some memory for Java runtime, the initialization fails. If I try first allocating memory for java it successfully allocates with 88% of requested data, but then it fails to import ImgLib2 helper classes. I am using interactive mode which starts java runtime environment on its own I believe and that is why I am unable to do it that way.

ij = imagej.init('C:/Users/User/Desktop/Fiji.app', mode='interactive')

after this command I get:

***Invalid initialization: ImageJ2 was not found***
   Please update your initialization call to include an ImageJ2 application or endpoint (e.g. net.imagej:imagej).
   NOTE: You MUST restart your python interpreter as Java can only be started once.

[tornikepirveli]

I am using plug-in in Fiji which is not standardly included in, so that is why I am using interactive mode, to run analysis and its other functionality. If there is some kind of workaround for it, please let me know :)

[NicoKiaru]

Maybe it's using a different jdk now that you've swapped your piece of code. Can you print the java version being used ?

from scyjava import jimport
System = jimport("java.lang.System")
print(str(System.getProperty("java.version")))

[tornikepirveli]

You are correct, It is using 1.8.0_312 but I have installed 11.0.16 on my system. When I run java --version on cmd it returns:

java 11.0.16 2022-07-19 LTS
Java(TM) SE Runtime Environment 18.9 (build 11.0.16+11-LTS-199)

Is there a problem with scyjava or what am I doing here wrong?

[NicoKiaru]

I guess it can be the problem. I'm not sure because Java 11 should work as well.

Are you using a conda environment ? Can you use openjdk-8 in it ?

I'm not expert, but with a similar problem I faced, and I had to force openjdk 8 in an environment file:

https://github.com/NicoKiaru/ABBA-Python/blob/main/environment.yml

[ctrueden]

I don't think Java 8 vs Java 11 is the problem. The "ImageJ2 was not found" error means that your Java environment was created successfully, but no imagej-2.9.0.jar (or whatever version) is present on your classpath.

Areas I would suggest investigating:

1. Is your path to Fiji.app correct?

2. Is your local Fiji.app fully up-to-date with all JAR files present and accounted for?

3. Are you using the newest versions of scyjava and pyimagej in your Python environment?

[tornikepirveli]

@ctrueden and how should I add or check whether my local imagej package is present on my classpath or not?

[ctrueden]

1. Is your path to Fiji.app correct?

>>> os.path.exists('C:/Users/Desktop/Thunderstorm/Fiji.app')
False
>>> os.path.exists('C:/Users/User/Desktop/Fiji.app')
True

And make sure the path you are passing to imagej.init is returning True there.

2. Is your local Fiji.app fully up-to-date with all JAR files present and accounted for?

Launch the ImageJ-win64.exe inside of that Fiji.app and run Help → Update... and make sure there are are no updates pending. If there are any updates, install them, restart Fiji, and check again, until there are no updates left.

Relatedly: which update sites do you have enabled for this local Fiji.app?

3. Are you using the newest versions of scyjava and pyimagej in your Python environment?

>>> import scyjava
>>> scyjava.__version__
'1.6.0'
>>> import imagej
>>> imagej.__version__
'1.2.1'

If scyjava or pyimagej are older than the above versions, you should update them. E.g.:

mamba update pyimagej scyjava

Or

mamba update --update-all

Or via pip if you aren't using mamba/conda.

If all three of the above avenues check out for you, then I'm not sure why your environment wouldn't be finding ImageJ2 upon initialization...

[tornikepirveli]

@ctrueden First and second one checked out. but checking on scyjava I got an error on scyjava.__version__ -> AttributeError: module 'scyjava' has no attribute '__version__'

So what seems to be a problem here?

(P.S imagej version was '1.2.1')

Updated packages are: ImageJ Fiji Java-8

[ctrueden]

Looks like your scyjava is earlier than 1.6.0. Can you please try updating it?

[tornikepirveli]

@ctrueden You are correct, I had 1.5.1 version and just updated it to 1.6.0 but the code outputs remains the same:

from scyjava import config, jimport
import imagej

def mem(gb):
    config.add_option(f"-Xmx{gb}g")
    Runtime = jimport("java.lang.Runtime")
    maxmem = Runtime.getRuntime().maxMemory()
    mb = maxmem // (2**20)
    mib = maxmem // (10**6)
    percent = 100 * mb // (1024 * gb)
    print(f"{mb} MB ({mib} MiB) available: {percent}% of requested")

mem(12)

10923 MB (11453 MiB) available: 88% of requested

System = jimport("java.lang.System")
print(str(System.getProperty("java.version"))) 

1.8.0_312

ij = imagej.init('C:/Users/Desktop/Thunderstorm/Fiji.app', mode='interactive')

Invalid initialization: ImageJ2 was not found Please update your initialization call to include an ImageJ2 application or endpoint (e.g. net.imagej:imagej). NOTE: You MUST restart your python interpreter as Java can only be started once.

[tornikepirveli]

Maybe I need specifically imagej2 and not fiji?

But as I remember Fiji was just an improvement of Imagej2 so I don't get what is a problem here.

[ctrueden]

@tornikepirveli Ah! Thank you for sharing the additional code. The problem is that you are using jimport before calling imagej.init. You must call imagej.init before the JVM is started, or else you will get a JVM without the ImageJ2 bits.

Here is a script that should work for you:

from scyjava import config, jimport
import imagej

# Set the JVM memory size.
gb = 12
config.add_option(f"-Xmx{gb}g")

# Initialize ImageJ2 and the JVM, wrapping local Fiji installation.
ij = imagej.init('C:/Users/Desktop/Thunderstorm/Fiji.app', mode='interactive')

# Print out memory statistics.
Runtime = jimport("java.lang.Runtime")
maxmem = Runtime.getRuntime().maxMemory()
mb = maxmem // (2**20)
mib = maxmem // (10**6)
percent = 100 * mb // (1024 * gb)
print(f"{mb} MB ({mib} MiB) available: {percent}% of requested")

# Print out Java version.
System = jimport("java.lang.System")
print(str(System.getProperty("java.version"))) 

# Print out ImageJ2 version.
print(ij.getVersion())

[tornikepirveli]

Thanks again for the insight!

It solved the problem 👍

The output was:

10923 MB (11453 MiB) available: 88% of requested
1.8.0_312
2.9.0/1.53t