Closed will-moore closed 1 year ago
Using --max_workers=1
doesn't seem to make a difference:
$ bioformats2raw WB.xml 9846152_mw1.zarr --max_workers=1 --debug > bf2raw_mw1.log
OpenJDK 64-Bit Server VM warning: You have loaded library /tmp/opencv_openpnp1435327528824061833/nu/pattern/opencv/linux/x86_64/libopencv_java342.so which might have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
Killed
$ find 9846152_mw1.zarr/ -type f | wc -l
18211
Seb: "from var/log/messages"
(base) [sbesson@pilot-zarr1-dev ~]$ sudo grep Out /var/log/messages
Jan 17 15:24:41 pilot-zarr1-dev kernel: Out of memory: Kill process 28742 (java) score 976 or sacrifice child
Jan 18 12:57:11 pilot-zarr1-dev kernel: Out of memory: Kill process 29765 (java) score 975 or sacrifice child
Dom: see https://docs.openmicroscopy.org/bio-formats/5.9.1/users/comlinetools/index.html#command-line-environment Trying...
(base) [wmoore@pilot-zarr1-dev idr0048]$ export BF_MAX_MEM=10g
(base) [wmoore@pilot-zarr1-dev idr0048]$ bioformats2raw WB.xml 9846152_mem10g.zarr --debug > bf2raw_mem10g.log
I wouldn't expect setting BF_MAX_MEM
to do anything here. That is specific to Bio-Formats command line tools like bfconvert
and showinf
; bioformats2raw does not recognize it. To change the heap size for bioformats2raw, use something like:
$ JAVA_OPTS="-Xmx10g" bioformats2raw ...
I was able to reproduce this problem after a while with 0.6.0, default memory settings, and 1 worker. For anyone else following along, note this is public data: https://downloads.openmicroscopy.org/images/BDV/idr0048/
I don't see anything obvious to suggest that this is fundamentally a bioformats2raw issue; the image dimensions are well within what we've successfully converted from other original data formats. I suspect that fixing this will require some updates to the BDV reader to reduce memory consumption during tile reading.
I don't see anything obvious to suggest that this is fundamentally a bioformats2raw issue;
Did you mean a bioformats issue as per your comment about the BDV reader?
Thanks @melissalinkert - trying JAVA_OPTS="-Xmx10g" bioformats2raw WB.xml 9846152_mem10g.zarr --debug > bf2raw_mem10g.log
...
I don't see anything obvious to suggest that this is fundamentally a bioformats2raw issue;
Did you mean a bioformats issue as per your comment about the BDV reader?
Sorry, what I meant is I think BDV reader is the place to investigate first. That means there may not be anything we can do here in the bioformats2raw repo until any BDV fixes are released and the Bio-Formats version number is ready to be bumped.
Hmmm - looks like that memory boost had no effect:
(base) [wmoore@pilot-zarr1-dev idr0048]$ JAVA_OPTS="-Xmx10g" bioformats2raw WB.xml 9846152_mem10g.zarr --debug > bf2raw_mem10g.log
OpenJDK 64-Bit Server VM warning: You have loaded library /tmp/opencv_openpnp600874666168285863/nu/pattern/opencv/linux/x86_64/libopencv_java342.so which might have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
Killed
(base) [wmoore@pilot-zarr1-dev idr0048]$ ls 9846152_mem10g.zarr/0/
0
(base) [wmoore@pilot-zarr1-dev idr0048]$ find 9846152_mem10g.zarr/ -type f | wc -l
18203
Closing here, since memory usage issues will need to be addressed in BDV reader (and I can't transfer the issue across organizations).
@will-moore could you open an issue on ome/bioformats
describing the problem? so it is captured
I'm trying to convert https://idr.openmicroscopy.org/webclient/?show=image-9846152 (requested at https://forum.image.sc/t/converting-other-idr-images-into-public-zarr/44025/13) to OME-NGFF.
Working on
pilot-zarr1-dev
I have downloaded the files to/data/idr0048
withChecked that the downloaded files match what's on IDR:
Running in a terminal screen, I see:
The 3 times I have tried this, I get about the same number of files generated:
$ cat 9846152.zarr/.zattrs
``` { "bioformats2raw.layout" : 3 }(base) [wmoore@pilot-zarr1-dev idr0048]$ cat 9846152.zarr/0/.zattrs { "multiscales" : [ { "metadata" : { "method" : "loci.common.image.SimpleImageScaler", "version" : "Bio-Formats 6.10.1" }, "axes" : [ { "name" : "t", "type" : "time" }, { "name" : "c", "type" : "channel" }, { "unit" : "pixel", "name" : "z", "type" : "space" }, { "unit" : "pixel", "name" : "y", "type" : "space" }, { "unit" : "pixel", "name" : "x", "type" : "space" } ], "name" : "P_t00000, W_s00_0", "datasets" : [ { "path" : "0", "coordinateTransformations" : [ { "scale" : [ 1.0, 1.0, 1.0, 1.0, 1.0 ], "type" : "scale" } ] }, { "path" : "1", "coordinateTransformations" : [ { "scale" : [ 1.0, 1.0, 1.0, 2.0, 2.0 ], "type" : "scale" } ] }, { "path" : "2", "coordinateTransformations" : [ { "scale" : [ 1.0, 1.0, 1.0, 4.0, 4.0 ], "type" : "scale" } ] }, { "path" : "3", "coordinateTransformations" : [ { "scale" : [ 1.0, 1.0, 1.0, 8.0, 8.0 ], "type" : "scale" } ] }, { "path" : "4", "coordinateTransformations" : [ { "scale" : [ 1.0, 1.0, 1.0, 16.0, 16.0 ], "type" : "scale" } ] }, { "path" : "5", "coordinateTransformations" : [ { "scale" : [ 1.0, 1.0, 1.0, 32.0, 32.0 ], "type" : "scale" } ] }, { "path" : "6", "coordinateTransformations" : [ { "scale" : [ 1.0, 1.0, 1.0, 64.0, 64.0 ], "type" : "scale" } ] }, { "path" : "7", "coordinateTransformations" : [ { "scale" : [ 1.0, 1.0, 1.0, 128.0, 128.0 ], "type" : "scale" } ] } ], "version" : "0.4" } ], "omero" : { "channels" : [ { "color" : "FF0000", "coefficient" : 1, "active" : true, "label" : "Channel 0", "window" : { "min" : 0.0, "max" : 65535.0, "start" : 0.0, "end" : 65535.0 }, "family" : "linear", "inverted" : false }, { "color" : "00FF00", "coefficient" : 1, "active" : true, "label" : "Channel 1", "window" : { "min" : 0.0, "max" : 65535.0, "start" : 0.0, "end" : 65535.0 }, "family" : "linear", "inverted" : false }, { "color" : "0000FF", "coefficient" : 1, "active" : true, "label" : "Channel 2", "window" : { "min" : 0.0, "max" : 65535.0, "start" : 0.0, "end" : 65535.0 }, "family" : "linear", "inverted" : false } ], "rdefs" : { "defaultT" : 0, "model" : "color", "defaultZ" : 45 } } } ```but only the highest resolution directory exists (and looks incomplete):
The log file looks like this (middle repeated section not included):
bf2raw.log
``` 2023-01-17 13:16:21,753 [main] DEBUG loci.formats.ClassList - Could not find loci.formats.in.URLReader java.lang.ClassNotFoundException: loci.formats.in.URLReader at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at loci.formats.ClassList.parseLine(ClassList.java:196) at loci.formats.ClassList.parseFile(ClassList.java:258) at loci.formats.ClassList.