SaibotMagd
commented
2 years ago Interesting behavior. When I try to load a file it rename it:
Its a smaller one but still not working.
Open SaibotMagd opened 2 years ago
SaibotMagd
commented
2 years ago Interesting behavior. When I try to load a file it rename it:
Its a smaller one but still not working.
yoda-vid
commented
2 years ago Thanks for asking about this! Yeah memory shouldn't be an issue here because of memmap (and your computer's huge amount of RAM!). Are you trying to load the stitched.npy, 202.5 GB file, but then nothing loads? Was the stitched_image5d.npy text file created instead?
If stitched.npy was imported by another program, it may not work directly in mag because mag expects the NumPy array to be in t, z, y, x, [c] format and also looks for a metadata file (stitched_meta.yml). Does it work if you reimport the original file, which I'm assuming is stitched.tif? You could move aside the existing stitched_image5d.npy (to trigger a re-import) and run:
./run.py stitched.tif`This should import it to stitched_image5d.npy and stitched_meta.yml. Then you can launch it by ./run.py stitched (which looks for the _image5d.npy file) or ./run.py stitched_image5d.npy. Let me know if that works.
SaibotMagd
commented
2 years ago I tried this but it doesn't work:
2021-12-16 10:41:36,272 - root - INFO - Looking for files for multi-channel images matching the format: /scratch/mouse_201205/201205_Standard_mouse_4855_TOPRO-3_topro_ch_*.tif
2021-12-16 10:41:36,273 - root - INFO - Using the given single file /scratch/mouse_201205/201205_Standard_mouse_4855_TOPRO-3_topro.tif
2021-12-16 10:41:36,284 - root - INFO - Extracting metadata for image import, may take awhile...
2021-12-16 10:41:36,286 - magmap.io.importer - INFO - Starting Java for Bioformats using JAVA_HOME set to: /home/anaconda3/envs/mag
2021-12-16 10:42:07,207 - root - INFO - tag: {http://www.openmicroscopy.org/Schemas/OME/2015-01}Image
2021-12-16 10:42:07,208 - root - INFO - tag: {http://www.openmicroscopy.org/Schemas/SA/2015-01}StructuredAnnotations
2021-12-16 10:42:07,208 - root - INFO - names: ['201205_Standard_mouse_4855_TOPRO-3_topro.tif']
2021-12-16 10:42:07,208 - root - INFO - sizes: [(2581, 1, 7578, 5735, 1)]
2021-12-16 10:42:07,208 - root - INFO - resolutions: [(1.0, -6979350000.0, -6979320000.0)]
2021-12-16 10:42:07,208 - root - INFO - zoom: 1, magnification: 1
2021-12-16 10:42:07,208 - root - INFO - pixel_type: uint16
2021-12-16 10:43:59,323 - root - INFO - filename: /scratch/mouse_201205/201205_Standard_mouse_4855_TOPRO-3_topro
2021-12-16 10:43:59,324 - root - INFO - filename_base: /scratch/mouse_201205/201205_Standard_mouse_4855_TOPRO-3_topro
2021-12-16 10:43:59,324 - root - INFO - Initializing multiplane image import planes to "/scratch/mouse_201205/201205_Standard_mouse_4855_TOPRO-3_topro_image5d.npy", may take awhile...
2021-12-16 10:43:59,326 - root - INFO - Loading file /scratch/mouse_201205/201205_Standard_mouse_4855_TOPRO-3_topro.tif for import
2021-12-16 10:43:59,327 - magmap.io.importer - INFO - Starting Java for Bioformats using JAVA_HOME set to: /home/anaconda3/envs/mag
2021-12-16 10:43:59,494 - root - INFO - loading planes from time 0, z 0, channel 0
2021-12-16 10:44:03,431 - root - INFO - setting image5d array for series 0 with shape: (1, 2581, 7578, 5735)
2021-12-16 10:44:04,242 - root - INFO - loading planes from time 0, z 1, channel 0
2021-12-16 10:44:04,262 - magmap.gui.import_threads - ERROR - Invalid Z index: 1/1
Traceback (most recent call last):
File "/home/toolboxes/magellanmapper/magmap/gui/import_threads.py", line 85, in run
fn_feedback=self.fn_feedback)
File "/home/toolboxes/magellanmapper/magmap/io/importer.py", line 995, in import_multiplane_images
series=series, rescale=False)
File "/home/anaconda3/envs/mag/lib/python3.6/site-packages/bioformats/formatreader.py", line 824, in read
index = self.rdr.getIndex(z,c,t)
File "/home/anaconda3/envs/mag/lib/python3.6/site-packages/javabridge/jutil.py", line 961, in method
result = call(self.o, name, sig, *args)
File "/home/anaconda3/envs/mag/lib/python3.6/site-packages/javabridge/jutil.py", line 892, in call
result = fn(*nice_args)
File "/home/anaconda3/envs/mag/lib/python3.6/site-packages/javabridge/jutil.py", line 859, in fn
raise JavaException(x)
javabridge.jutil.JavaException: Invalid Z index: 1/1But he still creates the image5d.npy file even if he can't load it afterwards (there's no .yml file). I think its an issue regarding to the z-t dimension. Sometimes, I can't find out when, imageJ saves the slices as timepoints so I have to retransform it, but I can't do this now because it costs to much time (at least 2 days), I try it again afterwards.
In the meantime I cutted a small 100vx³ block out of the dataset to check for the cell-detection in isolation. But the learning curve for the GUI and functions is so steep I can't handle it now, so I will do the cellcount using ClearMap2 as I did before. Time is running out and there's so much left to do.
yoda-vid
commented
2 years ago But he still creates the image5d.npy file even if he can't load it afterwards (there's no .yml file). I think its an issue regarding to the z-t dimension.
Thanks for posting this. Yeah I agree that it looks like something to do with the z- and t-axes. Is the shape of your image z = 2581, y = 7578, x = 5735? The image is being interpreted that way, though I am not sure why it triggers the invalid z index in Bio-Formats. Did your smaller block load ok?
But the learning curve for the GUI and functions is so steep I can't handle it now, so I will do the cellcount using ClearMap2 as I did before. Time is running out and there's so much left to do.
I completely understand. Makes good sense to use what you're familiar with. Always open to feedback on how to flatten the learning curve whenever you get the chance. Thanks for all your feedback thus far!
By the way, I am working on an update to load TIF files directly, without requiring import and also without requiring the whole image to be loaded. I hope this will help the setup/learning curve at least a little.
SaibotMagd
commented
2 years ago Thanks for posting this. Yeah I agree that it looks like something to do with the z- and t-axes. Is the shape of your image z = 2581, y = 7578, x = 5735? The image is being interpreted that way, though I am not sure why it triggers the invalid z index in Bio-Formats. Did your smaller block load ok?
Yes it did, also the detection seems ok, but I can't validate it because of GUI issues. Also for me all of the "settings" like "scale detection" (is says just 0 and I can't change it), ROI-editor: Filter, Bordered, Seg, Grid, MIP doesn't do anything. But the detection itself seems ok, but I can't validate them because I couldn't find out how to plot the detected cells onto the large top images (I can only see them in the smaller images below in the ROI-editor view). I don't know if this means anything but clearmap found 2400 Cells while mag only found 62, but its most likely a threshold issue.
By the way, I am working on an update to load TIF files directly, without requiring import and also without requiring the whole image to be loaded. I hope this will help the setup/learning curve at least a little.
That sounds like a good idea. For me, the most important challenge is the better handling of larger data sets. When I see that Imaris is able to display a dataset of over 500gb in 3D in less than 10s with their proprietary IMS format and scale it according to the zoom level, that's the benchmark (and the filesize is only 50% of the TIF). The same dataset can be exported by Imaris in 2 hours as Tif files, which takes ImageJ at least 4 days. I don't think the problem can be solved using Tifs, because as far as I know it is very difficult or impossible to load this format in parallel and piecewise (similar to memmap). Maybe you know napari and how they handle the problem (https://napari.org/tutorials/applications/dask.html)?
yoda-vid
commented
2 years ago Thanks for your feedback, as always!
Also for me all of the "settings" like "scale detection" (is says just 0 and I can't change it)
Yeah I realize this is confusing. The slider scales the sizes of blobs only in the 3D viewer and not in the ROI Editor.
ROI-editor: Filter, Bordered, Seg, Grid, MIP doesn't do anything
I couldn't find out how to plot the detected cells onto the large top images (I can only see them in the smaller images below in the ROI-editor view)
Currently, the blobs are only shown in the smaller images. But good to know they could be useful to see in the overview images as well! Will note this as a possible future feature.
For me, the most important challenge is the better handling of larger data sets. When I see that Imaris is able to display a dataset of over 500gb in 3D in less than 10s with their proprietary IMS format
I agree that handling large files on commodity hardware is key. Imaris has definitely made admirable strides! I am trying a simpler approach by using the Tifffile library, which has memmap support for at least some types of TIFs. I've opened a PR (#90) to memory map these types of TIFs such as those output by ImageJ and BigStitcher to bypass the import step at least in these cases. It's not nearly as advanced as pyramidal images or using Dask (thanks for the link, by the way!) but can hopefully serve as a drop-in replacement for importing these files.
I try to load my example dataset (280GB) into mag using the roi parameter. But it won't load. I also tried to load it using the GUI, it say "loading image" but does nothing at all (I checked the CPU and w/r load for my system, the process is on hold). I've got 512 GB Ram in the system so this shouldn't be an issue (but mag using memmap anyway). I will now cut it down and try again to find out the "maximal possible" filesize.
I already transformed it into the npy format: