BIOP / ijp-operetta-importer

PerkinElmer Operetta Importer using BioFormats!
GNU General Public License v3.0
8 stars 3 forks source link

Fiji Memory allocation #16

Closed arronsullivan closed 1 year ago

arronsullivan commented 2 years ago

Hello, first of all, I love this plugin it is very helpful!

I notice that with some import jobs it will hang forever and never process an image (tested up to 5hr). The example below shows the beginning log output. If you notice the 0.11GB per field, for 5 fields, for 62 wells is ~34GB and more than the allotted ~20GB, I think the issue stems from this?

Perhaps it just needs to chunk fewer wells if total memory is greater than Fiji allocated memory?

This is with the latest updates via Fiji update sites for Operetta Importer.

Parsing XML File... please wait
'XML Parser' took : 0.876315700 s
One Field of CZT image stack is 0.11 GB.
There are 5 fields in each well
And you have 20.68 GB of RAM
--->We will try to work on 62 wells in parallel and 10 extra threads to process your data

Thank you

NicoKiaru commented 2 years ago

Do you fuse fields ? Can you try to minimize the available memory of Fiji in Edit > Options > Memory & Threads (maybe set it to 2Gb to quickly test if the export starts)

NicoKiaru commented 2 years ago

Also, do you use DPC ? There's an issue with 32 bits tiff that may be generated with this sort of channel

arronsullivan commented 1 year ago

Hi, thanks for the reply, no fused fields and 16bit images only.

Testing with 2048MB: Plugin settings in BIOP_1 image Log and output file (empty at about 2 min running): BIOP_2 image Log and output file at about 7 min, Out of memory message but no file output: BIOP_3 image Interestingly it continues to try to process more images and gives another memory error about 15min in, still no output files: BIOP_4 image

BIOP_1 image BIOP_1_2GB_test BIOP_2 image BIOP_2_2GB_test BIOP_2 image BIOP_3_2GB_test BIOP_4 image BIOP_4_2GB_test

NicoKiaru commented 1 year ago

Sorry for replying so late.

It seems to me that you are not using this repository, but the old script that was made some time ago.

If you want to try the new plugin, you need to activate the Operetta Importer update site, then click Operetta Importer... (and not BIOP Operetta Importer):

image

Please let us know if you still have an issue with the new plugin

arronsullivan commented 1 year ago

Hi, Thanks so much! I apparently was updating the new one but using the old one like you say, I did not realize it moved out the Importers category. It seems to work as expected now, thanks again!