No module named bfio | backsub error

[josenimo]

Dear @kbestak and @FloWuenne ,

I am trying to run an mcmicro fork of mine https://github.com/josenimo/mcmicro. It crashed starting background subtraction, it seems we are missing the bfio module from aicsimageio.readers.bfio_reader.OmeTiledTiffReader. Have you seen this before?

mcmicro was running the following from defaults.yml background: name: backsub container: ghcr.io/schapirolabor/background_subtraction version: v0.3.3

Here is the full exit message:

Workflow execution completed unsuccessfully
The exit status of the task that caused the workflow execution to fail was: 137

Error executing process > 'background:backsub (1)'

Caused by:
  Process `background:backsub (1)` terminated with an error exit status (137)

Command executed:

  python3 /background_subtraction/background_sub.py -o Whole_HNC_Sonja_backsub.ome.tif -mo ./markers_bs.csv -r Whole_HNC_Sonja.ome.tif -m markers.csv

Command exit status:
  137

Command output:
  (empty)

Command error:
  INFO:    Environment variable SINGULARITYENV_TMP is set, but APPTAINERENV_TMP is preferred
  INFO:    Environment variable SINGULARITYENV_TMPDIR is set, but APPTAINERENV_TMPDIR is preferred
  INFO:    Environment variable SINGULARITYENV_NXF_DEBUG is set, but APPTAINERENV_NXF_DEBUG is preferred
  Attempted file (Whole_HNC_Sonja.ome.tif) load with reader: aicsimageio.readers.bfio_reader.OmeTiledTiffReader failed with error: No module named 'bfio'
  /opt/conda/lib/python3.9/site-packages/ome_types/_convenience.py:105: FutureWarning: The default XML parser will be changing from 'xmlschema' to 'lxml' in version 0.4.0.  To silence this warning, please provide the `parser` argument, specifying either 'lxml' (to opt into the new behavior), or'xmlschema' (to retain the old behavior).
    d = to_dict(os.fspath(xml), parser=parser, validate=validate)
  .command.sh: line 2:    37 Killed                  python3 /background_subtraction/background_sub.py -o Whole_HNC_Sonja_backsub.ome.tif -mo ./markers_bs.csv -r Whole_HNC_Sonja.ome.tif -m markers.csv

Work dir:
  /fast/AG_Coscia/Jose/Nextflow/Work/7f/53b1c68ed01ed67ae9bb409d28f252

Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named `.command.sh`

Best, Jose

[kbestak]

Dear @josenimo

As written in the comment I'm linking, the 'bfio' module missing error doesn't affect background subtraction, as it just indicates that one reader did not work so another one was used.

Regarding why the process was terminated, it was likely due to not providing enough RAM. Currently, Backsub requires 2-3x memory as the input image size and I'm working on increasing the efficiency.

Best, Krešo

[mdposkus]

Hi @kbestak,

I also encountered this issue when working with large (>100GB) datasets. It would be impractical for us to purchase 100-200GB additional ram due to cost (our computer already has 128GB), so I plan to downsample these images, which is not preferable. Are there still plans on reducing the memory utilization? I am using this package via MCMICRO and this memory utilization is a bottleneck for analysis. Thank you for supporting this valuable tool!

Thank you, Matt

[kbestak]

Hi @mdposkus,

I completely understand the impracticality of the RAM usage required by Backsub currently. I am definitely planning on optimising the tool as soon as possible as I'm certain it can be more memory efficient. Ideally, this would happen in November. I agree downsampling should not be an option, and apologies for the tool being the bottleneck!

Best, Kreso

[kbestak]

Hi @josenimo and @mdposkus,

I've developed a new, RAM-efficient version of Backsub with v0.4.1 --> https://github.com/SchapiroLabor/Background_subtraction/pull/14.

I was able to process an 88GB large image with only 13GB of RAM so I hope it will suffice for your usecases!

Best, Kreso

[mdposkus]

Hi @kbestak,

Thank you for the update! I look forward testing it in the coming weeks and analyzing some full-resolution images. If the RAM usage scaling is similar for my image sets then this will perfectly suit our needs.

Best, Matt

[josenimo]

Hey @kbestak , wonderful news, you should update the MCMICRO config.nf :) will report back on its use

[kbestak]

Almost there ;) https://github.com/labsyspharm/mcmicro/pull/528