OME Zarr Converter Plugin: issues with fileExtension and filePattern parameters

[ktaletsk]

Summary

Hi @Nicholas-Schaub, I identified multiple issues with OME Zarr Converter Plugin.

What is the current bug behavior?

1. Using --fileExtension .abcdef just produces output files with .abcdef extension

2. Using --filePattern leads to division by zero run-time error. Logs:

   29-Aug-22 17:08:04 - main     - INFO     - Parsing arguments...
   29-Aug-22 17:08:04 - main     - INFO     - inpDir = /data/intensity
   29-Aug-22 17:08:04 - main     - INFO     - filePattern = S1_R1_C1-C11_A1_c0{c}0.ome.tif
   29-Aug-22 17:08:04 - main     - INFO     - fileExtension = .zarr
   29-Aug-22 17:08:04 - main     - INFO     - outDir = /data/zarr
   Traceback (most recent call last):
   File "/opt/executables/main.py", line 168, in <module>
   main(inpDir=inpDir, filePattern=filePattern, fileExtension=fileExtension, outDir=outDir)
   File "/opt/executables/main.py", line 111, in main
   ProcessManager.join_processes()
   File "/usr/local/lib/python3.9/dist-packages/preadator/preadator.py", line 362, in join_processes
   ProcessManager.main_logger.info(f'Processes {100*len(done)/len(ProcessManager._processes):6.2f}% complete')
   ZeroDivisionError: division by zero

What is the expected correct behavior?

1. --fileExtension input parameter shouldn't exist and should be removed
2. Division by zero bug when using --filePattern should be fixed

Steps to reproduce

For both cases, I downloaded a subset of Brain Hyperplex dataset from Polus-Data

1. docker run -v <path to polus-data>/polus/images/MaricRatBrain2019/subset:/data labshare/polus-ome-zarr-converter-plugin:0.5.0 --inpDir /data/intensity --outDir /data/zarr --fileExtension .abcdef
2. docker run -v <path to polus-data>/polus/images/MaricRatBrain2019/subset:/data labshare/polus-ome-zarr-converter-plugin:0.5.0 --inpDir /data/intensity --outDir /data/zarr --filePattern S1_R1_C1-C11_A1_c0{c}0.ome.tif --fileExtension .zarr

[hamshkhawar]

@ktaletsk I have fixed the issue and you can use following docker image polusai/ome-zarr-converter:0.2.1 I removed fileExtension as input argument and defined as environmental variable it requires three input arguments --inpDir --filePattern --outDir The main reason you had the problem as filePattern was incorrect for MaricRatBrain2019 subset dataset. You have to use following filePattern ='S1_R{r}_C1-C11_A1_y0{yy}_x0{xx}_c0{cc}.ome.tif' Let me know if this works for you

[ktaletsk]

@hamshkhawar I can confirm that it runs, however I have a different problem in the Notebooks environment -- I cannot delete the output files with error 'Permission denied'. In case of the old version (labshare/polus-ome-zarr-converter-plugin:0.5.0), I could delete the output files without any issues. Also, I can delete the outputs if I run the container on my local machine using Docker.

[hamshkhawar]

@Nicholas-Schaub Do you have any insight into it?

[Nicholas-Schaub]

I have run into this issue on my Linux machine in the past. @ktaletsk This is usually associated with how the container is run. The standard docker run command usually runs the container using the container default user which is root. This can lock out users who run the container from accessing the files it's generated since the files require root or sudo access to modify. Can we try running the container as a user and see if this resolves the issue?

[ktaletsk]

@Nicholas-Schaub I tried running as a user and it solved the issue:

FROM labshare/polus-bfio-util:2.1.9

# environment variables defined in labshare/polus-bfio-util
# ENV EXEC_DIR="/opt/executables"
# ENV DATA_DIR="/data"
# ENV POLUS_LOG="INFO" # Change to WARNING for fewer logs, and DEBUG for debugging

# Work directory defined in the base container
# WORKDIR ${EXEC_DIR}
ENV POLUS_EXT=".ome.zarr"
COPY VERSION ${EXEC_DIR}
COPY src ${EXEC_DIR}/

RUN pip3 install -r ${EXEC_DIR}/requirements.txt --no-cache-dir

+ USER 1000

ENTRYPOINT ["python3", "main.py"]