ArtemSokolov
commented
1 year ago Hi @ShihongWu,
I wonder if Nextflow may be ignoring SINGULARITY_BIND and not be passing your images folder to the container correctly.
Can you try adding the following runOptions to your custom.config:
singularity {
enabled = true
autoMounts = true
cacheDir = '/gpfs3/well/immune-rep/users/tma392/mcmicro/images'
runOptions = '-C -H "$PWD" -B /gpfs3/well/immune-rep/users/tma392/mcmicro/images:/mounted_images'
}
ShihongWu
I am encountering an issue while running mesmer process using Nextflow with a Singularity/apptainer container. The mesmer process fails with an error exit status (1), and the command executed within the container produces an error. the error is listed below:
ERROR ~ Error executing process > 'segmentation:worker (mesmer-1)'
Caused by: Process
segmentation:worker (mesmer-1)terminated with an error exit status (1)Command executed:
python /usr/src/app/run_app.py mesmer --squeeze --output-directory . --output-name cell.tif --nuclear-image exemplar-001.ome.ti
Command exit status: 1
Command output: (empty)
Command error: INFO: Environment variable SINGULARITYENV_TMPDIR is set, but APPTAINERENV_TMPDIR is preferred 2023-08-22 20:28:21.290519: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/lib/python3.8/dist-packages/cv2/../../lib64:/.singularity.d/libs 2023-08-22 20:28:21.290567: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine. Traceback (most recent call last): File "/usr/src/app/run_app.py", line 60, in
run_application(dict(ARGS._get_kwargs()))
File "/usr/src/app/deepcell_applications/app_runners.py", line 52, in run_application
app = dca.utils.get_app(arg_dict['app'])
File "/usr/src/app/deepcell_applications/utils.py", line 44, in get_app
return app_map[name]'class'
File "/usr/local/lib/python3.8/dist-packages/deepcell/applications/mesmer.py", line 223, in init
with tarfile.open(archive_path, "r:gz") as archive:
File "/usr/lib/python3.8/tarfile.py", line 1621, in open
return func(name, filemode, fileobj, **kwargs)
File "/usr/lib/python3.8/tarfile.py", line 1667, in gzopen
fileobj = GzipFile(name, mode + "b", compresslevel, fileobj)
File "/usr/lib/python3.8/gzip.py", line 173, in init
fileobj = self.myfileobj = builtins.open(filename, mode or 'rb')
FileNotFoundError: [Errno 2] No such file or directory: '/mounted_images/MultiplexSegmentation-9.tar.gz'
Work dir: /gpfs3/well/immune-rep/users/tma392/mcmicro/images/work/b5/232cd350517008765e9b4ef0c7115f
Tip: you can replicate the issue by changing to the process work dir and entering the command
bash .command.run-- Check '.nextflow.log' file for details
The steps I have taken are listed below:
Edited the mesmer.py script to incorporate necessary changes. The reason for the modification is that our High-Performance Computing (HPC) cluster is not connected to the internet, so we need to manually download the model. The changes I made here is : def init(self, model=None):
Rebuilt the mesmer image using my_image.def which is this:
Use the base image
FROM vanvalenlab/deepcell-applications:0.4.0
Replace mesmer.py with the modified version
RUN rm /usr/local/lib/python3.8/dist-packages/deepcell/applications/mesmer.py COPY mesmer.py /usr/local/lib/python3.8/dist-packages/deepcell/applications/
Additional Information:
Environment:
Nextflow version: 23.04.2 Singularity version: 1.1.9-1.el7 Operating System: NAME="CentOS Linux" VERSION="7 (Core)" ID="centos" ID_LIKE="rhel fedora" VERSION_ID="7" PRETTY_NAME="CentOS Linux 7 (Core)" ANSI_COLOR="0;31" CPE_NAME="cpe:/o:centos:centos:7" HOME_URL="https://www.centos.org/" BUG_REPORT_URL="https://bugs.centos.org/"
CENTOS_MANTISBT_PROJECT="CentOS-7" CENTOS_MANTISBT_PROJECT_VERSION="7" REDHAT_SUPPORT_PRODUCT="centos" REDHAT_SUPPORT_PRODUCT_VERSION="7"
Requested Help: I'm seeking assistance in identifying the root cause of the issue and finding a solution to successfully run the mesmer segmentation process using the Singularity container in Nextflow, especially considering the manual model download and changes made to mesmer.py.