Error with CPU Part 1, execution of Full Multiome Tutorial

[emix-c]

Checks before submitting the issue

• [x] I installed the latest Dictys version successfully
• [x] I downloaded the whole folder for the tutorial
• [x] I ran the tutorial cells and notebooks (if more than one) in the correct order and only once
• [x] The preceeding cells and notebooks finished successfully (i.e. the reported error is the first encountered)

Describe the error

• How did you install Dictys (which option)? I installed using conda.
• Which tutorial? The full multiome tutorial.
• What (full message) and where (cell and notebook being executed) was the error? I am encountering an error when trying to run the CPU Part1, execution block. I am specifically running into an error with Subset5, I am not seeing this error with any else. I adjusted the number of threads so that my system could handle it. Before, when I had it set to -j 36, some of the make commands to make expression0.tsv.gz were getting killed. Whenever I made adjustments, I deleted the entire tmp_static folder and reran the below block.

%%bash set -eo pipefail cd ..

Run CPU part of inference

make -f makefiles/static.mk -j 8 -k cpu || true

I ran into this error: Traceback (most recent call last): File "/home/exacloud/gscratch/CEDAR/chaoe/miniconda3/envs/dictys/lib/python3.9/runpy.py", line 197, in _run_module_as_main return _run_code(code, main_globals, None, File "/home/exacloud/gscratch/CEDAR/chaoe/miniconda3/envs/dictys/lib/python3.9/runpy.py", line 87, in _run_code exec(code, run_globals) File "/home/exacloud/gscratch/CEDAR/chaoe/miniconda3/envs/dictys/lib/python3.9/site-packages/dictys/main.py", line 13, in docstringrunner(package) File "/home/exacloud/gscratch/CEDAR/chaoe/miniconda3/envs/dictys/lib/python3.9/site-packages/docstring2argparse/init.py", line 340, in docstringrunner run_args(pkgname,funcs,args) File "/home/exacloud/gscratch/CEDAR/chaoe/miniconda3/envs/dictys/lib/python3.9/site-packages/docstring2argparse/init.py", line 330, in run_args return func(*a,**ka) File "/home/exacloud/gscratch/CEDAR/chaoe/miniconda3/envs/dictys/lib/python3.9/site-packages/dictys/chromatin.py", line 67, in macs2 d2 = shell.cmdfile(cmd,[],infiles={'cellnames.txt': namestxt},quiet=False,cd=True) File "/home/exacloud/gscratch/CEDAR/chaoe/miniconda3/envs/dictys/lib/python3.9/site-packages/dictys/utils/shell.py", line 181, in cmdfile raise RuntimeError('Command failed, possibly due to program error: ' + cmda) RuntimeError: Command failed, possibly due to program error: /home/exacloud/gscratch/CEDAR/chaoe/miniconda3/envs/dictys/lib/python3.9/site-packages/dictys/scripts/chromatin_macs2.sh cellnames.txt /home/groups/CEDAR/chaoe/projects/Dictys/data/bams /home/groups/CEDAR/chaoe/projects/Dictys/tmp_static/Subset5/reads.bam /home/groups/CEDAR/chaoe/projects/Dictys/tmp_static/Subset5/reads.bai /home/groups/CEDAR/chaoe/projects/Dictys/tmp_static/Subset5/peaks.bed hs 0.05 8 [bam_sort_core] merging from 2 files and 8 in-memory blocks... /home/exacloud/gscratch/CEDAR/chaoe/miniconda3/envs/dictys/lib/python3.9/site-packages/dictys/scripts/chromatin_macs2.sh: line 38: 1597134 Killed samtools sort -o "$output_bam" $nodes2 -l 1 02-filtered.bam

make: *** [makefiles/common.mk:138: tmp_static/Subset5/reads.bam] Error 1 makefiles/common.mk:138: warning: pattern recipe did not update peer target 'tmp_static/Subset5/peaks.bed'. makefiles/common.mk:138: warning: pattern recipe did not update peer target 'tmp_static/Subset5/reads.bai'.

• Did you deviate from any of the tutorial instructions? No I didn't.

Optional steps (may accelerate troubleshooting) I did not do any optional steps.

• Attach the error notebook and all preceeding notebooks with run output

[lingfeiwang]

Hi emix-c,

Probably your computer ran out of memory. Could you try again with -j 4 instead?

Lingfei

[emix-c]

Hi Lingfei, Yes I got it working by reducing the number of jobs! I do have a quick question, since I'm running pytorch with CPU only, I only run 1.1 CPU part execution and then 4.1 Aggregating Network part execution, so I skip 2.1 GPU execution and 3.1 CPU part 2, execution, is that correct?

I ran 1.1 CPU part execution and tried to run 4.1 but got these errors.

WARNING:2724116:2024-10-22 09:36:31,129:/home/exacloud/gscratch/CEDAR/chaoe/miniconda3/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:403:Skipping cell subset Subset1 due to error: FileNotFoundError(2, 'No such file or directory') WARNING:2724116:2024-10-22 09:36:31,156:/home/exacloud/gscratch/CEDAR/chaoe/miniconda3/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:403:Skipping cell subset Subset1 due to error: FileNotFoundError(2, 'No such file or directory') WARNING:2724116:2024-10-22 09:36:31,158:/home/exacloud/gscratch/CEDAR/chaoe/miniconda3/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:403:Skipping cell subset Subset10 due to error: FileNotFoundError(2, 'No such file or directory') WARNING:2724116:2024-10-22 09:36:31,159:/home/exacloud/gscratch/CEDAR/chaoe/miniconda3/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:403:Skipping cell subset Subset10 due to error: FileNotFoundError(2, 'No such file or directory') WARNING:2724116:2024-10-22 09:36:31,602:/home/exacloud/gscratch/CEDAR/chaoe/miniconda3/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:403:Skipping cell subset Subset11 due to error: FileNotFoundError(2, 'No such file or directory') WARNING:2724116:2024-10-22 09:36:31,603:/home/exacloud/gscratch/CEDAR/chaoe/miniconda3/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:403:Skipping cell subset Subset12 due to error: FileNotFoundError(2, 'No such file or directory') WARNING:2724116:2024-10-22 09:36:31,604:/home/exacloud/gscratch/CEDAR/chaoe/miniconda3/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:403:Skipping cell subset Subset12 due to error: FileNotFoundError(2, 'No such file or directory') WARNING:2724116:2024-10-22 09:36:31,682:/home/exacloud/gscratch/CEDAR/chaoe/miniconda3/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:403:Skipping cell subset Subset14 due to error: FileNotFoundError(2, 'No such file or directory') WARNING:2724116:2024-10-22 09:36:31,683:/home/exacloud/gscratch/CEDAR/chaoe/miniconda3/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:403:Skipping cell subset Subset2 due to error: FileNotFoundError(2, 'No such file or directory') WARNING:2724116:2024-10-22 09:36:31,683:/home/exacloud/gscratch/CEDAR/chaoe/miniconda3/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:403:Skipping cell subset Subset2 due to error: FileNotFoundError(2, 'No such file or directory') WARNING:2724116:2024-10-22 09:36:31,683:/home/exacloud/gscratch/CEDAR/chaoe/miniconda3/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:403:Skipping cell subset Subset3 due to error: FileNotFoundError(2, 'No such file or directory') WARNING:2724116:2024-10-22 09:36:31,684:/home/exacloud/gscratch/CEDAR/chaoe/miniconda3/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:403:Skipping cell subset Subset3 due to error: FileNotFoundError(2, 'No such file or directory') WARNING:2724116:2024-10-22 09:36:31,684:/home/exacloud/gscratch/CEDAR/chaoe/miniconda3/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:403:Skipping cell subset Subset4 due to error: FileNotFoundError(2, 'No such file or directory') WARNING:2724116:2024-10-22 09:36:31,684:/home/exacloud/gscratch/CEDAR/chaoe/miniconda3/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:403:Skipping cell subset Subset4 due to error: FileNotFoundError(2, 'No such file or directory') WARNING:2724116:2024-10-22 09:36:31,684:/home/exacloud/gscratch/CEDAR/chaoe/miniconda3/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:403:Skipping cell subset Subset5 due to error: FileNotFoundError(2, 'No such file or directory') WARNING:2724116:2024-10-22 09:36:31,685:/home/exacloud/gscratch/CEDAR/chaoe/miniconda3/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:403:Skipping cell subset Subset5 due to error: FileNotFoundError(2, 'No such file or directory') WARNING:2724116:2024-10-22 09:36:31,685:/home/exacloud/gscratch/CEDAR/chaoe/miniconda3/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:403:Skipping cell subset Subset6 due to error: FileNotFoundError(2, 'No such file or directory') WARNING:2724116:2024-10-22 09:36:31,685:/home/exacloud/gscratch/CEDAR/chaoe/miniconda3/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:403:Skipping cell subset Subset6 due to error: FileNotFoundError(2, 'No such file or directory') WARNING:2724116:2024-10-22 09:36:31,686:/home/exacloud/gscratch/CEDAR/chaoe/miniconda3/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:403:Skipping cell subset Subset7 due to error: FileNotFoundError(2, 'No such file or directory') WARNING:2724116:2024-10-22 09:36:31,686:/home/exacloud/gscratch/CEDAR/chaoe/miniconda3/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:403:Skipping cell subset Subset7 due to error: FileNotFoundError(2, 'No such file or directory') WARNING:2724116:2024-10-22 09:36:31,686:/home/exacloud/gscratch/CEDAR/chaoe/miniconda3/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:403:Skipping cell subset Subset8 due to error: FileNotFoundError(2, 'No such file or directory') WARNING:2724116:2024-10-22 09:36:31,687:/home/exacloud/gscratch/CEDAR/chaoe/miniconda3/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:403:Skipping cell subset Subset8 due to error: FileNotFoundError(2, 'No such file or directory') WARNING:2724116:2024-10-22 09:36:31,687:/home/exacloud/gscratch/CEDAR/chaoe/miniconda3/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:403:Skipping cell subset Subset9 due to error: FileNotFoundError(2, 'No such file or directory') WARNING:2724116:2024-10-22 09:36:31,687:/home/exacloud/gscratch/CEDAR/chaoe/miniconda3/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:403:Skipping cell subset Subset9 due to error: FileNotFoundError(2, 'No such file or directory') WARNING:2724116:2024-10-22 09:37:00,168:/home/exacloud/gscratch/CEDAR/chaoe/miniconda3/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:510:Removing 11/14 failed subsets.

[lingfeiwang]

These errors are not anticipated in the tutorial notebook. Did you specify DEVICE to cpu in 2-makefile.ipynb first? Are these the first error you encountered? I suggest to rerun the whole tutorial in a new folder to find and report the first error you encounter.