OSError: [Errno 28] No space left on device

[lscdanson]

Ask away!

Hi i encountered a new error related to I/O that might be specific to my system and i wonder if you could help me out with the issue. Tried troubleshooting by changing SINGULARITY_CACHEDIR but didn't seem to work. I'm running it on a HPC as a SLURM job using singularity with custom config:

nextflow run epi2me-labs/wf-single-cell \ -profile singularity \ -r v2.0.3 \ --ref_genome_dir /ceph/project/cribbslab/shared/proj048/analyses/BM_Sample1/epi2me/refdata-gex-GRCh38-2020-A \ -c /ceph/project/cribbslab/shared/proj048/analyses/BM_Sample1/epi2me/my_config.cfg \ --threads 32 \ --fastq /ceph/project/cribbslab/shared/proj048/analyses/BM_Sample2/data.dir/BM_Sample2.fastq.gz \ --kit_name 3prime \ --kit_version v3 \ --expected_cells 2000 \ --full_length_only True \ --out_dir /ceph/project/cribbslab/shared/proj048/analyses/BM_Sample2/epi2me/output \ -resume

Config file: my_config_20240617.txt my_config_20240617.txt my_config_20240617.txt

Error message: executor > local (324) [90/929b23] process > fastcat (1) [100%] 1 of 1 ✔ [11/41ed10] process > parse_kit_metadata (1) [100%] 1 of 1 ✔ [4e/cae50f] process > pipeline:getVersions [100%] 1 of 1 ✔ [c0/3ab1a5] process > pipeline:getParams [100%] 1 of 1 ✔ [4d/31dc07] process > pipeline:preprocess:call_pa... [100%] 1 of 1 ✔ [bb/a811f1] process > pipeline:preprocess:get_chr... [100%] 1 of 1 ✔ [ae/f06a9f] process > pipeline:preprocess:build_m... [100%] 1 of 1 ✔ [65/747683] process > pipeline:preprocess:call_ad... [100%] 75 of 75 ✔ [16/975768] process > pipeline:process_bams:split... [100%] 1 of 1 ✔ [68/5203d5] process > pipeline:process_bams:gener... [100%] 1 of 1 ✔ [37/392382] process > pipeline:process_bams:assig... [100%] 75 of 75 ✔ [66/cfc009] process > pipeline:process_bams:cat_t... [100%] 1 of 1 ✔ [17/c839fd] process > pipeline:process_bams:merge... [100%] 1 of 1 ✔ [b0/4572b7] process > pipeline:process_bams:strin... [100%] 40 of 40 ✔ [f9/124217] process > pipeline:process_bams:align... [100%] 40 of 40 ✔ [c2/8ca75b] process > pipeline:process_bams:assig... [100%] 39 of 39 ✔ [df/b2946b] process > pipeline:process_bams:creat... [100%] 39 of 39 ✔ [a1/76432a] process > pipeline:process_bams:proce... [100%] 2 of 2, failed: 2 ✘ [5b/d0a07b] process > pipeline:process_bams:merge... [100%] 1 of 1 ✔ [40/044f2f] process > pipeline:process_bams:combi... [ 0%] 0 of 1 [45/e19f76] process > pipeline:process_bams:tag_b... [ 0%] 0 of 1 [- ] process > pipeline:process_bams:umi_g... - [- ] process > pipeline:processbams:pack... - [- ] process > pipeline:prepare_report_data - [- ] process > pipeline:makeReport - ERROR ~ Error executing process > 'pipeline:process_bams:process_matrix (2)'

Caused by: Process pipeline:process_bams:process_matrix (2) terminated with an error exit status (1)

Command executed:

export NUMBA_NUM_THREADS=1 workflow-glue process_matrix inputs/matrix*.hdf --feature transcript --raw transcript_raw_feature_bc_matrix --processed transcript_processed_feature_bc_matrix --per_cell_mito transcript.expression.mito-per-cell.tsv --per_cell_expr transcript.expression.mean-per-cell.tsv --umap_tsv transcript.expression.umap.tsv --enable_filtering --min_features 200 --min_cells 3 --max_mito 20 --mito_prefixes MT- --norm_count 10000 --enable_umap --replicates 3

Command exit status: 1

Command output: (empty)

Command error: @numba.jit() /home/epi2melabs/conda/lib/python3.8/site-packages/umap/distances.py:1086: NumbaDeprecationWarning: The 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details. @numba.jit() Traceback (most recent call last): File "/home/epi2melabs/conda/lib/python3.8/site-packages/numba/core/caching.py", line 469, in save data_name = overloads[key] KeyError: ('f4(f4[::1],f4[::1])', ('x86_64-unknown-linux-gnu', 'znver3', '+64bit,+adx,+aes,-amx-bf16,-amx-int8,-amx-tile,+avx,+avx2,-avx512bf16,-avx512bitalg,-avx512bw,-avx512cd,-avx512dq,-avx512er,-avx512f,-avx512fp16,-avx512ifma,-avx512pf,-avx512vbmi,-avx512vbmi2,-avx512vl,-avx512vnni,-avx512vp2intersect,-avx512vpopcntdq,-avxvnni,+bmi,+bmi2,-cldemote,+clflushopt,+clwb,+clzero,+cmov,+crc32,+cx16,+cx8,-enqcmd,+f16c,+fma,-fma4,+fsgsbase,+fxsr,-gfni,-hreset,+invpcid,-kl,-lwp,+lzcnt,+mmx,+movbe,-movdir64b,-movdiri,+mwaitx,+pclmul,-pconfig,+pku,+popcnt,-prefetchwt1,+prfchw,-ptwrite,+rdpid,+rdrnd,+rdseed,-rtm,+sahf,-serialize,-sgx,+sha,+shstk,+sse,+sse2,+sse3,+sse4.1,+sse4.2,+sse4a,+ssse3,-tbm,-tsxldtrk,-uintr,+vaes,+vpclmulqdq,-waitpkg,+wbnoinvd,-widekl,-xop,+xsave,+xsavec,+xsaveopt,+xsaves'), ('0c41297f1f6a9ad7b5e76ac3455427cf5bda8e992eb29997e17af36f3f4e8fbb', 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/d/dloi/.nextflow/assets/epi2me-labs/wf-single-cell/bin/workflow-glue", line 7, in cli() File "/home/d/dloi/.nextflow/assets/epi2me-labs/wf-single-cell/bin/workflow_glue/init.py", line 66, in cli components = get_components(allowed_components=[sys.argv[1]]) File "/home/d/dloi/.nextflow/assets/epi2me-labs/wf-single-cell/bin/workflow_glue/init.py", line 29, in get_components mod = importlib.import_module(f"{_package_name}.{name}") File "/home/epi2melabs/conda/lib/python3.8/importlib/init.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 1014, in _gcd_import File "", line 991, in _find_and_load File "", line 975, in _find_and_load_unlocked File "", line 671, in _load_unlocked File "", line 843, in exec_module File "", line 219, in _call_with_frames_removed File "/home/d/dloi/.nextflow/assets/epi2me-labs/wf-single-cell/bin/workflow_glue/processmatrix.py", line 8, in import umap File "/home/epi2melabs/conda/lib/python3.8/site-packages/umap/init.py", line 2, in from .umap import UMAP File "/home/epi2melabs/conda/lib/python3.8/site-packages/umap/umap_.py", line 41, in from umap.layouts import ( File "/home/epi2melabs/conda/lib/python3.8/site-packages/umap/layouts.py", line 40, in def rdist(x, y): File "/home/epi2melabs/conda/lib/python3.8/site-packages/numba/core/decorators.py", line 241, in wrapper disp.compile(sig) File "/home/epi2melabs/conda/lib/python3.8/site-packages/numba/core/dispatcher.py", line 972, in compile self._cache.save_overload(sig, cres) File "/home/epi2melabs/conda/lib/python3.8/site-packages/numba/core/caching.py", line 652, in save_overload self._save_overload(sig, data) File "/home/epi2melabs/conda/lib/python3.8/site-packages/numba/core/caching.py", line 662, in _save_overload self._cache_file.save(key, data) File "/home/epi2melabs/conda/lib/python3.8/site-packages/numba/core/caching.py", line 478, in save self._save_index(overloads) File "/home/epi2melabs/conda/lib/python3.8/site-packages/numba/core/caching.py", line 525, in _save_index f.write(data) File "/home/epi2melabs/conda/lib/python3.8/contextlib.py", line 120, in exit next(self.gen) File "/home/epi2melabs/conda/lib/python3.8/site-packages/numba/core/caching.py", line 562, in _open_for_write yield f OSError: [Errno 28] No space left on device

Work dir: /ceph/project/cribbslab/shared/proj048/analyses/BM_Sample2/epi2me/work/fa/7398278c39a3cfe22a95f9e307e34f

Tip: you can replicate the issue by changing to the process work dir and entering the command bash .command.run

-- Check '.nextflow.log' file for details

Log file: nextflow_20240617.log nextflow_20240617.log nextflow_20240617.log

Many thanks!

[cjw85]

Hi @lscdanson,

Can you trying the advice in the Troubleshooting section of the documentation. We've often seen odd filesystem errors when using this workflow with singularity.

[lscdanson]

Hi @cjw85, thanks for your suggestion. Yes I've read through the troubleshooting guide but there's no information related to this particular error. While it might not be relevant or useful, I've submitted another job increasing the process memory:

process {
withName:process_bams:process_matrix {
memory = “1000 GB”
}
}

I will keep you posted.

[cjw85]

Sorry, I meant simply that you should apply what is mentioned there regardless of the fact that it discusses a different issue.

[nrhorner]

@lscdanson Any updates on how that last run went?

[sdguzman]

I am encountering the same error when running with Singularity on my campus HPC. Has this bug been addressed in any recent updates? A colleague of mine was able to bypass the issue by modifying the source code to disable the UMAP feature, which I don't need for my work. It would be very helpful if there was an argument or flag that could be passed to disable the UMAP features directly. Any guidance or updates on this would be greatly appreciated. Thank you!

[nrhorner]

Hi @sdguzman

Could you include your logs please?

[nrhorner]

Closing through lack of response