search

10XGenomics / cellranger

10x Genomics Single Cell Analysis
https://www.10xgenomics.com/support/software/cell-ranger
Other
340 stars 91 forks source link

cellranger multi fails on parallel file systems with shutil error #210

Closed dmalzl closed 1 year ago

dmalzl commented 1 year ago

Hi,

I am running cellranger on a cluster equipped with a parallel filesystem and encountered the following error during the sanitize_map step in cellranger multi:

Log message:
Traceback (most recent call last):
  File "/software/2020/software/cellranger/6.0.1/lib/python/cellranger/io.py", line 195, in hardlink_with_fallback
    shutil.copytree(src, dst, copy_function=os.link)
  File "/software/2020/software/cellranger/6.0.1/external/anaconda/lib/python3.7/shutil.py", line 368, in copytree
    raise Error(errors)
shutil.Error: [('/scratch-cbe/users/daniel.malzl/scanalysis/P09_V02/SC_MULTI_CS/SC_MULTI_CORE/SAMPLE_ANALYZER/SC_RNA_ANALYZER/SUMMARIZE_ANALYSIS/fork_Blood/join-u2ca92c6174/files/analysis_csv/diffexp/kmeans_5_clusters/differential_expression.csv', '/scra
tch-cbe/users/daniel.malzl/scanalysis/P09_V02/SC_MULTI_CS/SANITIZE_MAP_CALLS/fork0/chnk0-u2ca92c6295/files/Blood_analysis_csv/diffexp/kmeans_5_clusters/differential_expression.csv', "[Errno 1] Operation not permitted: '/scratch-cbe/users/daniel.malzl/sca
nalysis/P09_V02/SC_MULTI_CS/SC_MULTI_CORE/SAMPLE_ANALYZER/SC_RNA_ANALYZER/SUMMARIZE_ANALYSIS/fork_Blood/join-u2ca92c6174/files/analysis_csv/diffexp/kmeans_5_clusters/differential_expression.csv' -> '/scratch-cbe/users/daniel.malzl/scanalysis/P09_V02/SC_M
ULTI_CS/SANITIZE_MAP_CALLS/fork0/chnk0-u2ca92c6295/files/Blood_analysis_csv/diffexp/kmeans_5_clusters/differential_expression.csv'"), ...]

reading up a bit on shutil this most likely occurs due to some problems the library has when trying to move files from one filesystem to another (which might happen on the parallel file system setup as far as I understand). Running the same pipeline on a different part of the cluster with a traditional filesystem completed without issues. So I guess it is really just the file system configuration that shutil.copytree is unable to handle. Maybe it is worth to revisit this code at some point and change it but for now I leave this here as a reference to users encountering the same problem.

evolvedmicrobe commented 1 year ago

Curious, are you able to produce this error on the most recent version of Cell Ranger? We've had a few reports of shutil.Error over the years, one was caused by using the BeeGFS file system, which was fixed prior to the 6.0 release, and since then we've had reports due to transient NFS issues that we fixed in our 7.0 release.

dmalzl commented 1 year ago

Hmm interesting this is exactly the error I am dealing with (the parallel file system is BeeGFS) and I am using version 6.0.1 so the error definitely persists after 6.0 but I'll give the newest version a run tomorrow to see how this behaves.

dmalzl commented 1 year ago

runs fine with cellranger 7.1 so I guess this is fixed in the newer versions