search

hukai916 / scATACpipe

MIT License
22 stars 2 forks source link

Issue with symbolic links on 10XGENOMICS preprocessing workflow #6

Open AdrianParrilla opened 1 month ago

AdrianParrilla commented 1 month ago

Hi!

I was trying to run the pipeline using a SMB-mounted directory in a network cluster, but I got an error at the beginning regarding symbolic links:

"Command error: ln: failed to create symbolic link 'samplesheet_ATAC.csv': Operation not supported"

The pipeline was under the /mnt/smb/TDA directory and the input command with a custom genome was:

nextflow run main.nf -profile docker --preprocess 10xgenomics --outdir /mnt/smb/TDA/scATAC_files/results --input_fastq /mnt/smb/TDA/scATAC_files/samplesheet_ATAC.csv --ref_fasta /mnt/smb/TDA/scATAC_files/masked_genome.fa --ref_gtf /mnt/smb/TDA/scATAC_files/Homo_sapiens.GRCh38.112.gtf --species_latin_name 'homo sapiens'

Attached you'll find the .nextflow.log file I got: nextflow.log

Do I have to change any additional configuration of the pipeline?

Thanks in advance for you help,

Best,

Adrian

hukai916 commented 1 month ago

This seems like a SMB issue as some protocols may not support soft links, which is used by Nextflow to link input files in each module's working directory. Let me post a question to the Nextflow help channel at Slack and see if there is a workaround.

hukai916 commented 1 month ago

Check out a potential solution (provide by Pontus): the stageInMode might help.

"For SMB, it's somewhat complicated, but it's not not supported. Anyway, running a system without supporting symbolic links sounds like an interesting excercise. If there's nothing to be done about it, you can check out stageInMode https://www.nextflow.io/docs/latest/process.html#stageinmode and set it to copy instead of using symbolic links. In case publishDir is used, that must also be adjusted to not try to do symbolic links. Of course, if you have large data sets, doing extra copies will quickly eat up your space as well as consume runtime, so it may well be worth trying to raise the issue with your systems people."

Original Slack chat: https://nextflow.slack.com/archives/C02T98A23U7/p1723213331350029?thread_ts=1723212523.062139&cid=C02T98A23U7