search

nextflow-io / nextflow

A DSL for data-driven computational pipelines
http://nextflow.io
Apache License 2.0
2.68k stars 621 forks source link

Spurious "File system `ftp://...` already exists" errors when staging #1725

Closed jpfeuffer closed 4 years ago

jpfeuffer commented 4 years ago

Bug report

Expected behavior and actual behavior

Expected: no errors during the staging of files Actual: in roughly 30% of the cases I get the error with the output below

Steps to reproduce the problem

nextflow run nf-core/proteomicslfq -r dev -profile docker --input https://raw.githubusercontent.com/bigbio/proteomics-metadata-standard/master/annotated-projects/PXD001819/sdrf.tsv --database https://raw.githubusercontent.com/nf-core/test-datasets/proteomicslfq/testdata-aws/uniprot_yeast_reviewed_isoforms_ups1_crap.fasta_td.fasta

Program output

Error executing process > 'raw_file_conversion (11)'

Caused by:
  File system `ftp://ftp.pride.ebi.ac.uk` already exists

Environment

Additional context

I am reading ftp URIs from a csv file and passing them to a subsequent step via input: path(...)

This is most likely a race condition when creating the file system. I can see the error here: https://github.com/nextflow-io/nextflow/blob/553c3119cc81012c3c4667ba280f174b0a7c7f8b/modules/nf-httpfs/src/main/nextflow/file/http/XFileSystemProvider.groovy#L78

pditommaso commented 4 years ago

Could you please provide the .nextflow.log file? tx!

jpfeuffer commented 4 years ago

Here you go. nextflow.log

pditommaso commented 4 years ago

Oh ! 

java.lang.IllegalStateException: File system `ftp://ftp.pride.ebi.ac.uk` already exists
    at nextflow.file.http.XFileSystemProvider.newFileSystem(XFileSystemProvider.groovy:78)
    at nextflow.file.http.XFileSystemProvider.getFileSystem(XFileSystemProvider.groovy:128)
    at nextflow.file.http.XFileSystemProvider.getPath(XFileSystemProvider.groovy:138)
    at java.nio.file.Paths.get(Paths.java:143)
    at nextflow.file.FileHelper.asPath(FileHelper.groovy:278)
    at nextflow.file.FileHelper.asPath(FileHelper.groovy:261)
    at nextflow.file.FileHelper$asPath$0.call(Unknown Source)
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:115)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:127)
    at nextflow.processor.TaskProcessor.normalizeToPath(TaskProcessor.groovy:1655)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.codehaus.groovy.runtime.callsite.PlainObjectMetaMethodSite.doInvoke(PlainObjectMetaMethodSite.java:43)
    at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:190)
    at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:58)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:168)
    at nextflow.processor.TaskProcessor.normalizeInputToFiles(TaskProcessor.groovy:1670)
jpfeuffer commented 4 years ago

Right! After some retrying it usually assembles all the tasks correctly. Sometimes it does not, though.

pditommaso commented 4 years ago

I guess it's race condition while loading that package.

pditommaso commented 4 years ago

Could you please include the output of nextflow info -d executed in the system where it's failing ?

pditommaso commented 4 years ago

Never mind, it's not related to the problem.

pditommaso commented 4 years ago

Fixed f65296be

jpfeuffer commented 4 years ago

That was quick. Thanks a lot! Looking forward to v20.10 then.