error executing process

Kiliankleemann commented 1 year ago

Description of the bug

Hi, I go the following error:

ERROR ~ Error executing process > 'NFCORE_FETCHNGS:SRA:SRA_FASTQ_FTP (SRX14737000_SRR18609980)'

Caused by: Process NFCORE_FETCHNGS:SRA:SRA_FASTQ_FTP (SRX14737000_SRR18609980) terminated with an error exit status (56)

Command executed:

curl \ --retry 5 --continue-at - --max-time 1200 \ -L ftp.sra.ebi.ac.uk/vol1/fastq/SRR186/080/SRR18609980/SRR18609980_1.fastq.gz \ -o SRX14737000_SRR18609980_1.fastq.gz

echo "51357cee238d9e90d306fe6c580bc058 SRX14737000_SRR18609980_1.fastq.gz" > SRX14737000_SRR18609980_1.fastq.gz.md5 md5sum -c SRX14737000_SRR18609980_1.fastq.gz.md5

curl \ --retry 5 --continue-at - --max-time 1200 \ -L ftp.sra.ebi.ac.uk/vol1/fastq/SRR186/080/SRR18609980/SRR18609980_2.fastq.gz \ -o SRX14737000_SRR18609980_2.fastq.gz

echo "ab93cc6edc517b8601f0450e202d2614 SRX14737000_SRR18609980_2.fastq.gz" > SRX14737000_SRR18609980_2.fastq.gz.md5 md5sum -c SRX14737000_SRR18609980_2.fastq.gz.md5

cat <<-END_VERSIONS > versions.yml "NFCORE_FETCHNGS:SRA:SRA_FASTQ_FTP": curl: $(echo $(curl --version | head -n 1 | sed 's/^curl //; s/ .*$//')) END_VERSIONS

Command exit status: 56

Command output: (empty)

Command error: % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed

0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0

curl: (56) Recv failure: Connection reset by peer

The flow initiated successfully and this is the first echo that I saw: executor > local (459) [a8/3c6cca] process > NFCORE_FETCHNGS:SRA:SRA_IDS_TO_RUNINFO (SRR18610076) [100%] 114 of 114 ✔ [0b/f7f9ce] process > NFCORE_FETCHNGS:SRA:SRA_RUNINFO_TO_FTP (114) [100%] 114 of 114 ✔ [fd/dc60e8] process > NFCORE_FETCHNGS:SRA:SRA_FASTQ_FTP (SRX14737000_SRR18609979) [ 66%] 228 of 342, failed: 228, retries: 228 [de/ddb1a0] process > NFCORE_FETCHNGS:SRA:FASTQ_DOWNLOAD_PREFETCH_FASTERQDUMP_SRATOOLS:CUSTOM_SRATOOLSNCBISETTINGS (ncbi-settings) [100%] 1 of 1 ✔ [- ] process > NFCORE_FETCHNGS:SRA:FASTQ_DOWNLOAD_PREFETCH_FASTERQDUMP_SRATOOLS:SRATOOLS_PREFETCH -

Would greatly appreciate advice. Kind regards Kilian

Command used and terminal output

nextflow run nf-core/fetchngs --input SRR_Acc_List.txt -profile docker --outdir fastq

Relevant files

SRR_Acc_List.txt

System information

Version: 23.04.3 build 5875 Created: 11-08-2023 18:37 UTC (20:37 CEST) System: Linux 6.2.0-32-generic Runtime: Groovy 3.0.16 on OpenJDK 64-Bit Server VM 17.0.8-internal+0-adhoc..src Encoding: UTF-8 (UTF-8)

Container is Docker: Client: Docker Engine - Community Version: 24.0.6 Context: default Debug Mode: false Plugins: buildx: Docker Buildx (Docker Inc.) Version: v0.11.2 Path: /usr/libexec/docker/cli-plugins/docker-buildx compose: Docker Compose (Docker Inc.) Version: v2.21.0 Path: /usr/libexec/docker/cli-plugins/docker-compose

Server: Containers: 2 Running: 0 Paused: 0 Stopped: 2 Images: 3 Server Version: 24.0.6 Storage Driver: overlay2 Backing Filesystem: extfs Supports d_type: true Using metacopy: false Native Overlay Diff: true userxattr: false Logging Driver: json-file Cgroup Driver: systemd Cgroup Version: 2 Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog Swarm: inactive Runtimes: io.containerd.runc.v2 runc Default Runtime: runc Init Binary: docker-init containerd version: 8165feabfdfe38c65b599c4993d227328c231fca runc version: v1.1.8-0-g82f18fe init version: de40ad0 Security Options: apparmor seccomp Profile: builtin cgroupns Kernel Version: 6.2.0-32-generic Operating System: Ubuntu 22.04.3 LTS OSType: linux Architecture: x86_64 CPUs: 32 Total Memory: 62.5GiB Name: kilian-Alienware-Aurora-R15 ID: 1a6c2e37-dec8-4ca9-9811-ad714e49340f Docker Root Dir: /var/lib/docker Debug Mode: false Experimental: false Insecure Registries: 127.0.0.0/8 Live Restore Enabled: false

ellisrichardj commented 1 year ago

Hi - I am getting exactly the same problem. I think its linked to the curl set up as I can get the files using wget without a problem

rjpbonnal commented 1 year ago

We have the same issue, curl detects ftp as default protocol because the url starts with ftp. I you prepend https:// in the curl commands it works as expected. A possible and transparent solution is to force curl using:

--proto-default https

configuring the ext.args within a config file.


    process {

        withName: SRA_FASTQ_FTP {
            ext.args = '--retry 5 --continue-at - --max-time 1200 --proto-default https'
        }

drpatelh commented 9 months ago

This should have been fixed in https://github.com/nf-core/fetchngs/pull/241. We will be using wget instead of curl in the next release. Please feel to re-open if you have any more issues.

nf-core / fetchngs