Pipeline Error

[rakeshponnala]

Hello, I'm trying to RNAseq pipeline on canine ref genome as below. But get the error. Wondering - if it has to do with as s3 permission issues.

nextflow run nf-core/rnaseq -r 1.4.2 -profile docker --reads '*_R{1,2}*.fastq.gz' --genome CanFam3.1
N E X T F L O W  ~  version 19.10.0
Launching `nf-core/rnaseq` [boring_lamarck] - revision: 3b6df9bd10 [1.4.2]
----------------------------------------------------
                                        ,--./,-.
        ___     __   __   __   ___     /,-._.--~'
  |\ | |__  __ /  ` /  \ |__) |__         }  {
  | \| |       \__, \__/ |  \ |___     \`-._,-`-,
                                        `._,._,'
  nf-core/rnaseq v1.4.2
----------------------------------------------------
Pipeline Release  : 1.4.2
Run Name          : boring_lamarck
Reads             : *_R{1,2}*.fastq.gz
Data Type         : Paired-End
Genome            : CanFam3.1
Strandedness      : None
Trimming          : 5'R1: 0 / 5'R2: 0 / 3'R1: 0 / 3'R2: 0 / NextSeq Trim: 0
Aligner           : STAR
STAR Index        : s3://ngi-igenomes/igenomes//Canis_familiaris/Ensembl/CanFam3.1/Sequence/STARIndex/
GTF Annotation    : s3://ngi-igenomes/igenomes//Canis_familiaris/Ensembl/CanFam3.1/Annotation/Genes/genes.gtf
BED Annotation    : s3://ngi-igenomes/igenomes//Canis_familiaris/Ensembl/CanFam3.1/Annotation/Genes/genes.bed
Remove Ribosomal RNA: false
Biotype GTF field : gene_biotype
Save prefs        : Ref Genome: No / Trimmed FastQ: No / Alignment intermediates: No
Max Resources     : 128 GB memory, 16 cpus, 10d time per job
Container         : docker - nfcore/rnaseq:1.4.2
Output dir        : ./results
Launch dir        : /home/ponnar02/data
Working dir       : /home/ponnar02/data/work
Script dir        : /home/ponnar02/.nextflow/assets/nf-core/rnaseq
User              : ponnar02
Config Profile    : docker
----------------------------------------------------
executor >  local (4)
executor >  local (4)
[ed/c74db2] process > get_software_versions                 [  0%] 0 of 1
executor >  local (4)
[ed/c74db2] process > get_software_versions                 [  0%] 0 of 1
[2c/a8dff2] process > fastqc (02-D07_RightKidney2_S14)      [  0%] 0 of 1
executor >  local (4)
[ed/c74db2] process > get_software_versions                 [100%] 1 of 1, failed: 1
[2c/a8dff2] process > fastqc (02-D07_RightKidney2_S14)      [100%] 1 of 1, failed: 1
executor >  local (4)
[ed/c74db2] process > get_software_versions                 [100%] 1 of 1, failed: 1
[2c/a8dff2] process > fastqc (02-D07_RightKidney2_S14)      [100%] 1 of 1, failed: 1
executor >  local (4)
[ed/c74db2] process > get_software_versions                 [100%] 1 of 1, failed: 1
[2c/a8dff2] process > fastqc (02-D07_RightKidney2_S14)      [100%] 1 of 1, failed: 1
[bd/19229d] process > trim_galore (02-D07_RightKidney2_S14) [100%] 1 of 1, failed: 1
[-        ] process > star                                  -
[-        ] process > rseqc                                 -
[-        ] process > preseq                                -
[-        ] process > markDuplicates                        -
[-        ] process > qualimap                              -
[-        ] process > dupradar                              -
[-        ] process > featureCounts                         -
[-        ] process > merge_featureCounts                   -
[-        ] process > stringtieFPKM                         -
[-        ] process > sample_correlation                    -
[-        ] process > multiqc                               -
[a0/dab906] process > output_documentation                  [100%] 1 of 1, failed: 1
[0;35m[nf-core/rnaseq] Pipeline completed with errors
WARN: Killing pending tasks (4)
WARN: To render the execution DAG in the required format it is required to install Graphviz -- See http://www.graphviz.org for more info.
Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: E49B7D2EE4931BCE; S3 Extended Request ID: BjAdG6YxFBnwYB1Kxuf/HrCsHG+yn4l8GS4WMYjO/rnZW5L5+X0dVpaTo4TfEwQMyft0BZV1d3Y=)
Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 19D4BEC3DC55B0B0; S3 Extended Request ID: jDfsXVUZfiLfadKYLfhU9N2Wno4N0EooIiCnLhXIPtfPdi+3OgQWtCnl/ZI8nM4j8LMQJmWAjKw=)
Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: CB81BF4A1B0B1A53; S3 Extended Request ID: MdZITxMIcu2EdviiLzn2X3+tirYxmmgnI3HIOLiPPly8yxwmfVSFJfK82iGXOx+Uk8Igw9nheao=)

[apeltzer]

You're using -profile docker on your local infrastructure it seems. Can you check whether you can access the s3://igenomes URLs manually?

[rakeshponnala]

I am able to access s3://igenomes. Below is an example: (base) [ponnar02@ip-172-17-4-123 data]$ aws s3 --region eu-west-1 ls --request-payer requester s3://ngi-igenomes/igenomes/Homo_sapiens/NCBI/build37.2/Sequence/WholeGenomeFasta/ 2017-04-13 14:30:35 4237 GenomeSize.xml 2017-04-13 14:30:36 3950 genome.dict 2017-04-13 14:30:36 3139918279 genome.fa 2017-04-13 14:30:40 714 genome.fa.fai

Any other way, to run this pipeline , ?

[thomasyu888]

Hi @apeltzer, I am in us-east-1 region and I also get

Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID:

aws s3 cp s3://ngi-igenomes/igenomes//Homo_sapiens/Ensembl/GRCh37/Annotation/Genes/genes.gtf ./
fatal error: An error occurred (403) when calling the HeadObject operation: Forbidden

Edit: This seems to have worked:

aws --no-sign-request s3 cp s3://ngi-igenomes/igenomes/Homo_sapiens/Ensembl/GRCh37/Annotation/Genes/genes.gtf ./

[drpatelh]

Hi @rakeshponnala! Did @thomasyu888 suggestion work for you? Will close for now because this seems to be a known issue with staging the genome files across multiple regions and will have to be fixed upstream in Nextflow.

[anhong11]

Does it get fixed or not? I still have this problem with version 3.8.1 as:

Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: XR8E68BQ0V46WM7Q; S3 Extended Request ID: ND93qxrrFs+PmwwEjNShqfo3bv0VpJt+BSAFj6VuEVU72gh9A7LsfGgNA0MRNc2v5BMv56W0GyQ=; Proxy: null)

 -- Check script '/home/hongan/.nextflow/assets/nf-core/rnaseq/./workflows/rnaseq.nf' at line: 26 or see '.nextflow.log' file for more details

I try to manually change the download cmd, but can not find where the cmd is in this pipeline

[xianggenti]

Does it get fixed or not? I still have this problem with version 3.8.1 as:

Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: XR8E68BQ0V46WM7Q; S3 Extended Request ID: ND93qxrrFs+PmwwEjNShqfo3bv0VpJt+BSAFj6VuEVU72gh9A7LsfGgNA0MRNc2v5BMv56W0GyQ=; Proxy: null)

 -- Check script '/home/hongan/.nextflow/assets/nf-core/rnaseq/./workflows/rnaseq.nf' at line: 26 or see '.nextflow.log' file for more details

I try to manually change the download cmd, but can not find where the cmd is in this pipeline

nf-core will install a plugin as nf-amazon, I don't know where to modify the cmd in order to add --no-sign-request however, a temporary solution is to set default aws credential as empty. It works for me as following:

[default]
aws_access_key_id =
aws_secret_access_key =

[anhong11]

Does it get fixed or not? I still have this problem with version 3.8.1 as:

Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: XR8E68BQ0V46WM7Q; S3 Extended Request ID: ND93qxrrFs+PmwwEjNShqfo3bv0VpJt+BSAFj6VuEVU72gh9A7LsfGgNA0MRNc2v5BMv56W0GyQ=; Proxy: null)

 -- Check script '/home/hongan/.nextflow/assets/nf-core/rnaseq/./workflows/rnaseq.nf' at line: 26 or see '.nextflow.log' file for more details

I try to manually change the download cmd, but can not find where the cmd is in this pipeline

nf-core will install a plugin as nf-amazon, I don't know where to modify the cmd in order to add --no-sign-request however, a temporary solution is to set default aws credential as empty. It works for me as following:

[default]
aws_access_key_id =
aws_secret_access_key =

Thanks for your response. Can you let me know where to add --no-sign-request parameter, and where or which file to add the [default] aws_access_key_id = aws_secret_access_key =

[xianggenti]

You can add it to ~/.aws/credentials [default] aws_access_key_id = aws_secret_access_key =