search

nf-core / marsseq

MARS-seq v2 pre-processing pipeline with velocity
https://nf-co.re/marsseq
MIT License
5 stars 3 forks source link

AWS full testing ignores staging of s3 files #4

Closed matq007 closed 1 year ago

matq007 commented 1 year ago

Description of the bug

Currently, when testing pipeline on AWS, only reference building is executed. The issues is that in the execution, the PREPARE_PIPELINE module can't access the s3 bucket files, they are not staged (dowloaded as FASTQ files, not sure why). Here is a link to tower execution.

You can see that as the docs state, the s3 link is converted to /nf-core/test-datasets/marsseq/testdata/SB26/, however, the files are not staged and therefore the pipeline fails.

I've checked that the files are present in the bucket. 

$ aws --no-sign-request s3 ls s3://ngi-igenomes/test-data/marsseq/

2023-06-06 16:10:32  312138696 Undetermined_S0_L001_R1_001.fastq.gz
2023-06-06 16:10:32  133363364 Undetermined_S0_L001_R2_001.fastq.gz
2023-06-06 16:10:32       9516 amp_batches.xlsx
2023-06-06 16:10:32        567 design.csv
2023-06-06 16:10:32       8737 seq_batches.xlsx
2023-06-06 16:10:32     180347 wells_cells.xlsx

I can't test this with internal dangpu cluster because I don't have AWS credentials and whenever I try to execute the pipeline it will timeout, probably because of the missing credentials. I think this could be associated with how the bucket is public/not public, see nextflow/issues/3281.

Possible solution

Check if the files are coming from s3 bucket, if yes, then convert the s3 to normal url format and download the files with WGET module.

http://test-bucket.s3.amazonaws.com/test-folder/test-file.txt http://ngi-igenomes.s3.amazonaws.com/test-data/marsseq/wells_cells.xlsx

The question still remains, will the reference folder be staged correctly, or will it fail as well?

Command used and terminal output

$ nextflow run . -profile ku_sund_dangpu --genome mm10 --outdir ./results --input /home/fdb589/Brickman/projects/proks_et_al_2022/pipeline/SB26/design_remote.csv

```csv
batch,fastq_1,fastq_2,amp_batches,seq_batches,well_cells
SB26,s3://ngi-igenomes/test-data/marsseq/Undetermined_S0_L001_R1_001.fastq.gz,s3://ngi-igenomes/test-data/marsseq/Undetermined_S0_L001_R2_001.fastq.gz,s3://ngi-igenomes/test-data/marsseq/amp_batches.xlsx,s3://ngi-igenomes/test-data/marsseq/seq_batches.xlsx,s3://ngi-igenomes/test-data/marsseq/wells_cells.xlsx

### Relevant files

```log
Jun-21 22:00:52.235 [main] DEBUG nextflow.cli.Launcher - $> nextflow run . -profile ku_sund_dangpu --genome mm10 --outdir ./results --input /home/fdb589/Brickman/projects/proks_et_al_2022/pipeline/SB26/design_remote.csv
Jun-21 22:00:52.448 [main] INFO  nextflow.cli.CmdRun - N E X T F L O W  ~  version 23.04.1
Jun-21 22:00:52.467 [main] DEBUG nextflow.plugin.PluginsFacade - Setting up plugin manager > mode=prod; embedded=false; plugins-dir=/scratch/Brickman/pipelines/nfcore/plugins; core-plugins: nf-amazon@1.16.2,nf-azure@1.0.1,nf-codecommit@0.1.4,nf-console@1.0.5,nf-ga4gh@1.0.5,nf-google@1.7.3,nf-tower@1.5.12,nf-wave@0.8.2
Jun-21 22:00:52.477 [main] INFO  org.pf4j.DefaultPluginStatusProvider - Enabled plugins: []
Jun-21 22:00:52.478 [main] INFO  org.pf4j.DefaultPluginStatusProvider - Disabled plugins: []
Jun-21 22:00:52.481 [main] INFO  org.pf4j.DefaultPluginManager - PF4J version 3.4.1 in 'deployment' mode
Jun-21 22:00:52.490 [main] INFO  org.pf4j.AbstractPluginManager - No plugins
Jun-21 22:00:53.367 [main] DEBUG nextflow.config.ConfigBuilder - Found config local: /scratch/Brickman/pipelines/nf-core-marsseq/nextflow.config
Jun-21 22:00:53.367 [main] DEBUG nextflow.config.ConfigBuilder - Parsing config file: /scratch/Brickman/pipelines/nf-core-marsseq/nextflow.config
Jun-21 22:00:53.377 [main] DEBUG nextflow.config.ConfigBuilder - Applying config profile: `ku_sund_dangpu`
Jun-21 22:00:54.362 [main] DEBUG nextflow.config.ConfigBuilder - Available config profiles: [cfc_dev, ifb_core, denbi_qbic, alice, mjolnir_globe, uppmax, uge, rosalind_uge, lugh, unibe_ibu, vai, czbiohub_aws, jax, ccga_med, scw, tigem, tubingen_apg, google, ipop_up, googlels, eddie, medair, apptainer, bi, bigpurple, sbc_sharc, adcra, cedars, vsc_kul_uhasselt, pawsey_nimbus, ucl_myriad, utd_ganymede, charliecloud, icr_davros, ceres, munin, arm, rosalind, hasta, cfc, uzh, ebc, ku_sund_dangpu, ccga_dx, crick, marvin, biohpc_gen, shifter, mana, mamba, wehi, awsbatch, imperial, maestro, genotoul, abims, janelia, nihbiowulf, nu_genomics, googlebatch, oist, sahmri, mpcdf, leicester, vsc_ugent, sage, cambridge, podman, cheaha, xanadu, test, computerome, seg_globe, sanger, dkfz, pasteur, test_full, azurebatch, hki, crukmi, docker, engaging, gis, psmn, eva, fgcz, conda, crg, singularity, prince, utd_sysbio, debug, genouest, cbe, phoenix, gitpod, seawulf, uct_hpc, aws_tower, binac]
Jun-21 22:00:54.395 [main] DEBUG nextflow.cli.CmdRun - Applied DSL=2 from script declararion
Jun-21 22:00:54.411 [main] INFO  nextflow.cli.CmdRun - Launching `./main.nf` [deadly_elion] DSL2 - revision: 691bff9081
Jun-21 22:00:54.412 [main] DEBUG nextflow.plugin.PluginsFacade - Plugins default=[]
Jun-21 22:00:54.412 [main] DEBUG nextflow.plugin.PluginsFacade - Plugins resolved requirement=[]
Jun-21 22:00:54.417 [main] DEBUG nextflow.secret.LocalSecretsProvider - Secrets store: /scratch/Brickman/pipelines/nfcore/secrets/store.json
Jun-21 22:00:54.420 [main] DEBUG nextflow.secret.SecretsLoader - Discovered secrets providers: [nextflow.secret.LocalSecretsProvider@29bd85db] - activable => nextflow.secret.LocalSecretsProvider@29bd85db
Jun-21 22:00:54.477 [main] DEBUG nextflow.Session - Session UUID: 9eeae9d5-2f33-4bcc-bc65-9a05757e3280
Jun-21 22:00:54.477 [main] DEBUG nextflow.Session - Run name: deadly_elion
Jun-21 22:00:54.478 [main] DEBUG nextflow.Session - Executor pool size: 224
Jun-21 22:00:54.488 [main] DEBUG nextflow.util.ThreadPoolBuilder - Creating thread pool 'FileTransfer' minSize=10; maxSize=672; workQueue=LinkedBlockingQueue[10000]; allowCoreThreadTimeout=false
Jun-21 22:00:54.514 [main] DEBUG nextflow.cli.CmdRun - 
  Version: 23.04.1 build 5866
  Created: 15-04-2023 06:51 UTC (08:51 CEST)
  System: Linux 4.18.0-425.3.1.el8.x86_64
  Runtime: Groovy 3.0.16 on Java HotSpot(TM) 64-Bit Server VM 11.0.15+8-LTS-149
  Encoding: UTF-8 (UTF-8)
  Process: 1904246@dangpu01fl.unicph.domain [10.84.1.4]
  CPUs: 224 - Mem: 3.9 TB (181 GB) - Swap: 4 GB (0)
Jun-21 22:00:54.539 [main] DEBUG nextflow.Session - Work-dir: /scratch/Brickman/pipelines/nf-core-marsseq/work [xfs]
Jun-21 22:00:54.558 [main] DEBUG nextflow.executor.ExecutorFactory - Extension executors providers=[]
Jun-21 22:00:54.568 [main] DEBUG nextflow.Session - Observer factory: DefaultObserverFactory
Jun-21 22:00:54.618 [main] DEBUG nextflow.cache.CacheFactory - Using Nextflow cache factory: nextflow.cache.DefaultCacheFactory
Jun-21 22:00:54.629 [main] DEBUG nextflow.util.CustomThreadPool - Creating default thread pool > poolSize: 225; maxThreads: 1000
Jun-21 22:00:54.693 [main] DEBUG nextflow.Session - Session start
Jun-21 22:00:54.698 [main] DEBUG nextflow.trace.TraceFileObserver - Workflow started -- trace file: /scratch/Brickman/pipelines/nf-core-marsseq/results/pipeline_info/execution_trace_2023-06-21_22-00-54.txt
Jun-21 22:00:54.707 [main] DEBUG nextflow.Session - Using default localLib path: /scratch/Brickman/pipelines/nf-core-marsseq/lib
Jun-21 22:00:54.710 [main] DEBUG nextflow.Session - Adding to the classpath library: /scratch/Brickman/pipelines/nf-core-marsseq/lib
Jun-21 22:00:54.710 [main] DEBUG nextflow.Session - Adding to the classpath library: /scratch/Brickman/pipelines/nf-core-marsseq/lib/nfcore_external_java_deps.jar
Jun-21 22:00:55.465 [main] DEBUG nextflow.script.ScriptRunner - > Launching execution
Jun-21 22:00:55.512 [main] INFO  nextflow.Nextflow - 

------------------------------------------------------
                                        ,--./,-.
        ___     __   __   __   ___     /,-._.--~'
  |\ | |__  __ /  ` /  \ |__) |__         }  {
  | \| |       \__, \__/ |  \ |___     \`-._,-`-,
                                        `._,._,'
  nf-core/marsseq v1.0.0dev
------------------------------------------------------
Core Nextflow options
  runName                   : deadly_elion
  containerEngine           : singularity
  launchDir                 : /scratch/Brickman/pipelines/nf-core-marsseq
  workDir                   : /scratch/Brickman/pipelines/nf-core-marsseq/work
  projectDir                : /scratch/Brickman/pipelines/nf-core-marsseq
  userName                  : fdb589
  profile                   : ku_sund_dangpu
  configFiles               : /scratch/Brickman/pipelines/nf-core-marsseq/nextflow.config

Input/output options
  input                     : /home/fdb589/Brickman/projects/proks_et_al_2022/pipeline/SB26/design_remote.csv
  outdir                    : ./results
  genomes_base              : ./results/references

Reference genome options
  fasta                     : ./results/references/mm10/GRCm39.fa
  gtf                       : ./results/references/mm10/GRCm39.gtf
  bowtie2_index             : ./results/references/mm10/bowtie2/
  star_index                : ./results/references/mm10/star/

RNA velocity
  read_length               : 75

Institutional config options
  config_profile_description: dangpufl01 configuration
  config_profile_contact    : Adrija Kalvisa <adrija.kalvisa@sund.ku.dk>

Max job request options
  max_cpus                  : 30
  max_memory                : 200 GB
  max_time                  : 3d

!! Only displaying parameters that differ from the pipeline defaults !!
------------------------------------------------------
If you use nf-core/marsseq for your analysis please cite:

* The nf-core framework
  https://doi.org/10.1038/s41587-020-0439-x

* Software dependencies
  https://github.com/nf-core/marsseq/blob/master/CITATIONS.md
------------------------------------------------------
Jun-21 22:00:57.764 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withLabel:process_single` matches labels `process_single` for process with name NFCORE_MARSSEQ:MARSSEQ:INPUT_CHECK:SAMPLESHEET_CHECK
Jun-21 22:00:57.765 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withName:SAMPLESHEET_CHECK` matches process NFCORE_MARSSEQ:MARSSEQ:INPUT_CHECK:SAMPLESHEET_CHECK
Jun-21 22:00:57.770 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: slurm
Jun-21 22:00:57.770 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'slurm'
Jun-21 22:00:57.779 [main] DEBUG nextflow.executor.Executor - [warm up] executor > slurm
Jun-21 22:00:57.783 [main] DEBUG n.processor.TaskPollingMonitor - Creating task monitor for executor 'slurm' > capacity: 5; pollInterval: 5s; dumpInterval: 5m 
Jun-21 22:00:57.787 [main] DEBUG n.executor.AbstractGridExecutor - Creating executor 'slurm' > queue-stat-interval: 1m
Jun-21 22:00:57.891 [main] DEBUG nextflow.Session - Workflow process names [dsl2]: FASTQC, ERCC_CREATE, BOWTIE2_BUILD, MULTIQC, CUTADAPT, VELOCITY_WHITELIST, DOWNLOAD_GTF, VELOCITY_STARSOLO, GUNZIP_GTF, MERGE_READS, DOWNLOAD_FASTA, CAT_CAT, QC_ALIGNED, WGET, DEMULTIPLEX, GUNZIP, SAMPLESHEET_CHECK, VELOCITY_TRIM, QC_REPORT, GUNZIP_FASTA, VELOCITY_CONVERT, FASTP_SPLIT, CAT_FASTA, NFCORE_MARSSEQ:MARSSEQ:INPUT_CHECK:SAMPLESHEET_CHECK, QC_BATCH, PREPARE, STAR_GENOMEGENERATE, CUSTOM_DUMPSOFTWAREVERSIONS, BOWTIE2_ALIGN, CUT_SAM, STAR_ALIGN, EXTRACT_LABELS
Jun-21 22:00:57.896 [main] DEBUG nextflow.Session - Igniting dataflow network (11)
Jun-21 22:00:57.901 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > NFCORE_MARSSEQ:MARSSEQ:INPUT_CHECK:SAMPLESHEET_CHECK
Jun-21 22:00:57.902 [main] DEBUG nextflow.script.ScriptRunner - > Awaiting termination 
Jun-21 22:00:57.902 [main] DEBUG nextflow.Session - Session await
Jun-21 22:00:57.906 [PathVisitor-3] DEBUG nextflow.plugin.PluginUpdater - Installing plugin nf-amazon version: 1.16.2
Jun-21 22:00:57.914 [PathVisitor-3] INFO  org.pf4j.AbstractPluginManager - Plugin 'nf-amazon@1.16.2' resolved
Jun-21 22:00:57.914 [PathVisitor-3] INFO  org.pf4j.AbstractPluginManager - Start plugin 'nf-amazon@1.16.2'
Jun-21 22:00:57.932 [PathVisitor-3] DEBUG nextflow.plugin.BasePlugin - Plugin started nf-amazon@1.16.2
Jun-21 22:00:57.947 [PathVisitor-3] DEBUG nextflow.file.FileHelper - > Added 'S3FileSystemProvider' to list of installed providers [s3]
Jun-21 22:00:57.947 [PathVisitor-3] DEBUG nextflow.file.FileHelper - Started plugin 'nf-amazon' required to handle file: s3://ngi-igenomes/test-data/marsseq/wells_cells.xlsx
Jun-21 22:00:57.954 [PathVisitor-3] DEBUG nextflow.file.FileHelper - Creating a file system instance for provider: S3FileSystemProvider
Jun-21 22:00:57.960 [PathVisitor-3] DEBUG nextflow.file.FileHelper - AWS S3 config details: {max_error_retry=5}
Jun-21 22:00:57.977 [Actor Thread 14] DEBUG nextflow.container.SingularityCache - Singularity found local store for image=https://depot.galaxyproject.org/singularity/python:3.8.3; path=/scratch/Brickman/singularity-images/depot.galaxyproject.org-singularity-python-3.8.3.img
Jun-21 22:00:58.119 [Task submitter] DEBUG nextflow.executor.GridTaskHandler - [SLURM] submitted process NFCORE_MARSSEQ:MARSSEQ:INPUT_CHECK:SAMPLESHEET_CHECK (design_remote.csv) > jobId: 61809; workDir: /scratch/Brickman/pipelines/nf-core-marsseq/work/e1/168e23842e46da72bb193df6d24e09
Jun-21 22:00:58.120 [Task submitter] INFO  nextflow.Session - [e1/168e23] Submitted process > NFCORE_MARSSEQ:MARSSEQ:INPUT_CHECK:SAMPLESHEET_CHECK (design_remote.csv)
Jun-21 22:00:58.297 [PathVisitor-3] DEBUG com.upplication.s3fs.AmazonS3Client - Setting S3 glacierRetrievalTier=null
Jun-21 22:01:02.797 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[jobId: 61809; id: 1; name: NFCORE_MARSSEQ:MARSSEQ:INPUT_CHECK:SAMPLESHEET_CHECK (design_remote.csv); status: COMPLETED; exit: 0; error: -; workDir: /scratch/Brickman/pipelines/nf-core-marsseq/work/e1/168e23842e46da72bb193df6d24e09 started: 1687377662794; exited: 2023-06-21T20:01:02.132786Z; ]
Jun-21 22:01:02.816 [Task monitor] DEBUG nextflow.util.ThreadPoolBuilder - Creating thread pool 'PublishDir' minSize=10; maxSize=672; workQueue=LinkedBlockingQueue[10000]; allowCoreThreadTimeout=false
Jun-21 22:01:02.820 [main] DEBUG nextflow.Session - Session await > all processes finished
Jun-21 22:01:02.835 [main] DEBUG nextflow.Session - Session await > all barriers passed
Jun-21 22:01:08.136 [Actor Thread 19] WARN  c.a.i.InstanceMetadataServiceResourceFetcher - Fail to retrieve token 
com.amazonaws.SdkClientException: Failed to connect to service endpoint: 
    at com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:100)
    at com.amazonaws.internal.InstanceMetadataServiceResourceFetcher.getToken(InstanceMetadataServiceResourceFetcher.java:91)
    at com.amazonaws.internal.InstanceMetadataServiceResourceFetcher.readResource(InstanceMetadataServiceResourceFetcher.java:69)
    at com.amazonaws.internal.EC2ResourceFetcher.readResource(EC2ResourceFetcher.java:66)
    at com.amazonaws.auth.InstanceMetadataServiceCredentialsFetcher.getCredentialsEndpoint(InstanceMetadataServiceCredentialsFetcher.java:60)
    at com.amazonaws.auth.InstanceMetadataServiceCredentialsFetcher.getCredentialsResponse(InstanceMetadataServiceCredentialsFetcher.java:48)
    at com.amazonaws.auth.BaseCredentialsFetcher.fetchCredentials(BaseCredentialsFetcher.java:124)
    at com.amazonaws.auth.BaseCredentialsFetcher.getCredentials(BaseCredentialsFetcher.java:80)
    at com.amazonaws.auth.InstanceProfileCredentialsProvider.getCredentials(InstanceProfileCredentialsProvider.java:166)
    at com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper.getCredentials(EC2ContainerCredentialsProviderWrapper.java:75)
    at com.amazonaws.auth.AWSCredentialsProviderChain.getCredentials(AWSCredentialsProviderChain.java:118)
    at com.amazonaws.services.s3.S3CredentialsProviderChain.getCredentials(S3CredentialsProviderChain.java:36)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1269)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1290)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1157)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:814)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:781)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:755)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:697)
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:561)
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:541)
    at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5456)
    at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5403)
    at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5397)
    at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:928)
    at com.upplication.s3fs.AmazonS3Client.listObjects(AmazonS3Client.java:167)
    at com.upplication.s3fs.util.S3ObjectSummaryLookup.lookup(S3ObjectSummaryLookup.java:112)
    at com.upplication.s3fs.S3FileSystemProvider.checkAccess(S3FileSystemProvider.java:668)
    at java.base/java.nio.file.Files.exists(Files.java:2440)
    at java.base/jdk.internal.reflect.GeneratedMethodAccessor53.invoke(Unknown Source)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:107)
    at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
    at org.codehaus.groovy.runtime.callsite.StaticMetaMethodSite.invoke(StaticMetaMethodSite.java:44)
    at org.codehaus.groovy.runtime.callsite.StaticMetaMethodSite.call(StaticMetaMethodSite.java:89)
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
    at org.codehaus.groovy.runtime.callsite.StaticMetaMethodSite.call(StaticMetaMethodSite.java:94)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:139)
    at nextflow.extension.FilesEx.exists(FilesEx.groovy:456)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at org.codehaus.groovy.runtime.metaclass.ReflectionMetaMethod.invoke(ReflectionMetaMethod.java:54)
    at org.codehaus.groovy.runtime.metaclass.NewInstanceMetaMethod.invoke(NewInstanceMetaMethod.java:54)
    at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1258)
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1035)
    at groovy.runtime.metaclass.NextflowDelegatingMetaClass.invokeMethod(NextflowDelegatingMetaClass.java:64)
    at org.codehaus.groovy.runtime.callsite.PojoMetaClassSite.call(PojoMetaClassSite.java:44)
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:130)
    at Script_80cf5f94.create_fastq_channel(Script_80cf5f94:31)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:107)
    at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
    at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:362)
    at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.callCurrent(PogoMetaClassSite.java:61)
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:51)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:171)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:185)
    at Script_80cf5f94$_runScript_closure1$_closure2$_closure3.doCall(Script_80cf5f94:15)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:107)
    at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
    at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:274)
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1035)
    at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:38)
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:139)
    at nextflow.extension.MapOp$_apply_closure1.doCall(MapOp.groovy:56)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:107)
    at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
    at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:274)
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1035)
    at groovy.lang.Closure.call(Closure.java:412)
    at groovyx.gpars.dataflow.operator.DataflowOperatorActor.startTask(DataflowOperatorActor.java:120)
    at groovyx.gpars.dataflow.operator.DataflowOperatorActor.onMessage(DataflowOperatorActor.java:108)
    at groovyx.gpars.actor.impl.SDAClosure$1.call(SDAClosure.java:43)
    at groovyx.gpars.actor.AbstractLoopingActor.runEnhancedWithoutRepliesOnMessages(AbstractLoopingActor.java:293)
    at groovyx.gpars.actor.AbstractLoopingActor.access$400(AbstractLoopingActor.java:30)
    at groovyx.gpars.actor.AbstractLoopingActor$1.handleMessage(AbstractLoopingActor.java:93)
    at groovyx.gpars.util.AsyncMessagingCore.run(AsyncMessagingCore.java:132)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.net.SocketException: Network is unreachable (connect failed)
    at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:412)
    at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:255)
    at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:237)
    at java.base/java.net.Socket.connect(Socket.java:608)
    at java.base/sun.net.NetworkClient.doConnect(NetworkClient.java:177)
    at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:474)
    at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:569)
    at java.base/sun.net.www.http.HttpClient.<init>(HttpClient.java:242)
    at java.base/sun.net.www.http.HttpClient.New(HttpClient.java:341)
    at java.base/sun.net.www.http.HttpClient.New(HttpClient.java:362)
    at java.base/sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1253)
    at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1232)
    at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1081)
    at java.base/sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:1015)
    at com.amazonaws.internal.ConnectionUtils.connectToEndpoint(ConnectionUtils.java:95)
    at com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:80)
    ... 99 common frames omitted
Jun-21 22:01:24.646 [main] DEBUG nextflow.util.ThreadPoolManager - Thread pool 'PublishDir' shutdown completed (hard=false)
Jun-21 22:01:24.649 [main] INFO  nextflow.Nextflow - -[nf-core/marsseq] Pipeline completed successfully-
Jun-21 22:01:24.649 [main] INFO  nextflow.Nextflow - -[nf-core/marsseq] Pipeline completed successfully-
Jun-21 22:01:24.655 [main] DEBUG nextflow.trace.WorkflowStatsObserver - Workflow completed > WorkflowStats[succeededCount=1; failedCount=0; ignoredCount=0; cachedCount=0; pendingCount=0; submittedCount=0; runningCount=0; retriesCount=0; abortedCount=0; succeedDuration=1s; failedDuration=0ms; cachedDuration=0ms;loadCpus=0; loadMemory=0; peakRunning=1; peakCpus=1; peakMemory=6 GB; ]
Jun-21 22:01:24.655 [main] DEBUG nextflow.trace.TraceFileObserver - Workflow completed -- saving trace file
Jun-21 22:01:24.657 [main] DEBUG nextflow.trace.ReportObserver - Workflow completed -- rendering execution report
Jun-21 22:01:24.682 [main] DEBUG nextflow.trace.ReportObserver - Execution report summary data:
  [{"cpuUsage":{"mean":65.4,"min":65.4,"q1":65.4,"q2":65.4,"q3":65.4,"max":65.4,"minLabel":"NFCORE_MARSSEQ:MARSSEQ:INPUT_CHECK:SAMPLESHEET_CHECK (design_remote.csv)","maxLabel":"NFCORE_MARSSEQ:MARSSEQ:INPUT_CHECK:SAMPLESHEET_CHECK (design_remote.csv)","q1Label":"NFCORE_MARSSEQ:MARSSEQ:INPUT_CHECK:SAMPLESHEET_CHECK (design_remote.csv)","q2Label":"NFCORE_MARSSEQ:MARSSEQ:INPUT_CHECK:SAMPLESHEET_CHECK (design_remote.csv)","q3Label":"NFCORE_MARSSEQ:MARSSEQ:INPUT_CHECK:SAMPLESHEET_CHECK (design_remote.csv)"},"process":"SAMPLESHEET_CHECK","mem":{"mean":4808704,"min":4808704,"q1":4808704,"q2":4808704,"q3":4808704,"max":4808704,"minLabel":"NFCORE_MARSSEQ:MARSSEQ:INPUT_CHECK:SAMPLESHEET_CHECK (design_remote.csv)","maxLabel":"NFCORE_MARSSEQ:MARSSEQ:INPUT_CHECK:SAMPLESHEET_CHECK (design_remote.csv)","q1Label":"NFCORE_MARSSEQ:MARSSEQ:INPUT_CHECK:SAMPLESHEET_CHECK (design_remote.csv)","q2Label":"NFCORE_MARSSEQ:MARSSEQ:INPUT_CHECK:SAMPLESHEET_CHECK (design_remote.csv)","q3Label":"NFCORE_MARSSEQ:MARSSEQ:INPUT_CHECK:SAMPLESHEET_CHECK (design_remote.csv)"},"memUsage":{"mean":0.07,"min":0.07,"q1":0.07,"q2":0.07,"q3":0.07,"max":0.07,"minLabel":"NFCORE_MARSSEQ:MARSSEQ:INPUT_CHECK:SAMPLESHEET_CHECK (design_remote.csv)","maxLabel":"NFCORE_MARSSEQ:MARSSEQ:INPUT_CHECK:SAMPLESHEET_CHECK (design_remote.csv)","q1Label":"NFCORE_MARSSEQ:MARSSEQ:INPUT_CHECK:SAMPLESHEET_CHECK (design_remote.csv)","q2Label":"NFCORE_MARSSEQ:MARSSEQ:INPUT_CHECK:SAMPLESHEET_CHECK (design_remote.csv)","q3Label":"NFCORE_MARSSEQ:MARSSEQ:INPUT_CHECK:SAMPLESHEET_CHECK (design_remote.csv)"},"timeUsage":{"mean":0.01,"min":0.01,"q1":0.01,"q2":0.01,"q3":0.01,"max":0.01,"minLabel":"NFCORE_MARSSEQ:MARSSEQ:INPUT_CHECK:SAMPLESHEET_CHECK (design_remote.csv)","maxLabel":"NFCORE_MARSSEQ:MARSSEQ:INPUT_CHECK:SAMPLESHEET_CHECK (design_remote.csv)","q1Label":"NFCORE_MARSSEQ:MARSSEQ:INPUT_CHECK:SAMPLESHEET_CHECK (design_remote.csv)","q2Label":"NFCORE_MARSSEQ:MARSSEQ:INPUT_CHECK:SAMPLESHEET_CHECK (design_remote.csv)","q3Label":"NFCORE_MARSSEQ:MARSSEQ:INPUT_CHECK:SAMPLESHEET_CHECK (design_remote.csv)"},"vmem":{"mean":31240192,"min":31240192,"q1":31240192,"q2":31240192,"q3":31240192,"max":31240192,"minLabel":"NFCORE_MARSSEQ:MARSSEQ:INPUT_CHECK:SAMPLESHEET_CHECK (design_remote.csv)","maxLabel":"NFCORE_MARSSEQ:MARSSEQ:INPUT_CHECK:SAMPLESHEET_CHECK (design_remote.csv)","q1Label":"NFCORE_MARSSEQ:MARSSEQ:INPUT_CHECK:SAMPLESHEET_CHECK (design_remote.csv)","q2Label":"NFCORE_MARSSEQ:MARSSEQ:INPUT_CHECK:SAMPLESHEET_CHECK (design_remote.csv)","q3Label":"NFCORE_MARSSEQ:MARSSEQ:INPUT_CHECK:SAMPLESHEET_CHECK (design_remote.csv)"},"reads":{"mean":2072025,"min":2072025,"q1":2072025,"q2":2072025,"q3":2072025,"max":2072025,"minLabel":"NFCORE_MARSSEQ:MARSSEQ:INPUT_CHECK:SAMPLESHEET_CHECK (design_remote.csv)","maxLabel":"NFCORE_MARSSEQ:MARSSEQ:INPUT_CHECK:SAMPLESHEET_CHECK (design_remote.csv)","q1Label":"NFCORE_MARSSEQ:MARSSEQ:INPUT_CHECK:SAMPLESHEET_CHECK (design_remote.csv)","q2Label":"NFCORE_MARSSEQ:MARSSEQ:INPUT_CHECK:SAMPLESHEET_CHECK (design_remote.csv)","q3Label":"NFCORE_MARSSEQ:MARSSEQ:INPUT_CHECK:SAMPLESHEET_CHECK (design_remote.csv)"},"cpu":{"mean":65.4,"min":65.4,"q1":65.4,"q2":65.4,"q3":65.4,"max":65.4,"minLabel":"NFCORE_MARSSEQ:MARSSEQ:INPUT_CHECK:SAMPLESHEET_CHECK (design_remote.csv)","maxLabel":"NFCORE_MARSSEQ:MARSSEQ:INPUT_CHECK:SAMPLESHEET_CHECK (design_remote.csv)","q1Label":"NFCORE_MARSSEQ:MARSSEQ:INPUT_CHECK:SAMPLESHEET_CHECK (design_remote.csv)","q2Label":"NFCORE_MARSSEQ:MARSSEQ:INPUT_CHECK:SAMPLESHEET_CHECK (design_remote.csv)","q3Label":"NFCORE_MARSSEQ:MARSSEQ:INPUT_CHECK:SAMPLESHEET_CHECK (design_remote.csv)"},"time":{"mean":1000,"min":1000,"q1":1000,"q2":1000,"q3":1000,"max":1000,"minLabel":"NFCORE_MARSSEQ:MARSSEQ:INPUT_CHECK:SAMPLESHEET_CHECK (design_remote.csv)","maxLabel":"NFCORE_MARSSEQ:MARSSEQ:INPUT_CHECK:SAMPLESHEET_CHECK (design_remote.csv)","q1Label":"NFCORE_MARSSEQ:MARSSEQ:INPUT_CHECK:SAMPLESHEET_CHECK (design_remote.csv)","q2Label":"NFCORE_MARSSEQ:MARSSEQ:INPUT_CHECK:SAMPLESHEET_CHECK (design_remote.csv)","q3Label":"NFCORE_MARSSEQ:MARSSEQ:INPUT_CHECK:SAMPLESHEET_CHECK (design_remote.csv)"},"writes":{"mean":774,"min":774,"q1":774,"q2":774,"q3":774,"max":774,"minLabel":"NFCORE_MARSSEQ:MARSSEQ:INPUT_CHECK:SAMPLESHEET_CHECK (design_remote.csv)","maxLabel":"NFCORE_MARSSEQ:MARSSEQ:INPUT_CHECK:SAMPLESHEET_CHECK (design_remote.csv)","q1Label":"NFCORE_MARSSEQ:MARSSEQ:INPUT_CHECK:SAMPLESHEET_CHECK (design_remote.csv)","q2Label":"NFCORE_MARSSEQ:MARSSEQ:INPUT_CHECK:SAMPLESHEET_CHECK (design_remote.csv)","q3Label":"NFCORE_MARSSEQ:MARSSEQ:INPUT_CHECK:SAMPLESHEET_CHECK (design_remote.csv)"}}]
Jun-21 22:01:25.214 [main] DEBUG nextflow.trace.TimelineObserver - Workflow completed -- rendering execution timeline
Jun-21 22:01:25.318 [main] DEBUG nextflow.cache.CacheDB - Closing CacheDB done
Jun-21 22:01:25.318 [main] INFO  org.pf4j.AbstractPluginManager - Stop plugin 'nf-amazon@1.16.2'
Jun-21 22:01:25.318 [main] DEBUG nextflow.plugin.BasePlugin - Plugin stopped nf-amazon
Jun-21 22:01:25.350 [main] DEBUG nextflow.util.ThreadPoolManager - Thread pool 'FileTransfer' shutdown completed (hard=false)
Jun-21 22:01:25.351 [main] DEBUG nextflow.script.ScriptRunner - > Execution complete -- Goodbye

System information

DanGPU server

matq007 commented 1 year ago

One more remaining problem with velocity: https://tower.nf/orgs/nf-core/workspaces/AWSmegatests/watch/1NJxfbjgZqUonC