search

epi2me-labs / wf-cas9

Other
5 stars 2 forks source link

Network unreachable #7

Open lilif99 opened 3 months ago

lilif99 commented 3 months ago

Ask away!

Hi, I'm attempting to run the workflow with nextflow + apptainer and keep getting the error message:

ERROR ~ Error executing process > 'pipeline:getVersions'

Caused by:
  Failed to pull singularity image
  command: apptainer pull  --name ontresearch-wf-cas9-shaa7b95018145dc9c753d6092309ac6be5166a491a.img.pulling.1719541119915 docker://ontresearch/wf-cas9:shaa7b95018145dc9c753d6092309ac6be5166a491a > /dev/null
  status : 255
  message:
    FATAL:   While making image from oci registry: error fetching image to cache: failed to get checksum for docker://ontresearch/wf-cas9:shaa7b95018145dc9c753d6092309ac6be5166a491a: pinging container registry registry-1.docker.io: Get "https://registry-1.docker.io/v2/": dial tcp [2600:1f18:2148:bc02:2640:1b90:cea6:b6b5]:443: connect: network is unreachable

I also get the error message: "curl: (7) Failed to connect to 18.165.201.54: Network is unreachable" in my slurm.err file

I'd be really grateful for any help with this!

nrhorner commented 3 months ago

Hi @lilif99

This appears to a network issue. Would you sahre you .nextflow.log please?

nrhorner commented 3 months ago

Also, what is the result of doing the following?

apptainer shell docker://ontresearch/wf-cas9:shaa7b95018145dc9c753d6092309ac6be5166a491a

lilif99 commented 3 months ago

Hi @nrhorner, thanks for your response

This is my .nextflow.log:


Jun-28 09:33:00.022 [main] DEBUG nextflow.cli.Launcher - $> nextflow run epi2me-labs/wf-cas9 -r 7811221 -profile apptainer --fastq wf-cas9-demo/fastq/ --reference_genome wf-cas9-demo/grch38/grch38_chr19_22.fa.gz --targets wf-cas9-demo/targets.bed -with-apptainer ./epi2me -c ./nextflow.config
Jun-28 09:33:00.172 [main] INFO  nextflow.cli.CmdRun - N E X T F L O W  ~  version 23.10.1
Jun-28 09:33:00.201 [main] DEBUG nextflow.plugin.PluginsFacade - Setting up plugin manager > mode=prod; embedded=false; plugins-dir=/home/lmf1u23/.nextflow/plugins; core-plugins: nf-amazon@2.1.4,nf-azure@1.3.3,nf-cloudcache@0.3.0,nf-codecommit@0.1.5,nf-console@1.0.6,nf-ga4gh@1.1.0,nf-google@1.8.3,nf-tower@1.6.3,nf-wave@1.0.1
Jun-28 09:33:00.232 [main] INFO  o.pf4j.DefaultPluginStatusProvider - Enabled plugins: []
Jun-28 09:33:00.234 [main] INFO  o.pf4j.DefaultPluginStatusProvider - Disabled plugins: []
Jun-28 09:33:00.238 [main] INFO  org.pf4j.DefaultPluginManager - PF4J version 3.4.1 in 'deployment' mode
Jun-28 09:33:00.252 [main] INFO  org.pf4j.AbstractPluginManager - No plugins
Jun-28 09:33:00.328 [main] DEBUG nextflow.scm.ProviderConfig - Using SCM config path: /home/lmf1u23/.nextflow/scm
Jun-28 09:33:01.397 [main] DEBUG nextflow.scm.AssetManager - Git config: /home/lmf1u23/.nextflow/assets/epi2me-labs/wf-cas9/.git/config; branch: master; remote: origin; url: https://github.com/epi2me-labs/wf-cas9.git
Jun-28 09:33:01.427 [main] DEBUG nextflow.scm.RepositoryFactory - Found Git repository result: [RepositoryFactory]
Jun-28 09:33:01.438 [main] DEBUG nextflow.scm.AssetManager - Git config: /home/lmf1u23/.nextflow/assets/epi2me-labs/wf-cas9/.git/config; branch: master; remote: origin; url: https://github.com/epi2me-labs/wf-cas9.git
Jun-28 09:33:02.789 [main] DEBUG nextflow.scm.AssetManager - WARN: Failed to check remote Git revision
org.eclipse.jgit.api.errors.TransportException: https://github.com/epi2me-labs/wf-cas9.git: connection failed
    at org.eclipse.jgit.api.LsRemoteCommand.execute(LsRemoteCommand.java:192)
    at org.eclipse.jgit.api.LsRemoteCommand.call(LsRemoteCommand.java:131)
    at nextflow.scm.AssetManager.getRemoteCommitId(AssetManager.groovy:1003)
    at nextflow.scm.AssetManager.checkRemoteStatus0(AssetManager.groovy:1014)
    at nextflow.scm.AssetManager.checkRemoteStatus(AssetManager.groovy:1032)
    at nextflow.cli.CmdRun.getScriptFile0(CmdRun.groovy:540)
    at nextflow.cli.CmdRun.getScriptFile(CmdRun.groovy:462)
    at nextflow.cli.CmdRun.run(CmdRun.groovy:317)
    at nextflow.cli.Launcher.run(Launcher.groovy:500)
    at nextflow.cli.Launcher.main(Launcher.groovy:672)
Caused by: org.eclipse.jgit.errors.TransportException: https://github.com/epi2me-labs/wf-cas9.git: connection failed
    at org.eclipse.jgit.transport.TransportHttp.connect(TransportHttp.java:729)
    at org.eclipse.jgit.transport.TransportHttp.openFetch(TransportHttp.java:465)
    at org.eclipse.jgit.api.LsRemoteCommand.execute(LsRemoteCommand.java:170)
    ... 9 common frames omitted
Caused by: java.net.SocketException: Network is unreachable
    at java.base/sun.nio.ch.Net.connect0(Native Method)
    at java.base/sun.nio.ch.Net.connect(Net.java:579)
    at java.base/sun.nio.ch.Net.connect(Net.java:568)
    at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:593)
    at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327)
    at java.base/java.net.Socket.connect(Socket.java:633)
    at java.base/sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:304)
    at java.base/sun.net.NetworkClient.doConnect(NetworkClient.java:178)
    at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:533)
    at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:638)
    at java.base/sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:266)
    at java.base/sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:380)
    at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:193)
    at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1287)
    at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1128)
    at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:179)
    at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1665)
    at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1589)
    at java.base/java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:529)
    at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:308)
    at org.eclipse.jgit.transport.http.JDKHttpConnection.getResponseCode(JDKHttpConnection.java:85)
    at org.eclipse.jgit.util.HttpSupport.response(HttpSupport.java:232)
    at org.eclipse.jgit.transport.TransportHttp.connect(TransportHttp.java:654)
    ... 11 common frames omitted
Jun-28 09:33:02.807 [main] DEBUG nextflow.config.ConfigBuilder - Found config base: /home/lmf1u23/.nextflow/assets/epi2me-labs/wf-cas9/nextflow.config
Jun-28 09:33:02.809 [main] DEBUG nextflow.config.ConfigBuilder - Found config local: /mainfs/home/lmf1u23/nextflow.config
Jun-28 09:33:02.812 [main] DEBUG nextflow.config.ConfigBuilder - User config file: /mainfs/home/lmf1u23/./nextflow.config
Jun-28 09:33:02.813 [main] DEBUG nextflow.config.ConfigBuilder - Parsing config file: /home/lmf1u23/.nextflow/assets/epi2me-labs/wf-cas9/nextflow.config
Jun-28 09:33:02.813 [main] DEBUG nextflow.config.ConfigBuilder - Parsing config file: /mainfs/home/lmf1u23/nextflow.config
Jun-28 09:33:02.813 [main] DEBUG nextflow.config.ConfigBuilder - Parsing config file: /mainfs/home/lmf1u23/nextflow.config
Jun-28 09:33:02.827 [main] DEBUG nextflow.config.ConfigBuilder - Applying config profile: `apptainer`
Jun-28 09:33:02.979 [main] DEBUG nextflow.config.ConfigBuilder - Applying config profile: `apptainer`
Jun-28 09:33:03.032 [main] DEBUG nextflow.config.ConfigBuilder - Applying config profile: `apptainer`
Jun-28 09:33:03.042 [main] DEBUG nextflow.config.ConfigBuilder - Available config profiles: [standard, apptainer, conda, singularity, awsbatch, local]
Jun-28 09:33:03.076 [main] DEBUG nextflow.config.ConfigBuilder - Enabling execution in Apptainer container as requested by command-line option `-with-apptainer null`
Jun-28 09:33:03.114 [main] DEBUG nextflow.cli.CmdRun - Applied DSL=2 from script declararion
Jun-28 09:33:03.115 [main] INFO  nextflow.cli.CmdRun - Launching `https://github.com/epi2me-labs/wf-cas9` [zen_snyder] DSL2 - revision: 7811221dfb [v1.1.1]
Jun-28 09:33:03.117 [main] DEBUG nextflow.plugin.PluginsFacade - Plugins default=[]
Jun-28 09:33:03.117 [main] DEBUG nextflow.plugin.PluginsFacade - Plugins resolved requirement=[]
Jun-28 09:33:03.126 [main] DEBUG n.secret.LocalSecretsProvider - Secrets store: /home/lmf1u23/.nextflow/secrets/store.json
Jun-28 09:33:03.133 [main] DEBUG nextflow.secret.SecretsLoader - Discovered secrets providers: [nextflow.secret.LocalSecretsProvider@5b251fb9] - activable => nextflow.secret.LocalSecretsProvider@5b251fb9
Jun-28 09:33:03.246 [main] DEBUG nextflow.Session - Session UUID: fbdacf63-5f4d-4b76-a774-61be17e86d59
Jun-28 09:33:03.246 [main] DEBUG nextflow.Session - Run name: zen_snyder
Jun-28 09:33:03.247 [main] DEBUG nextflow.Session - Executor pool size: 16
Jun-28 09:33:03.259 [main] DEBUG nextflow.file.FilePorter - File porter settings maxRetries=3; maxTransfers=50; pollTimeout=null
Jun-28 09:33:03.265 [main] DEBUG nextflow.util.ThreadPoolBuilder - Creating thread pool 'FileTransfer' minSize=10; maxSize=48; workQueue=LinkedBlockingQueue[10000]; allowCoreThreadTimeout=false
Jun-28 09:33:03.308 [main] DEBUG nextflow.cli.CmdRun - 
  Version: 23.10.1 build 5891
  Created: 12-01-2024 22:01 UTC (22:01 BST)
  System: Linux 3.10.0-1160.36.2.el7.x86_64
  Runtime: Groovy 3.0.19 on OpenJDK 64-Bit Server VM 17.0.10+7
  Encoding: UTF-8 (ANSI_X3.4-1968)
  Process: 48093@gold52.cluster.local [10.13.39.107]
  CPUs: 16 - Mem: 1.5 TB (1.1 TB) - Swap: 465.7 GB (365.7 GB)
Jun-28 09:33:03.419 [main] DEBUG nextflow.Session - Work-dir: /mainfs/home/lmf1u23/work [gpfs]
Jun-28 09:33:03.455 [main] DEBUG nextflow.executor.ExecutorFactory - Extension executors providers=[]
Jun-28 09:33:03.473 [main] DEBUG nextflow.Session - Observer factory: DefaultObserverFactory
Jun-28 09:33:03.560 [main] DEBUG nextflow.cache.CacheFactory - Using Nextflow cache factory: nextflow.cache.DefaultCacheFactory
Jun-28 09:33:03.573 [main] DEBUG nextflow.util.CustomThreadPool - Creating default thread pool > poolSize: 17; maxThreads: 1000
Jun-28 09:33:03.702 [main] DEBUG nextflow.Session - Session start
Jun-28 09:33:03.709 [main] DEBUG nextflow.trace.TraceFileObserver - Workflow started -- trace file: /mainfs/home/lmf1u23/output/execution/trace.txt
Jun-28 09:33:03.747 [main] DEBUG nextflow.Session - Using default localLib path: /home/lmf1u23/.nextflow/assets/epi2me-labs/wf-cas9/lib
Jun-28 09:33:03.752 [main] DEBUG nextflow.Session - Adding to the classpath library: /home/lmf1u23/.nextflow/assets/epi2me-labs/wf-cas9/lib
Jun-28 09:33:03.753 [main] DEBUG nextflow.Session - Adding to the classpath library: /home/lmf1u23/.nextflow/assets/epi2me-labs/wf-cas9/lib/nfcore_external_java_deps.jar
Jun-28 09:33:06.257 [main] DEBUG nextflow.script.ScriptRunner - > Launching execution
Jun-28 09:33:07.051 [main] INFO  nextflow.Nextflow - 
||||||||||   _____ ____ ___ ____  __  __ _____      _       _
||||||||||  | ____|  _ \_ _|___ \|  \/  | ____|    | | __ _| |__  ___
|||||       |  _| | |_) | |  __) | |\/| |  _| _____| |/ _` | '_ \/ __|
|||||       | |___|  __/| | / __/| |  | | |__|_____| | (_| | |_) \__ \
||||||||||  |_____|_|  |___|_____|_|  |_|_____|    |_|\__,_|_.__/|___/
||||||||||  wf-cas9 v1.1.1-g7811221
--------------------------------------------------------------------------------
Core Nextflow options
  revision        : v1.1.1
  runName         : zen_snyder
  containerEngine : apptainer
  container       : [withLabel:cas9:ontresearch/wf-cas9:shaa7b95018145dc9c753d6092309ac6be5166a491a, withLabel:wf_common:ontresearch/wf-common:sha1c5febff9f75143710826498b093d9769a5edbb9, default:./epi2me]
  launchDir       : /mainfs/home/lmf1u23
  workDir         : /mainfs/home/lmf1u23/work
  projectDir      : /home/lmf1u23/.nextflow/assets/epi2me-labs/wf-cas9
  userName        : lmf1u23
  profile         : apptainer
  configFiles     : /home/lmf1u23/.nextflow/assets/epi2me-labs/wf-cas9/nextflow.config, /mainfs/home/lmf1u23/nextflow.config, /mainfs/home/lmf1u23/./nextflow.config

Input Options
  fastq           : wf-cas9-demo/fastq/
  reference_genome: wf-cas9-demo/grch38/grch38_chr19_22.fa.gz
  targets         : wf-cas9-demo/targets.bed

!! Only displaying parameters that differ from the pipeline defaults !!
--------------------------------------------------------------------------------
If you use epi2me-labs/wf-cas9 for your analysis please cite:

* The nf-core framework
  https://doi.org/10.1038/s41587-020-0439-x

--------------------------------------------------------------------------------
This is epi2me-labs/wf-cas9 v1.1.1-g7811221.
--------------------------------------------------------------------------------
Jun-28 09:33:07.143 [main] INFO  nextflow.Nextflow - Searching input for [.fastq, .fastq.gz, .fq, .fq.gz] files.
Jun-28 09:33:07.361 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withLabel:wf_common` matches labels `ingress,wf_common` for process with name fastcat
Jun-28 09:33:07.368 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: slurm
Jun-28 09:33:07.369 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'slurm'
Jun-28 09:33:07.382 [main] DEBUG nextflow.executor.Executor - [warm up] executor > slurm
Jun-28 09:33:07.389 [main] DEBUG n.processor.TaskPollingMonitor - Creating task monitor for executor 'slurm' > capacity: 100; pollInterval: 5s; dumpInterval: 5m 
Jun-28 09:33:07.391 [main] DEBUG n.processor.TaskPollingMonitor - >>> barrier register (monitor: slurm)
Jun-28 09:33:07.393 [main] DEBUG n.executor.AbstractGridExecutor - Creating executor 'slurm' > queue-stat-interval: 1m
Jun-28 09:33:07.492 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withLabel:cas9` matches labels `cas9` for process with name pipeline:build_index
Jun-28 09:33:07.494 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: slurm
Jun-28 09:33:07.494 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'slurm'
Jun-28 09:33:07.503 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withLabel:cas9` matches labels `cas9` for process with name pipeline:getVersions
Jun-28 09:33:07.504 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: slurm
Jun-28 09:33:07.505 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'slurm'
Jun-28 09:33:07.509 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withLabel:cas9` matches labels `cas9` for process with name pipeline:getParams
Jun-28 09:33:07.510 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: slurm
Jun-28 09:33:07.510 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'slurm'
Jun-28 09:33:07.521 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withLabel:cas9` matches labels `cas9` for process with name pipeline:align_reads
Jun-28 09:33:07.523 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: slurm
Jun-28 09:33:07.523 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'slurm'
Jun-28 09:33:07.529 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withLabel:cas9` matches labels `cas9` for process with name pipeline:make_tiles
Jun-28 09:33:07.530 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: slurm
Jun-28 09:33:07.531 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'slurm'
Jun-28 09:33:07.537 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withLabel:cas9` matches labels `cas9` for process with name pipeline:coverage_summary
Jun-28 09:33:07.539 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: slurm
Jun-28 09:33:07.539 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'slurm'
Jun-28 09:33:07.554 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withLabel:cas9` matches labels `cas9` for process with name pipeline:get_on_target_reads
Jun-28 09:33:07.555 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: slurm
Jun-28 09:33:07.555 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'slurm'
Jun-28 09:33:07.593 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withLabel:cas9` matches labels `cas9` for process with name pipeline:get_on_target_bams
Jun-28 09:33:07.594 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: slurm
Jun-28 09:33:07.595 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'slurm'
Jun-28 09:33:07.607 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withLabel:cas9` matches labels `cas9` for process with name pipeline:target_summary
Jun-28 09:33:07.609 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: slurm
Jun-28 09:33:07.609 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'slurm'
Jun-28 09:33:07.616 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withLabel:cas9` matches labels `cas9` for process with name pipeline:target_coverage
Jun-28 09:33:07.617 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: slurm
Jun-28 09:33:07.618 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'slurm'
Jun-28 09:33:07.623 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withLabel:cas9` matches labels `cas9` for process with name pipeline:background
Jun-28 09:33:07.624 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: slurm
Jun-28 09:33:07.624 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'slurm'
Jun-28 09:33:07.662 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withLabel:cas9` matches labels `cas9` for process with name pipeline:build_tables
Jun-28 09:33:07.662 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: slurm
Jun-28 09:33:07.662 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'slurm'
Jun-28 09:33:07.667 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withLabel:cas9` matches labels `cas9` for process with name pipeline:combine_stats
Jun-28 09:33:07.667 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: slurm
Jun-28 09:33:07.667 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'slurm'
Jun-28 09:33:07.673 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withLabel:cas9` matches labels `cas9` for process with name pipeline:makeReport
Jun-28 09:33:07.673 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: slurm
Jun-28 09:33:07.673 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'slurm'
Jun-28 09:33:07.686 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withLabel:cas9` matches labels `cas9` for process with name pipeline:pack_files_into_sample_dirs
Jun-28 09:33:07.687 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: slurm
Jun-28 09:33:07.687 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'slurm'
Jun-28 09:33:07.696 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withLabel:cas9` matches labels `cas9` for process with name output
Jun-28 09:33:07.696 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: slurm
Jun-28 09:33:07.696 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'slurm'
Jun-28 09:33:07.698 [main] DEBUG nextflow.Session - Workflow process names [dsl2]: move_or_compress_fq_file, pipeline:getParams, get_on_target_reads, sortBam, validate_sample_sheet, pipeline:makeReport, checkBamHeaders, mergeBams, pipeline:target_summary, output, coverage_summary, pipeline:align_reads, pipeline:build_tables, pipeline:background, makeReport, getVersions, pipeline:make_tiles, pack_files_into_sample_dirs, target_coverage, combine_stats, make_tiles, pipeline:coverage_summary, pipeline:build_index, get_on_target_bams, bamstats, pipeline:getVersions, align_reads, target_summary, build_index, background, fastcat, build_tables, pipeline:get_on_target_bams, pipeline:pack_files_into_sample_dirs, pipeline:get_on_target_reads, getParams, catSortBams, pipeline:target_coverage, pipeline:combine_stats
Jun-28 09:33:07.699 [main] DEBUG nextflow.Session - Igniting dataflow network (18)
Jun-28 09:33:07.706 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > fastcat
Jun-28 09:33:07.706 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > pipeline:build_index
Jun-28 09:33:07.706 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > pipeline:getVersions
Jun-28 09:33:07.706 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > pipeline:getParams
Jun-28 09:33:07.707 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > pipeline:align_reads
Jun-28 09:33:07.707 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > pipeline:make_tiles
Jun-28 09:33:07.707 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > pipeline:coverage_summary
Jun-28 09:33:07.707 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > pipeline:get_on_target_reads
Jun-28 09:33:07.707 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > pipeline:get_on_target_bams
Jun-28 09:33:07.707 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > pipeline:target_summary
Jun-28 09:33:07.707 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > pipeline:target_coverage
Jun-28 09:33:07.707 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > pipeline:background
Jun-28 09:33:07.707 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > pipeline:build_tables
Jun-28 09:33:07.707 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > pipeline:combine_stats
Jun-28 09:33:07.707 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > pipeline:makeReport
Jun-28 09:33:07.708 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > pipeline:pack_files_into_sample_dirs
Jun-28 09:33:07.708 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > output
Jun-28 09:33:07.711 [main] DEBUG nextflow.script.ScriptRunner - Parsed script files:
  Script_8f6fbe45829235e8: /home/lmf1u23/.nextflow/assets/epi2me-labs/wf-cas9/main.nf
  Script_52f11d1ddf9ca2a2: /home/lmf1u23/.nextflow/assets/epi2me-labs/wf-cas9/./lib/ingress.nf
Jun-28 09:33:07.711 [main] DEBUG nextflow.script.ScriptRunner - > Awaiting termination 
Jun-28 09:33:07.711 [main] DEBUG nextflow.Session - Session await
Jun-28 09:33:07.803 [Actor Thread 5] INFO  nextflow.container.SingularityCache - Pulling Apptainer image docker://ontresearch/wf-common:sha1c5febff9f75143710826498b093d9769a5edbb9 [cache /mainfs/scratch/lmf1u23/cache/ontresearch-wf-common-sha1c5febff9f75143710826498b093d9769a5edbb9.img]
Jun-28 09:33:07.804 [Actor Thread 9] INFO  nextflow.container.SingularityCache - Pulling Apptainer image docker://ontresearch/wf-cas9:shaa7b95018145dc9c753d6092309ac6be5166a491a [cache /mainfs/scratch/lmf1u23/cache/ontresearch-wf-cas9-shaa7b95018145dc9c753d6092309ac6be5166a491a.img]
Jun-28 09:33:08.274 [Actor Thread 17] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for
  task: name=pipeline:getParams; work-dir=null
  error [java.lang.IllegalStateException]: java.lang.IllegalStateException: Failed to pull singularity image
  command: apptainer pull  --name ontresearch-wf-cas9-shaa7b95018145dc9c753d6092309ac6be5166a491a.img.pulling.1719563587805 docker://ontresearch/wf-cas9:shaa7b95018145dc9c753d6092309ac6be5166a491a > /dev/null
  status : 255
  message:
    FATAL:   While making image from oci registry: error fetching image to cache: failed to get checksum for docker://ontresearch/wf-cas9:shaa7b95018145dc9c753d6092309ac6be5166a491a: pinging container registry registry-1.docker.io: Get "https://registry-1.docker.io/v2/": dial tcp [2600:1f18:2148:bc01:20a3:9c3e:d4a7:9fb]:443: connect: network is unreachable

Jun-28 09:33:08.274 [Actor Thread 7] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for
  task: name=fastcat (1); work-dir=null
  error [java.lang.IllegalStateException]: java.lang.IllegalStateException: Failed to pull singularity image
  command: apptainer pull  --name ontresearch-wf-common-sha1c5febff9f75143710826498b093d9769a5edbb9.img.pulling.1719563587805 docker://ontresearch/wf-common:sha1c5febff9f75143710826498b093d9769a5edbb9 > /dev/null
  status : 255
  message:
    FATAL:   While making image from oci registry: error fetching image to cache: failed to get checksum for docker://ontresearch/wf-common:sha1c5febff9f75143710826498b093d9769a5edbb9: pinging container registry registry-1.docker.io: Get "https://registry-1.docker.io/v2/": dial tcp 3.219.239.5:443: connect: network is unreachable

Jun-28 09:33:08.275 [Actor Thread 6] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for
  task: name=pipeline:build_index; work-dir=null
  error [java.lang.IllegalStateException]: java.lang.IllegalStateException: Failed to pull singularity image
  command: apptainer pull  --name ontresearch-wf-cas9-shaa7b95018145dc9c753d6092309ac6be5166a491a.img.pulling.1719563587805 docker://ontresearch/wf-cas9:shaa7b95018145dc9c753d6092309ac6be5166a491a > /dev/null
  status : 255
  message:
    FATAL:   While making image from oci registry: error fetching image to cache: failed to get checksum for docker://ontresearch/wf-cas9:shaa7b95018145dc9c753d6092309ac6be5166a491a: pinging container registry registry-1.docker.io: Get "https://registry-1.docker.io/v2/": dial tcp [2600:1f18:2148:bc01:20a3:9c3e:d4a7:9fb]:443: connect: network is unreachable

Jun-28 09:33:08.275 [Actor Thread 3] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for
  task: name=pipeline:getVersions; work-dir=null
  error [java.lang.IllegalStateException]: java.lang.IllegalStateException: Failed to pull singularity image
  command: apptainer pull  --name ontresearch-wf-cas9-shaa7b95018145dc9c753d6092309ac6be5166a491a.img.pulling.1719563587805 docker://ontresearch/wf-cas9:shaa7b95018145dc9c753d6092309ac6be5166a491a > /dev/null
  status : 255
  message:
    FATAL:   While making image from oci registry: error fetching image to cache: failed to get checksum for docker://ontresearch/wf-cas9:shaa7b95018145dc9c753d6092309ac6be5166a491a: pinging container registry registry-1.docker.io: Get "https://registry-1.docker.io/v2/": dial tcp [2600:1f18:2148:bc01:20a3:9c3e:d4a7:9fb]:443: connect: network is unreachable

Jun-28 09:33:08.281 [Actor Thread 14] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for
  task: name=pipeline:getVersions; work-dir=null
  error [java.lang.InterruptedException]: java.lang.InterruptedException
Jun-28 09:33:08.281 [Actor Thread 15] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for
  task: name=pipeline:build_index; work-dir=null
  error [java.lang.InterruptedException]: java.lang.InterruptedException
Jun-28 09:33:08.282 [Actor Thread 13] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for
  task: name=pipeline:getParams; work-dir=null
  error [java.lang.InterruptedException]: java.lang.InterruptedException
Jun-28 09:33:08.293 [Actor Thread 7] ERROR nextflow.processor.TaskProcessor - Error executing process > 'fastcat (1)'

Caused by:
  Failed to pull singularity image
  command: apptainer pull  --name ontresearch-wf-common-sha1c5febff9f75143710826498b093d9769a5edbb9.img.pulling.1719563587805 docker://ontresearch/wf-common:sha1c5febff9f75143710826498b093d9769a5edbb9 > /dev/null
  status : 255
  message:
    FATAL:   While making image from oci registry: error fetching image to cache: failed to get checksum for docker://ontresearch/wf-common:sha1c5febff9f75143710826498b093d9769a5edbb9: pinging container registry registry-1.docker.io: Get "https://registry-1.docker.io/v2/": dial tcp 3.219.239.5:443: connect: network is unreachable

java.lang.IllegalStateException: java.lang.IllegalStateException: Failed to pull singularity image
  command: apptainer pull  --name ontresearch-wf-common-sha1c5febff9f75143710826498b093d9769a5edbb9.img.pulling.1719563587805 docker://ontresearch/wf-common:sha1c5febff9f75143710826498b093d9769a5edbb9 > /dev/null
  status : 255
  message:
    FATAL:   While making image from oci registry: error fetching image to cache: failed to get checksum for docker://ontresearch/wf-common:sha1c5febff9f75143710826498b093d9769a5edbb9: pinging container registry registry-1.docker.io: Get "https://registry-1.docker.io/v2/": dial tcp 3.219.239.5:443: connect: network is unreachable

    at nextflow.container.SingularityCache.getCachePathFor(SingularityCache.groovy:354)
    at nextflow.container.ContainerHandler.createApptainerCache(ContainerHandler.groovy:106)
    at nextflow.container.ContainerHandler.normalizeImageName(ContainerHandler.groovy:83)
    at nextflow.container.resolver.DefaultContainerResolver.resolveImage(DefaultContainerResolver.groovy:44)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:568)
    at org.codehaus.groovy.runtime.callsite.PlainObjectMetaMethodSite.doInvoke(PlainObjectMetaMethodSite.java:48)
    at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSite.invoke(PogoMetaMethodSite.java:166)
    at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.call(PogoMetaMethodSite.java:69)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:148)
    at nextflow.processor.TaskRun.containerInfo0(TaskRun.groovy:634)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:568)
    at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:107)
    at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
    at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:362)
    at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.callCurrent(PogoMetaClassSite.java:61)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:176)
    at nextflow.processor.TaskRun$_containerInfo_closure7.doCall(TaskRun.groovy:620)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:568)
    at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:107)
    at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
    at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:274)
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1030)
    at groovy.lang.Closure.call(Closure.java:427)
    at org.codehaus.groovy.runtime.ConvertedClosure.invokeCustom(ConvertedClosure.java:50)
    at org.codehaus.groovy.runtime.ConversionHandler.invoke(ConversionHandler.java:112)
    at jdk.proxy1/jdk.proxy1.$Proxy33.apply(Unknown Source)
    at java.base/java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1708)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:568)
    at org.codehaus.groovy.runtime.callsite.PlainObjectMetaMethodSite.doInvoke(PlainObjectMetaMethodSite.java:48)
    at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoCachedMethodSiteNoUnwrap.invoke(PojoMetaMethodSite.java:198)
    at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:51)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:148)
    at nextflow.processor.TaskRun.containerInfo(TaskRun.groovy:620)
    at nextflow.processor.TaskRun.getContainer(TaskRun.groovy:642)
    at nextflow.processor.TaskRun.isContainerEnabled(TaskRun.groovy:687)
    at nextflow.processor.TaskRun$isContainerEnabled$9.call(Unknown Source)
    at nextflow.processor.TaskProcessor.createTaskHashKey(TaskProcessor.groovy:2109)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:568)
    at org.codehaus.groovy.runtime.callsite.PlainObjectMetaMethodSite.doInvoke(PlainObjectMetaMethodSite.java:48)
    at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:189)
    at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:57)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:185)
    at nextflow.processor.TaskProcessor.invokeTask(TaskProcessor.groovy:638)
    at nextflow.processor.InvokeTaskAdapter.call(InvokeTaskAdapter.groovy:52)
    at groovyx.gpars.dataflow.operator.DataflowOperatorActor.startTask(DataflowOperatorActor.java:120)
    at groovyx.gpars.dataflow.operator.ForkingDataflowOperatorActor.access$001(ForkingDataflowOperatorActor.java:35)
    at groovyx.gpars.dataflow.operator.ForkingDataflowOperatorActor$1.run(ForkingDataflowOperatorActor.java:58)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
    at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: java.lang.IllegalStateException: Failed to pull singularity image
  command: apptainer pull  --name ontresearch-wf-common-sha1c5febff9f75143710826498b093d9769a5edbb9.img.pulling.1719563587805 docker://ontresearch/wf-common:sha1c5febff9f75143710826498b093d9769a5edbb9 > /dev/null
  status : 255
  message:
    FATAL:   While making image from oci registry: error fetching image to cache: failed to get checksum for docker://ontresearch/wf-common:sha1c5febff9f75143710826498b093d9769a5edbb9: pinging container registry registry-1.docker.io: Get "https://registry-1.docker.io/v2/": dial tcp 3.219.239.5:443: connect: network is unreachable

    at nextflow.container.SingularityCache.runCommand(SingularityCache.groovy:306)
    at nextflow.container.SingularityCache.downloadContainerImage0(SingularityCache.groovy:273)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:568)
    at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:107)
    at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1254)
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1030)
    at org.codehaus.groovy.runtime.InvokerHelper.invokePogoMethod(InvokerHelper.java:1036)
    at org.codehaus.groovy.runtime.InvokerHelper.invokeMethod(InvokerHelper.java:1019)
    at org.codehaus.groovy.runtime.InvokerHelper.invokeMethodSafe(InvokerHelper.java:97)
    at nextflow.container.SingularityCache$_downloadContainerImage_closure1.doCall(SingularityCache.groovy:241)
    at nextflow.container.SingularityCache$_downloadContainerImage_closure1.call(SingularityCache.groovy)
    at nextflow.file.FileMutex.lock(FileMutex.groovy:106)
    at nextflow.container.SingularityCache.downloadContainerImage(SingularityCache.groovy:241)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:568)
    at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:107)
    at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1254)
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1030)
    at org.codehaus.groovy.runtime.InvokerHelper.invokePogoMethod(InvokerHelper.java:1036)
    at org.codehaus.groovy.runtime.InvokerHelper.invokeMethod(InvokerHelper.java:1019)
    at org.codehaus.groovy.runtime.InvokerHelper.invokeMethodSafe(InvokerHelper.java:97)
    at nextflow.container.SingularityCache$_getLazyImagePath_closure2.doCall(SingularityCache.groovy:331)
    at nextflow.container.SingularityCache$_getLazyImagePath_closure2.call(SingularityCache.groovy)
    at groovyx.gpars.dataflow.LazyDataflowVariable$1.run(LazyDataflowVariable.java:70)
    ... 3 common frames omitted
Jun-28 09:33:08.300 [Actor Thread 7] DEBUG nextflow.Session - Session aborted -- Cause: java.lang.IllegalStateException: Failed to pull singularity image
  command: apptainer pull  --name ontresearch-wf-common-sha1c5febff9f75143710826498b093d9769a5edbb9.img.pulling.1719563587805 docker://ontresearch/wf-common:sha1c5febff9f75143710826498b093d9769a5edbb9 > /dev/null
  status : 255
  message:
    FATAL:   While making image from oci registry: error fetching image to cache: failed to get checksum for docker://ontresearch/wf-common:sha1c5febff9f75143710826498b093d9769a5edbb9: pinging container registry registry-1.docker.io: Get "https://registry-1.docker.io/v2/": dial tcp 3.219.239.5:443: connect: network is unreachable

Jun-28 09:33:08.333 [Actor Thread 7] DEBUG nextflow.Session - The following nodes are still active:
[process] fastcat
  status=ACTIVE
  port 0: (queue) closed; channel: -
  port 1: (value) bound ; channel: extra_args
  port 2: (cntrl) -     ; channel: $

[process] pipeline:align_reads
  status=ACTIVE
  port 0: (value) OPEN  ; channel: genome_index.mmi
  port 1: (value) bound ; channel: reference_fasta
  port 2: (queue) OPEN  ; channel: -
  port 3: (cntrl) -     ; channel: $

[process] pipeline:make_tiles
  status=ACTIVE
  port 0: (value) OPEN  ; channel: chrom.sizes
  port 1: (value) bound ; channel: targets.bed
  port 2: (cntrl) -     ; channel: $

[process] pipeline:coverage_summary
  status=ACTIVE
  port 0: (value) bound ; channel: targets.bed
  port 1: (queue) OPEN  ; channel: -
  port 2: (cntrl) -     ; channel: $

[process] pipeline:get_on_target_reads
  status=ACTIVE
  port 0: (queue) OPEN  ; channel: -
  port 1: (cntrl) -     ; channel: $

[process] pipeline:get_on_target_bams
  status=ACTIVE
  port 0: (queue) OPEN  ; channel: -
  port 1: (cntrl) -     ; channel: $

[process] pipeline:target_summary
  status=ACTIVE
  port 0: (value) bound ; channel: targets.bed
  port 1: (value) OPEN  ; channel: tiles.bed
  port 2: (value) OPEN  ; channel: tiles_inter_targets.bed
  port 3: (value) OPEN  ; channel: chrom.sizes
  port 4: (queue) OPEN  ; channel: -
  port 5: (cntrl) -     ; channel: $

[process] pipeline:target_coverage
  status=ACTIVE
  port 0: (value) bound ; channel: targets.tsv
  port 1: (value) OPEN  ; channel: tiles.tsv
  port 2: (value) OPEN  ; channel: tile_target_intersection.tsv
  port 3: (queue) OPEN  ; channel: -
  port 4: (cntrl) -     ; channel: $

[process] pipeline:background
  status=ACTIVE
  port 0: (value) bound ; channel: targets.tsv
  port 1: (value) OPEN  ; channel: tiles.tsv
  port 2: (value) OPEN  ; channel: chrom_sizes.tsv
  port 3: (queue) OPEN  ; channel: -
  port 4: (cntrl) -     ; channel: $

[process] pipeline:build_tables
  status=ACTIVE
  port 0: (queue) OPEN  ; channel: read_to_target.tsv
  port 1: (queue) OPEN  ; channel: aln_summary.tsv
  port 2: (queue) OPEN  ; channel: target_summary.tsv
  port 3: (cntrl) -     ; channel: $

[process] pipeline:combine_stats
  status=ACTIVE
  port 0: (queue) OPEN  ; channel: -
  port 1: (cntrl) -     ; channel: $

[process] pipeline:makeReport
  status=ACTIVE
  port 0: (value) OPEN  ; channel: versions/*
  port 1: (value) OPEN  ; channel: params.json
  port 2: (queue) OPEN  ; channel: per-read-stats.tsv
  port 3: (queue) OPEN  ; channel: tile_coverage.tsv
  port 4: (queue) OPEN  ; channel: target_coverage
  port 5: (queue) OPEN  ; channel: target_summary_table.tsv
  port 6: (queue) OPEN  ; channel: coverage_summary.tsv
  port 7: (queue) OPEN  ; channel: off_target_hotspots
  port 8: (cntrl) -     ; channel: $

[process] pipeline:pack_files_into_sample_dirs
  status=ACTIVE
  port 0: (queue) OPEN  ; channel: -
  port 1: (cntrl) -     ; channel: $

[process] output
  status=ACTIVE
  port 0: (queue) OPEN  ; channel: fname
  port 1: (cntrl) -     ; channel: $

Jun-28 09:33:08.344 [Task monitor] DEBUG n.processor.TaskPollingMonitor - <<< barrier arrives (monitor: slurm) - terminating tasks monitor poll loop
Jun-28 09:33:08.344 [main] DEBUG nextflow.Session - Session await > all processes finished
Jun-28 09:33:08.345 [main] DEBUG nextflow.Session - Session await > all barriers passed
Jun-28 09:33:08.349 [Actor Thread 8] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for
  task: name=fastcat; work-dir=null
  error [java.lang.InterruptedException]: java.lang.InterruptedException
Jun-28 09:33:08.355 [Actor Thread 18] DEBUG nextflow.file.SortFileCollector - FileCollector temp dir not removed: null
Jun-28 09:33:08.355 [Actor Thread 6] DEBUG nextflow.file.SortFileCollector - FileCollector temp dir not removed: null
Jun-28 09:33:08.355 [Actor Thread 5] DEBUG nextflow.file.SortFileCollector - FileCollector temp dir not removed: null
Jun-28 09:33:08.355 [Actor Thread 3] DEBUG nextflow.file.SortFileCollector - FileCollector temp dir not removed: null
Jun-28 09:33:08.356 [Actor Thread 13] DEBUG nextflow.file.SortFileCollector - FileCollector temp dir not removed: null
Jun-28 09:33:08.356 [Actor Thread 15] DEBUG nextflow.file.SortFileCollector - FileCollector temp dir not removed: null
Jun-28 09:33:08.357 [Actor Thread 9] DEBUG nextflow.file.SortFileCollector - FileCollector temp dir not removed: null
Jun-28 09:33:08.362 [main] DEBUG n.trace.WorkflowStatsObserver - Workflow completed > WorkflowStats[succeededCount=0; failedCount=0; ignoredCount=0; cachedCount=0; pendingCount=0; submittedCount=0; runningCount=0; retriesCount=0; abortedCount=0; succeedDuration=0ms; failedDuration=0ms; cachedDuration=0ms;loadCpus=0; loadMemory=0; peakRunning=0; peakCpus=0; peakMemory=0; ]
Jun-28 09:33:08.362 [main] DEBUG nextflow.trace.TraceFileObserver - Workflow completed -- saving trace file
Jun-28 09:33:08.365 [main] DEBUG nextflow.trace.ReportObserver - Workflow completed -- rendering execution report
Jun-28 09:33:09.137 [main] DEBUG nextflow.trace.TimelineObserver - Workflow completed -- rendering execution timeline
Jun-28 09:33:09.266 [main] DEBUG nextflow.cache.CacheDB - Closing CacheDB done
Jun-28 09:33:09.292 [main] DEBUG nextflow.script.ScriptRunner - > Execution complete -- Goodbye

``
lilif99 commented 3 months ago

And this is the result of running: apptainer shell docker://ontresearch/wf-cas9:shaa7b95018145dc9c753d6092309ac6be5166a491a

INFO:    Using cached SIF image
INFO:    squashfuse not found, will not be able to mount SIF or other squashfs files
INFO:    fuse2fs not found, will not be able to mount EXT3 filesystems
INFO:    gocryptfs not found, will not be able to use gocryptfs
INFO:    Converting SIF file to temporary sandbox...
lilif99 commented 3 months ago

Update: I have it working on the command line and the demo data ran fine. But I am not able to submit a slurm script to run it or I get the network connection error.

lilif99 commented 3 months ago

Update: I have figured out the issue. Our compute nodes don't have internet access so I am unable to pull the workflow from github. I have tried to run it from a local version of the workflow instead and set --disable_ping to true but it appears to still be trying to pull something from the internet:

Pulling Apptainer image docker://ontresearch/wf-common:sha1c5febff9f75143710826498b093d9769a5edbb9 [cache /mainfs/scratch/lmf1u23/cache/ontresearch-wf-common-sha1c5febff9f75143710826498b093d9769a5edbb9.img]
ERROR ~ Error executing process > 'pipeline:getVersions'

Caused by:
  Failed to pull singularity image
  command: apptainer pull  --name ontresearch-wf-cas9-shaa7b95018145dc9c753d6092309ac6be5166a491a.img.pulling.1719573194755 docker://ontresearch/wf-cas9:shaa7b95018145dc9c753d6092309ac6be5166a491a > /dev/null
  status : 255
  message:
    FATAL:   While making image from oci registry: error fetching image to cache: failed to get checksum for docker://ontresearch/wf-cas9:shaa7b95018145dc9c753d6092309ac6be5166a491a: pinging container registry registry-1.docker.io: Get "https://registry-1.docker.io/v2/": dial tcp 34.226.69.105:443: connect: network is unreachable

Is there something I should be adding to the config file to stop it trying to pull?

nrhorner commented 2 months ago

Hi @lilif99

Thanks for the update.

The workflow still needs internet access to download a singularity container. A workaround might be to move the singularity image from your local workflow run to the the workflow work directory on your cluster. singularity images can be found in <workdir>/singularity where workdir is specified by -w in the workflow command.