search

epi2me-labs / wf-basecalling

Other
34 stars 14 forks source link

[Bug]: missing basecalling model? #8

Closed ChristopherRichie closed 1 year ago

ChristopherRichie commented 1 year ago

What happened?

A bug happened!

I am trying to run the wf/basecalling with the new models for RBK-114 with v14 flongles

I am getting this error.

-- Check '.nextflow.log' file for details ERROR ~ * --basecaller_cfg: dna_r10.4.1_e8.2_400bps_sup@v4.2.0 (5kHz) is not a valid enum value (dna_r10.4.1_e8.2_400bps_sup@v4.2.0 (5kHz))

does this mean the model is not part of the workflow yet? I do not see a command for listing the available models from the basecalling workflow.

I have successfully used my script to wf/basecalling with "dna_r10.4.1_e8.2_400bps_sup@v4.1.0"

thanks

Operating System

ubuntu 18.04

Workflow Execution

Other (please describe)

Workflow Execution - EPI2ME Labs Versions

nextflow

Workflow Execution - CLI Execution Profile

Singularity

Workflow Version

https://github.com/epi2me-labs/wf-basecalling [silly_lalande] DSL2 - revision: adc677af88 [master]

Relevant log output

Jun-06 12:57:55.805 [main] DEBUG nextflow.cli.Launcher - Setting http proxy: ProxyConfig[protocol=http; host=dtn02-e0; port=3128]
Jun-06 12:57:55.833 [main] DEBUG nextflow.cli.Launcher - Setting https proxy: ProxyConfig[protocol=https; host=dtn02-e0; port=3128]
Jun-06 12:57:55.833 [main] DEBUG nextflow.cli.Launcher - Setting ftp proxy: ProxyConfig[protocol=ftp; host=dtn02-e0; port=3128]
Jun-06 12:57:55.833 [main] DEBUG nextflow.cli.Launcher - $> nextflow run epi2me-labs/wf-basecalling -profile biowulf --input ./pod5_pass/barcode23 --dorado_ext pod5 --ref pOTTC2370_ref.fa --out_dir BC23_5dorado_output --basecaller_cfg 'dna_r10.4.1_e8.2_400bps_sup@v4.2.0 (5kHz)'
Jun-06 12:57:55.909 [main] INFO  nextflow.cli.CmdRun - N E X T F L O W  ~  version 23.04.1
Jun-06 12:57:55.934 [main] DEBUG nextflow.plugin.PluginsFacade - Setting up plugin manager > mode=prod; embedded=false; plugins-dir=/home/chrisr/.nextflow/23.04.1/plugins; core-plugins: nf-amazon@1.16.2,nf-azure@1.0.1,nf-codecommit@0.1.4,nf-console@1.0.5,nf-ga4gh@1.0.5,nf-google@1.7.3,nf-tower@1.5.12,nf-wave@0.8.2
Jun-06 12:57:55.946 [main] INFO  org.pf4j.DefaultPluginStatusProvider - Enabled plugins: []
Jun-06 12:57:55.947 [main] INFO  org.pf4j.DefaultPluginStatusProvider - Disabled plugins: []
Jun-06 12:57:55.951 [main] INFO  org.pf4j.DefaultPluginManager - PF4J version 3.4.1 in 'deployment' mode
Jun-06 12:57:55.962 [main] INFO  org.pf4j.AbstractPluginManager - No plugins
Jun-06 12:57:55.981 [main] DEBUG nextflow.scm.ProviderConfig - Using SCM config path: /home/chrisr/.nextflow/23.04.1/scm
Jun-06 12:57:56.946 [main] DEBUG nextflow.scm.AssetManager - Git config: /home/chrisr/.nextflow/23.04.1/assets/epi2me-labs/wf-basecalling/.git/config; branch: master; remote: origin; url: https://github.com/epi2me-labs/wf-basecalling.git
Jun-06 12:57:56.969 [main] DEBUG nextflow.scm.RepositoryFactory - Found Git repository result: [RepositoryFactory]
Jun-06 12:57:56.978 [main] DEBUG nextflow.scm.AssetManager - Git config: /home/chrisr/.nextflow/23.04.1/assets/epi2me-labs/wf-basecalling/.git/config; branch: master; remote: origin; url: https://github.com/epi2me-labs/wf-basecalling.git
Jun-06 12:57:57.578 [main] DEBUG nextflow.config.ConfigBuilder - Found config base: /home/chrisr/.nextflow/23.04.1/assets/epi2me-labs/wf-basecalling/nextflow.config
Jun-06 12:57:57.578 [main] DEBUG nextflow.config.ConfigBuilder - Found config local: /gpfs/gsfs12/users/chrisr/CR3343/no_sample/20230605_1801_MC-110461_aph332_6e6abe40/nextflow.config
Jun-06 12:57:57.579 [main] DEBUG nextflow.config.ConfigBuilder - Parsing config file: /home/chrisr/.nextflow/23.04.1/assets/epi2me-labs/wf-basecalling/nextflow.config
Jun-06 12:57:57.579 [main] DEBUG nextflow.config.ConfigBuilder - Parsing config file: /gpfs/gsfs12/users/chrisr/CR3343/no_sample/20230605_1801_MC-110461_aph332_6e6abe40/nextflow.config
Jun-06 12:57:57.591 [main] DEBUG nextflow.config.ConfigBuilder - Applying config profile: `biowulf`
Jun-06 12:57:57.746 [main] DEBUG nextflow.config.ConfigBuilder - Applying config profile: `biowulf`
Jun-06 12:57:57.851 [main] DEBUG nextflow.config.ConfigBuilder - Available config profiles: [standard, biowulf, conda, singularity, awsbatch, local]
Jun-06 12:57:57.882 [main] DEBUG nextflow.cli.CmdRun - Applied DSL=2 from script declararion
Jun-06 12:57:57.883 [main] INFO  nextflow.cli.CmdRun - Launching `https://github.com/epi2me-labs/wf-basecalling` [high_cantor] DSL2 - revision: adc677af88 [master]
Jun-06 12:57:57.883 [main] DEBUG nextflow.plugin.PluginsFacade - Plugins default=[]
Jun-06 12:57:57.883 [main] DEBUG nextflow.plugin.PluginsFacade - Plugins resolved requirement=[]
Jun-06 12:57:57.886 [main] DEBUG nextflow.secret.LocalSecretsProvider - Secrets store: /home/chrisr/.nextflow/23.04.1/secrets/store.json
Jun-06 12:57:57.890 [main] DEBUG nextflow.secret.SecretsLoader - Discovered secrets providers: [nextflow.secret.LocalSecretsProvider@525647f3] - activable => nextflow.secret.LocalSecretsProvider@525647f3
Jun-06 12:57:57.958 [main] DEBUG nextflow.Session - Session UUID: 8bdfb191-62de-4a78-ac8b-a02c212f73d5
Jun-06 12:57:57.959 [main] DEBUG nextflow.Session - Run name: high_cantor
Jun-06 12:57:57.959 [main] DEBUG nextflow.Session - Executor pool size: 14
Jun-06 12:57:57.972 [main] DEBUG nextflow.util.ThreadPoolBuilder - Creating thread pool 'FileTransfer' minSize=10; maxSize=42; workQueue=LinkedBlockingQueue[10000]; allowCoreThreadTimeout=false
Jun-06 12:57:57.993 [main] DEBUG nextflow.cli.CmdRun - 
  Version: 23.04.1 build 5866
  Created: 15-04-2023 06:51 UTC (02:51 EDT)
  System: Linux 3.10.0-862.14.4.el7.x86_64
  Runtime: Groovy 3.0.16 on Java HotSpot(TM) 64-Bit Server VM 12.0.1+12
  Encoding: UTF-8 (UTF-8)
  Process: 26465@cn4464 [10.2.14.12]
  CPUs: 14 - Mem: 125.6 GB (95.3 GB) - Swap: 2 GB (1.4 GB)
Jun-06 12:57:58.009 [main] DEBUG nextflow.Session - Work-dir: /gpfs/gsfs12/users/chrisr/CR3343/no_sample/20230605_1801_MC-110461_aph332_6e6abe40/work [gpfs]
Jun-06 12:57:58.027 [main] DEBUG nextflow.executor.ExecutorFactory - Extension executors providers=[]
Jun-06 12:57:58.038 [main] DEBUG nextflow.Session - Observer factory: DefaultObserverFactory
Jun-06 12:57:58.130 [main] DEBUG nextflow.cache.CacheFactory - Using Nextflow cache factory: nextflow.cache.DefaultCacheFactory
Jun-06 12:57:58.142 [main] DEBUG nextflow.util.CustomThreadPool - Creating default thread pool > poolSize: 15; maxThreads: 1000
Jun-06 12:57:58.261 [main] DEBUG nextflow.Session - Session start
Jun-06 12:57:58.266 [main] DEBUG nextflow.trace.TraceFileObserver - Workflow started -- trace file: /gpfs/gsfs12/users/chrisr/CR3343/no_sample/20230605_1801_MC-110461_aph332_6e6abe40/BC23_5dorado_output/execution/trace.txt
Jun-06 12:57:58.275 [main] DEBUG nextflow.Session - Using default localLib path: /home/chrisr/.nextflow/23.04.1/assets/epi2me-labs/wf-basecalling/lib
Jun-06 12:57:58.279 [main] DEBUG nextflow.Session - Adding to the classpath library: /home/chrisr/.nextflow/23.04.1/assets/epi2me-labs/wf-basecalling/lib
Jun-06 12:57:58.280 [main] DEBUG nextflow.Session - Adding to the classpath library: /home/chrisr/.nextflow/23.04.1/assets/epi2me-labs/wf-basecalling/lib/nfcore_external_java_deps.jar
Jun-06 12:57:59.265 [SIGINT handler] DEBUG nextflow.Session - Session aborted -- Cause: SIGINT
Jun-06 12:57:59.956 [main] DEBUG nextflow.script.ScriptRunner - > Launching execution
Jun-06 12:58:00.547 [main] WARN  nextflow.NextflowMeta$Preview - NEXTFLOW RECURSION IS A PREVIEW FEATURE - SYNTAX AND FUNCTIONALITY CAN CHANGE IN FUTURE RELEASES
Jun-06 12:58:00.582 [SIGINT handler] INFO  nextflow.Session - Adieu
SamStudio8 commented 1 year ago

Hi @ChristopherRichie, apologies our workflow is lagging a little behind the official model releases. We will be releasing an update to this workflow with the v4.2 models tomorrow, hang tight!

SamStudio8 commented 1 year ago

@ChristopherRichie wf-basecalling v0.7.0 is now released 🚀. The dna_r10.4.1_e8.2_400bps_sup@v4.2.0 model you are looking for is included in this release. Additionally --help in the latest version will list all valid basecalling model options, I hope that will be helpful in future. Please update any cached versions of the workflow you may have with nextflow pull epi2me-labs/wf-basecalling.

ChristopherRichie commented 1 year ago

Hi SamStudio8, thanks for pushing the update. I see v0.7.0, and using --help, I can see the available models... THIS IS VERY HELPFUL... thanks!

When I try to run the script now, I am getting a new failure (pasted FAILED.nextflow.log below, and LOG attached): Jun-09 12:37:27.087 [Actor Thread 12] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for task: name=getVersions; work-dir=null error [java.lang.IllegalStateException]: java.lang.IllegalStateException: Failed to pull singularity image command: singularity pull --name ontresearch-dorado-sha1433bfc3146fd0dc94ad9648452364f2327cf1b0.img.pulling.1686328635711 docker://ontresearch/dorado:sha1433bfc3146fd0dc94ad9648452364f2327cf1b0 > /dev/null status : 255 nextflow.log

Not sure what to do from here. thanks

ATTACHED FILE SHOULD BE: FAILED.nextflow.log

ChristopherRichie commented 1 year ago

I just received this warning:

Directory Used Limit %Used Available

/home/chrisr 15.6 GB 16.0 GB 97.81% 16.0 GB

So. perhaps the error i just posted about is due to no/low disk space. I have removed my .singularity folder, and when I ran the script again, nextflow spent time downloading everything fresh. it looks like it overcame the above error...

WORKING GREAT! THANKS!

SamStudio8 commented 1 year ago

Glad that's working for you now! 🚀