epi2me-labs / wf-human-variation

Other
100 stars 43 forks source link

[Bug]: v0.2.2 snp:pileup_variants fails with python error #9

Closed craigwindell closed 2 years ago

craigwindell commented 2 years ago

What happened?

snp:pileup_variants fails with python error: RuntimeError: module compiled against API version 0xe but this version of numpy is 0xd RuntimeError: module compiled against API version 0xe but this version of numpy is 0xd ImportError: numpy.core._multiarray_umath failed to import ImportError: numpy.core.umath failed to import

nextflow launch script: cat < nextflow.config process { executor = "local" } executor { \$local { cpus = 48 memory = "128.G" } } singularity { runOptions = "-B ${TMPDIR}" } EOF OUTPUT='results' nextflow run epi2me-labs/wf-human-variation -r v0.2.2 \ -resume \ --threads 48 \ -profile singularity \ --snp --sv \ --phase_vcf \ --use_longphase \ --tr_bed demo_data/demo.bed \ --model demo_data/ont_r104_e81_sup_g5015 \ --bam demo_data/demo.bam \ --ref demo_data/demo.fasta \ --out_dir ${OUTPUT}

Environment: HPC with PBS Pro scheduler SUSE12 Singularity: singularity-ce version 3.10.2-1 Nextflow: 22.04.5

With v0.1.1 of the workflow, this error does not occur

Operating System

ubuntu 20.04

Workflow Execution

Command line

Workflow Execution - EPI2ME Labs Versions

No response

Workflow Execution - Execution Profile

Singularity

Workflow Version

v0.2.2

Relevant log output

Oct-05 10:24:13.368 [main] DEBUG nextflow.cli.Launcher - $> nextflow run epi2me-labs/wf-human-variation -r v0.2.2 -resume --threads 48 -profile singularity --snp --sv --phase_vcf --use_longphase --tr_bed /work/ont/craig/demo_data/demo.bed --model /work/ont/craig/demo_data/ont_r104_e81_sup_g5015 --bam /work/ont/craig/demo_data/demo.bam --ref /work/ont/craig/demo_data/demo.fasta --out_dir results
Oct-05 10:24:13.448 [main] INFO  nextflow.cli.CmdRun - N E X T F L O W  ~  version 22.04.5
Oct-05 10:24:14.428 [main] DEBUG nextflow.scm.AssetManager - Git config: /home/craig/.nextflow/assets/epi2me-labs/wf-human-variation/.git/config; branch: master; remote: origin; url: https://github.com/epi2me-labs/wf-human-variation.git
Oct-05 10:24:14.439 [main] DEBUG nextflow.scm.AssetManager - Git config: /home/craig/.nextflow/assets/epi2me-labs/wf-human-variation/.git/config; branch: master; remote: origin; url: https://github.com/epi2me-labs/wf-human-variation.git
Oct-05 10:24:16.576 [main] INFO  nextflow.scm.AssetManager - NOTE: Your local project version looks outdated - a different revision is available in the remote repository [ee30d161c9]
Oct-05 10:24:16.596 [main] DEBUG nextflow.config.ConfigBuilder - Found config home: /home/craig/.nextflow/config
Oct-05 10:24:16.596 [main] DEBUG nextflow.config.ConfigBuilder - Found config base: /home/craig/.nextflow/assets/epi2me-labs/wf-human-variation/nextflow.config
Oct-05 10:24:16.597 [main] DEBUG nextflow.config.ConfigBuilder - Found config local: /mnt/work/ont/craig/dd48.1/nextflow.config
Oct-05 10:24:16.597 [main] DEBUG nextflow.config.ConfigBuilder - Parsing config file: /home/craig/.nextflow/config
Oct-05 10:24:16.597 [main] DEBUG nextflow.config.ConfigBuilder - Parsing config file: /home/craig/.nextflow/assets/epi2me-labs/wf-human-variation/nextflow.config
Oct-05 10:24:16.597 [main] DEBUG nextflow.config.ConfigBuilder - Parsing config file: /mnt/work/ont/craig/dd48.1/nextflow.config
Oct-05 10:24:16.605 [main] DEBUG nextflow.config.ConfigBuilder - Applying config profile: `singularity`
Oct-05 10:24:16.684 [main] DEBUG nextflow.config.ConfigBuilder - Applying config profile: `singularity`
Oct-05 10:24:16.828 [main] DEBUG nextflow.config.ConfigBuilder - Applying config profile: `singularity`
Oct-05 10:24:16.857 [main] DEBUG nextflow.config.ConfigBuilder - Available config profiles: [standard, conda, singularity, awsbatch, local]
Oct-05 10:24:16.876 [main] WARN  nextflow.config.ConfigBuilder - It appears you have never run this project before -- Option `-resume` is ignored
Oct-05 10:24:16.900 [main] DEBUG nextflow.cli.CmdRun - Applied DSL=2 from script declararion
Oct-05 10:24:16.901 [main] INFO  nextflow.cli.CmdRun - Launching `https://github.com/epi2me-labs/wf-human-variation` [chaotic_banach] DSL2 - revision: 898280e1ea [v0.2.2]
Oct-05 10:24:17.015 [main] DEBUG nextflow.plugin.PluginsFacade - Setting up plugin manager > mode=prod; plugins-dir=/home/craig/.nextflow/plugins; core-plugins: nf-amazon@1.7.2,nf-azure@0.13.2,nf-console@1.0.3,nf-ga4gh@1.0.3,nf-google@1.1.4,nf-sqldb@0.4.0,nf-tower@1.4.0
Oct-05 10:24:17.017 [main] DEBUG nextflow.plugin.PluginsFacade - Plugins default=[]
Oct-05 10:24:17.062 [main] INFO  org.pf4j.DefaultPluginStatusProvider - Enabled plugins: []
Oct-05 10:24:17.063 [main] INFO  org.pf4j.DefaultPluginStatusProvider - Disabled plugins: []
Oct-05 10:24:17.066 [main] INFO  org.pf4j.DefaultPluginManager - PF4J version 3.4.1 in 'deployment' mode
Oct-05 10:24:17.072 [main] INFO  org.pf4j.AbstractPluginManager - No plugins
Oct-05 10:24:17.072 [main] DEBUG nextflow.plugin.PluginUpdater - Installing plugin nf-tower version: 1.4.0
Oct-05 10:24:17.185 [main] INFO  org.pf4j.AbstractPluginManager - Plugin 'nf-tower@1.4.0' resolved
Oct-05 10:24:17.185 [main] INFO  org.pf4j.AbstractPluginManager - Start plugin 'nf-tower@1.4.0'
Oct-05 10:24:17.585 [main] DEBUG nextflow.plugin.BasePlugin - Plugin started nf-tower@1.4.0
Oct-05 10:24:17.777 [main] DEBUG nextflow.Session - Session uuid: c2ccc073-9f0c-4087-9aa4-99941ff0bec0
Oct-05 10:24:17.778 [main] DEBUG nextflow.Session - Run name: chaotic_banach
Oct-05 10:24:17.778 [main] DEBUG nextflow.Session - Executor pool size: 48
Oct-05 10:24:17.786 [main] DEBUG nextflow.config.ConfigBuilder - Found config home: /home/craig/.nextflow/config
Oct-05 10:24:17.786 [main] DEBUG nextflow.config.ConfigBuilder - Found config base: /home/craig/.nextflow/assets/epi2me-labs/wf-human-variation/nextflow.config
Oct-05 10:24:17.786 [main] DEBUG nextflow.config.ConfigBuilder - Found config local: /mnt/work/ont/craig/dd48.1/nextflow.config
Oct-05 10:24:17.786 [main] DEBUG nextflow.config.ConfigBuilder - Parsing config file: /home/craig/.nextflow/config
Oct-05 10:24:17.786 [main] DEBUG nextflow.config.ConfigBuilder - Parsing config file: /home/craig/.nextflow/assets/epi2me-labs/wf-human-variation/nextflow.config
Oct-05 10:24:17.786 [main] DEBUG nextflow.config.ConfigBuilder - Parsing config file: /mnt/work/ont/craig/dd48.1/nextflow.config
Oct-05 10:24:17.787 [main] DEBUG nextflow.config.ConfigBuilder - Applying config profile: `singularity`
Oct-05 10:24:17.820 [main] DEBUG nextflow.config.ConfigBuilder - Applying config profile: `singularity`
Oct-05 10:24:17.930 [main] DEBUG nextflow.config.ConfigBuilder - Applying config profile: `singularity`
Oct-05 10:24:17.946 [main] DEBUG nextflow.config.ConfigBuilder - Available config profiles: [standard, conda, singularity, awsbatch, local]
Oct-05 10:24:17.947 [main] WARN  nextflow.config.ConfigBuilder - It appears you have never run this project before -- Option `-resume` is ignored
Oct-05 10:24:18.002 [main] DEBUG nextflow.cli.CmdRun - 
  Version: 22.04.5 build 5708
  Created: 15-07-2022 16:09 UTC (16-07-2022 02:09 AEDT)
  System: Linux 4.12.14-122.130-default
  Runtime: Groovy 3.0.10 on Java HotSpot(TM) 64-Bit Server VM 11.0.15.1+2-LTS-10
  Encoding: UTF-8 (UTF-8)
  Process: 102843@cl5n021 [10.13.0.23]
  CPUs: 48 - Mem: 128 GB (127.4 GB) - Swap: 0 (0)
Oct-05 10:24:18.058 [main] DEBUG nextflow.Session - Work-dir: /mnt/work/ont/craig/dd48.1/work [nfs]
Oct-05 10:24:18.072 [main] DEBUG nextflow.executor.ExecutorFactory - Extension executors providers=[]
Oct-05 10:24:18.102 [main] DEBUG nextflow.Session - Observer factory: DefaultObserverFactory
Oct-05 10:24:18.114 [main] DEBUG nextflow.Session - Observer factory: TowerFactory
Oct-05 10:24:19.035 [main] DEBUG nextflow.cache.CacheFactory - Using Nextflow cache factory: nextflow.cache.DefaultCacheFactory
Oct-05 10:24:19.118 [main] DEBUG nextflow.util.CustomThreadPool - Creating default thread pool > poolSize: 49; maxThreads: 1000
Oct-05 10:24:19.628 [main] DEBUG nextflow.Session - Session start invoked
Oct-05 10:24:19.643 [main] DEBUG nextflow.trace.TraceFileObserver - Flow starting -- trace file: /mnt/work/ont/craig/dd48.1/results/execution/trace.txt
Oct-05 10:24:19.831 [main] DEBUG io.seqera.tower.plugin.TowerClient - Creating Tower observer -- endpoint=https://nftower.qut.edu.au/api; requestInterval=1s; aliveInterval=1m; maxRetries=5; backOffBase=3; backOffDelay=250
Oct-05 10:24:20.213 [main] DEBUG nextflow.Session - Using default localLib path: /home/craig/.nextflow/assets/epi2me-labs/wf-human-variation/lib
Oct-05 10:24:20.215 [main] DEBUG nextflow.Session - Adding to the classpath library: /home/craig/.nextflow/assets/epi2me-labs/wf-human-variation/lib
Oct-05 10:24:20.216 [main] DEBUG nextflow.Session - Adding to the classpath library: /home/craig/.nextflow/assets/epi2me-labs/wf-human-variation/lib/nfcore_external_java_deps.jar
Oct-05 10:24:21.961 [main] DEBUG nextflow.script.ScriptRunner - > Launching execution
Oct-05 10:24:24.734 [main] INFO  nextflow.Nextflow - Core Nextflow options
  revision           : v0.2.2
  runName            : chaotic_banach
  containerEngine    : singularity
  container          : ontresearch/wf-human-variation:sha90e1c9ae1b7cd51548c4c9b66df424fe3c3a81a8
  launchDir          : /mnt/work/ont/craig/dd48.1
  workDir            : /mnt/work/ont/craig/dd48.1/work
  projectDir         : /home/craig/.nextflow/assets/epi2me-labs/wf-human-variation
  userName           : windell
  profile            : singularity
  configFiles        : /home/craig/.nextflow/config, /home/craig/.nextflow/assets/epi2me-labs/wf-human-variation/nextflow.config, /mnt/work/ont/craig/dd48.1/nextflow.config

Basic Input/Output Options
  bam                : /work/ont/craig/demo_data/demo.bam
  ref                : /work/ont/craig/demo_data/demo.fasta
  model              : /work/ont/craig/demo_data/ont_r104_e81_sup_g5015
  out_dir            : results
  threads            : 48
  ubam_sort_threads  : 3
  ubam_bam2fq_threads: 1

wf-human-sv options
  sv                 : true
  tr_bed             : /work/ont/craig/demo_data/demo.bed

wf-human-snp options
  snp                : true
  phase_vcf          : true

Other parameters
  process_label      : wfdefault

!! Only displaying parameters that differ from the pipeline defaults !!
------------------------------------------------------
If you use epi2me-labs/wf-human-variation for your analysis please cite:

* The nf-core framework
  https://doi.org/10.1038/s41587-020-0439-x
  ...
Oct-05 10:25:29.131 [Task monitor] ERROR nextflow.processor.TaskProcessor - Error executing process > 'snp:pileup_variants (3)'

Caused by:
  Process `snp:pileup_variants (3)` terminated with an error exit status (1)

Command executed:

  export REF_PATH=ref_cache/%2s/%2s/%s
  python $(which clair3.py) CallVariantsFromCffi             --chkpnt_fn ont_r104_e81_sup_g5015/pileup             --bam_fn demo.bam             --call_fn pileup_chr20_3.vcf             --ref_fn demo.fasta             --ctgName chr20             --chunk_id 3             --chunk_num 13             --platform ont             --fast_mode False             --snp_min_af 0.08             --indel_min_af 0.15             --minMQ 5             --minCoverage 2             --call_snp_only False             --gvcf false             --temp_file_dir gvcf_tmp_path             --pileup

Command exit status:
  1

Command output:
  (empty)

Command error:
  2022-10-05 10:25:28.687667: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /.singularity.d/libs
  2022-10-05 10:25:28.687693: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
  RuntimeError: module compiled against API version 0xe but this version of numpy is 0xd
  RuntimeError: module compiled against API version 0xe but this version of numpy is 0xd
  ImportError: numpy.core._multiarray_umath failed to import
  ImportError: numpy.core.umath failed to import
  Traceback (most recent call last):
    File "/home/epi2melabs/conda/bin/clair3.py", line 94, in <module>
      main()
    File "/home/epi2melabs/conda/bin/clair3.py", line 81, in main
      submodule = import_module("%s.%s" % (directory, submodule_name))
    File "/home/epi2melabs/conda/lib/python3.9/importlib/__init__.py", line 127, in import_module
      return _bootstrap._gcd_import(name[level:], package, level)
...
SamStudio8 commented 2 years ago

Hi @craigwindell, from your log it looks as though the latest containers for this version may not have been downloaded by Nextflow which might explain this error. I have downloaded the container we released for wf-human-variation-snp for v0.2.2 and cannot replicate your import error.

Can you check which containers have been downloaded to your Singularity cache dir? You can also try deleting any wf-human-variation images from your Singularity cache dir and starting the workflow again.

craigwindell commented 2 years ago

Hi @SamStudio8,

singularity cache clean remove Nextflow's cache of images nextflow drop epi2me-labs/wf-human-variation

Same command as before in an empty folder

These are the images freshly pulled by v0.2.2: Pulling Singularity image docker://ontresearch/wf-human-variation-sv:sha04c8557a38dff4f529c859ad55cb2865782802a1 Pulling Singularity image docker://ontresearch/wf-human-variation:sha90e1c9ae1b7cd51548c4c9b66df424fe3c3a81a8 Pulling Singularity image docker://ontresearch/wf-human-variation-snp:shaa215006ce90c5d2ce97352b548400c59a5d0f67a



Same numpy error as before...

-Craig
SamStudio8 commented 2 years ago

Hi @craigwindell, thanks for confirming. Those look fine.

I now suspect that you have Python packages in your $HOME that might be automounted to the container by Nextflow and cause a clash with the container environment. Can you add the following lines to your custom Nextflow config and try again?

env {
  PYTHONNOUSERSITE = 1
}
craigwindell commented 2 years ago

Hi @SamStudio8, Bingo! I did have leakage from my home folder into the singularity container.

Thank you for your time helping me diagnose this issue.

Your suggestion solved the problem.

-Craig

SamStudio8 commented 2 years ago

Glad that's working for you now! I'll look at mitigating this in future, probably by adding that env block to our standard nextflow.config.