Failing at Medaka consensus stage both with condo and singularity

[callumparr]

❯ nextflow run epi2me-labs/wf-clone-validation --fastq .nextflow/assets/epi2me-labs/wf-clone-validation/test_data/test --samples .nextflow/assets/epi2me-labs/wf-clone-validation/test_data/sample_sheet.txt --out_dir my_clone_validation_experiment -profile conda --db_directory wf-clone-validation-db

N E X T F L O W  ~  version 21.10.6
Launching `epi2me-labs/wf-clone-validation` [hungry_stone] - revision: b7fd9cd569 [master]

WARN: Found unexpected parameters:
* --samples: .nextflow/assets/epi2me-labs/wf-clone-validation/test_data/sample_sheet.txt
- Ignore this warning: params.schema_ignore_params = "samples" 

Core Nextflow options
  revision    : master
  runName     : hungry_stone
  launchDir   : /home/callum
  workDir     : /home/callum/work
  projectDir  : /home/callum/.nextflow/assets/epi2me-labs/wf-clone-validation
  userName    : callum
  profile     : conda
  configFiles : /home/callum/.nextflow/assets/epi2me-labs/wf-clone-validation/nextflow.config

Input/output options
  fastq       : .nextflow/assets/epi2me-labs/wf-clone-validation/test_data/test
  out_dir     : my_clone_validation_experiment
  primers     : /home/callum/.nextflow/assets/epi2me-labs/wf-clone-validation/data/primers.tsv

Reference genome options
  db_directory: wf-clone-validation-db

!! Only displaying parameters that differ from the pipeline defaults !!
------------------------------------------------------
If you use wf-clone-validation for your analysis please cite:

* The nf-core framework
  https://doi.org/10.1038/s41587-020-0439-x

[-        ] process > start_ping:pingMessage -
[-        ] process > start_ping:pingMessage        -
executor >  slurm (1)
executor >  slurm (4)
executor >  slurm (6)
executor >  slurm (6)
executor >  slurm (6)
executor >  slurm (10)
[fd/9cdfe9] process > start_ping:pingMessage (1)        [100%] 1 of 1 ✔
[da/3f181c] process > pipeline:combineFastq (2)         [100%] 3 of 3 ✔
[c7/b62e48] process > pipeline:assembleCore (1)         [100%] 1 of 1 ✔
[f1/9f9d3a] process > pipeline:medakaPolishAssembly (1) [  0%] 0 of 1
[e2/bce954] process > pipeline:downsampledStats (1)     [100%] 1 of 1 ✔
[-        ] process > pipeline:assemblyMafs             -
[-        ] process > pipeline:findPrimers              -
[fc/533e09] process > pipeline:getVersions              [100%] 1 of 1 ✔
[ae/9dbdf4] process > pipeline:getParams                [100%] 1 of 1 ✔
[-        ] process > pipeline:report                   -
[-        ] process > output                            -
[15/b43d42] process > end_ping:pingMessage              [100%] 1 of 1 ✔
Error executing process > 'pipeline:medakaPolishAssembly (1)'

Caused by:
  Process `pipeline:medakaPolishAssembly (1)` terminated with an error exit status (1)

Command executed:

  STATUS=barcode01",Failed to polish assembly with Medaka"
  medaka_consensus -i barcode01.downsampled.fastq -d barcode01.reconciled.fasta -m r941_min_high_g360 -o . -t 4 -f
  echo ">barcode01" >> barcode01.final.fasta
  sed "2q;d" consensus.fasta >> barcode01.final.fasta
  STATUS=barcode01",Completed successfully"

Command exit status:
  1

Command output:
  Checking program versions
  This is medaka 1.2.3
  Program    Version    Required   Pass     
  bcftools   1.11       1.9        True     
  bgzip      1.11       1.9        True     
  minimap2   2.24       2.11       True     
  samtools   1.11       1.9        True     
  tabix      1.11       1.9        True     
  Warning: Output will be overwritten (-f flag)
  Aligning basecalls to draft
  Removing previous index file /home/callum/work/c7/b62e483f32f4ef969fae84ff3b15ac/barcode01.reconciled.fasta.mmi
  Removing previous index file /home/callum/work/c7/b62e483f32f4ef969fae84ff3b15ac/barcode01.reconciled.fasta.fai
  Running medaka consensus
  Failed to run medaka consensus.

Command error:
  Constructing minimap index.
  [M::mm_idx_gen::0.002*1.35] collected minimizers
  [M::mm_idx_gen::0.003*1.58] sorted minimizers
  [M::main::0.004*1.40] loaded/built the index for 1 target sequence(s)
  [M::mm_idx_stat] kmer size: 15; skip: 10; is_hpc: 0; #seq: 1
executor >  slurm (10)
[fd/9cdfe9] process > start_ping:pingMessage (1)        [100%] 1 of 1 ✔
[da/3f181c] process > pipeline:combineFastq (2)         [100%] 3 of 3 ✔
[c7/b62e48] process > pipeline:assembleCore (1)         [100%] 1 of 1 ✔
[f1/9f9d3a] process > pipeline:medakaPolishAssembly (1) [100%] 1 of 1, failed: 1 ✘
[e2/bce954] process > pipeline:downsampledStats (1)     [100%] 1 of 1 ✔
[-        ] process > pipeline:assemblyMafs             -
[-        ] process > pipeline:findPrimers              -
[fc/533e09] process > pipeline:getVersions              [100%] 1 of 1 ✔
[ae/9dbdf4] process > pipeline:getParams                [100%] 1 of 1 ✔
[-        ] process > pipeline:report                   -
[-        ] process > output                            -
[15/b43d42] process > end_ping:pingMessage              [100%] 1 of 1 ✔
Error executing process > 'pipeline:medakaPolishAssembly (1)'

Caused by:
  Process `pipeline:medakaPolishAssembly (1)` terminated with an error exit status (1)

Command executed:

  STATUS=barcode01",Failed to polish assembly with Medaka"
  medaka_consensus -i barcode01.downsampled.fastq -d barcode01.reconciled.fasta -m r941_min_high_g360 -o . -t 4 -f
  echo ">barcode01" >> barcode01.final.fasta
  sed "2q;d" consensus.fasta >> barcode01.final.fasta
  STATUS=barcode01",Completed successfully"

Command exit status:
  1

Command output:
  Checking program versions
  This is medaka 1.2.3
  Program    Version    Required   Pass     
  bcftools   1.11       1.9        True     
  bgzip      1.11       1.9        True     
  minimap2   2.24       2.11       True     
  samtools   1.11       1.9        True     
  tabix      1.11       1.9        True     
  Warning: Output will be overwritten (-f flag)
  Aligning basecalls to draft
  Removing previous index file /home/callum/work/c7/b62e483f32f4ef969fae84ff3b15ac/barcode01.reconciled.fasta.mmi
  Removing previous index file /home/callum/work/c7/b62e483f32f4ef969fae84ff3b15ac/barcode01.reconciled.fasta.fai
  Running medaka consensus
  Failed to run medaka consensus.

Command error:
  Constructing minimap index.
  [M::mm_idx_gen::0.002*1.35] collected minimizers
  [M::mm_idx_gen::0.003*1.58] sorted minimizers
  [M::main::0.004*1.40] loaded/built the index for 1 target sequence(s)
  [M::mm_idx_stat] kmer size: 15; skip: 10; is_hpc: 0; #seq: 1
  [M::mm_idx_stat::0.004*1.39] distinct minimizers: 1001 (100.00% are singletons); average occurrences: 1.000; average spacing: 5.377; total length: 5382
  [M::main] Version: 2.24-r1122
  [M::main] CMD: minimap2 -I 16G -x map-ont --MD -d /home/callum/work/c7/b62e483f32f4ef969fae84ff3b15ac/barcode01.reconciled.fasta.mmi /home/callum/work/c7/b62e483f32f4ef969fae84ff3b15ac/barcode01.reconciled.fasta
  [M::main] Real time: 0.006 sec; CPU: 0.006 sec; Peak RSS: 0.003 GB
  [M::main::0.003*1.28] loaded/built the index for 1 target sequence(s)
  [M::mm_mapopt_update::0.003*1.27] mid_occ = 10
  [M::mm_idx_stat] kmer size: 15; skip: 10; is_hpc: 0; #seq: 1
  [M::mm_idx_stat::0.003*1.26] distinct minimizers: 1001 (100.00% are singletons); average occurrences: 1.000; average spacing: 5.377; total length: 5382
  [M::worker_pipeline::0.098*3.38] mapped 113 sequences
  [M::main] Version: 2.24-r1122
  [M::main] CMD: minimap2 -x map-ont --MD -t 4 -a -A 2 -B 4 -O 4,24 -E 2,1 /home/callum/work/c7/b62e483f32f4ef969fae84ff3b15ac/barcode01.reconciled.fasta.mmi /home/callum/work/c7/b62e483f32f4ef969fae84ff3b15ac/barcode01.downsampled.fastq
  [M::main] Real time: 0.099 sec; CPU: 0.334 sec; Peak RSS: 0.008 GB
  [13:56:50 - Predict] Reducing threads to 2, anymore is a waste.
  [13:56:50 - Predict] It looks like you are running medaka without a GPU and attempted to set a high number of threads. We have scaled this down to an optimal number. If you wish to improve performance please see https://nanoporetech.github.io/medaka/installation.html#improving-parallelism.
  [13:56:50 - Predict] Setting tensorflow inter/intra-op threads to 2/1.
  [13:56:50 - Predict] Processing region(s): cluster_001_consensus:0-5382
  [13:56:50 - Predict] Using model: /home/callum/work/conda/epi2melabs-wf-clone-validation-2feb60f4a8e4ee12789121e375386ea6/lib/python3.6/site-packages/medaka/data/r941_min_high_g360_model.hdf5.
  [13:56:50 - Predict] Processing 1 long region(s) with batching.
  [13:56:50 - ModelStore] filepath /home/callum/work/conda/epi2melabs-wf-clone-validation-2feb60f4a8e4ee12789121e375386ea6/lib/python3.6/site-packages/medaka/data/r941_min_high_g360_model.hdf5
  [13:56:52 - ModelStore] ModelStore exception <class 'AttributeError'>
  Traceback (most recent call last):
    File "/home/callum/work/conda/epi2melabs-wf-clone-validation-2feb60f4a8e4ee12789121e375386ea6/bin/medaka", line 11, in <module>
      sys.exit(main())
    File "/home/callum/work/conda/epi2melabs-wf-clone-validation-2feb60f4a8e4ee12789121e375386ea6/lib/python3.6/site-packages/medaka/medaka.py", line 684, in main
      args.func(args)
    File "/home/callum/work/conda/epi2melabs-wf-clone-validation-2feb60f4a8e4ee12789121e375386ea6/lib/python3.6/site-packages/medaka/prediction.py", line 157, in predict
      model = model_store.load_model(time_steps=args.chunk_len)
    File "/home/callum/work/conda/epi2melabs-wf-clone-validation-2feb60f4a8e4ee12789121e375386ea6/lib/python3.6/site-packages/medaka/datastore.py", line 78, in load_model
      model.load_weights(self.filepath)
    File "/home/callum/work/conda/epi2melabs-wf-clone-validation-2feb60f4a8e4ee12789121e375386ea6/lib/python3.6/site-packages/tensorflow/python/keras/engine/training.py", line 250, in load_weights
      return super(Model, self).load_weights(filepath, by_name, skip_mismatch)
    File "/home/callum/work/conda/epi2melabs-wf-clone-validation-2feb60f4a8e4ee12789121e375386ea6/lib/python3.6/site-packages/tensorflow/python/keras/engine/network.py", line 1266, in load_weights
      hdf5_format.load_weights_from_hdf5_group(f, self.layers)
    File "/home/callum/work/conda/epi2melabs-wf-clone-validation-2feb60f4a8e4ee12789121e375386ea6/lib/python3.6/site-packages/tensorflow/python/keras/saving/hdf5_format.py", line 659, in load_weights_from_hdf5_group
      original_keras_version = f.attrs['keras_version'].decode('utf8')
  AttributeError: 'str' object has no attribute 'decode'

Work dir:
  /home/callum/work/f1/9f9d3a83e9b4de5e2a20df2d89c1c7

Tip: you can replicate the issue by changing to the process work dir and entering the command `bash .command.run`

move to the particular workspace and run the command for specific error message

❯ bash .command.run
Checking program versions
This is medaka 1.2.3
Program    Version    Required   Pass     
bcftools   1.11       1.9        True     
bgzip      1.11       1.9        True     
minimap2   2.24       2.11       True     
samtools   1.11       1.9        True     
tabix      1.11       1.9        True     
Warning: Output will be overwritten (-f flag)
Aligning basecalls to draft
Removing previous index file /home/callum/work/c7/b62e483f32f4ef969fae84ff3b15ac/barcode01.reconciled.fasta.mmi
Removing previous index file /home/callum/work/c7/b62e483f32f4ef969fae84ff3b15ac/barcode01.reconciled.fasta.fai
Constructing minimap index.
[M::mm_idx_gen::0.002*1.79] collected minimizers
[M::mm_idx_gen::0.002*1.93] sorted minimizers
[M::main::0.004*1.62] loaded/built the index for 1 target sequence(s)
[M::mm_idx_stat] kmer size: 15; skip: 10; is_hpc: 0; #seq: 1
[M::mm_idx_stat::0.004*1.60] distinct minimizers: 1001 (100.00% are singletons); average occurrences: 1.000; average spacing: 5.377; total length: 5382
[M::main] Version: 2.24-r1122
[M::main] CMD: minimap2 -I 16G -x map-ont --MD -d /home/callum/work/c7/b62e483f32f4ef969fae84ff3b15ac/barcode01.reconciled.fasta.mmi /home/callum/work/c7/b62e483f32f4ef969fae84ff3b15ac/barcode01.reconciled.fasta
[M::main] Real time: 0.006 sec; CPU: 0.006 sec; Peak RSS: 0.003 GB
[M::main::0.003*1.26] loaded/built the index for 1 target sequence(s)
[M::mm_mapopt_update::0.003*1.24] mid_occ = 10
[M::mm_idx_stat] kmer size: 15; skip: 10; is_hpc: 0; #seq: 1
[M::mm_idx_stat::0.003*1.23] distinct minimizers: 1001 (100.00% are singletons); average occurrences: 1.000; average spacing: 5.377; total length: 5382
[M::worker_pipeline::0.252*1.27] mapped 113 sequences
[M::main] Version: 2.24-r1122
[M::main] CMD: minimap2 -x map-ont --MD -t 4 -a -A 2 -B 4 -O 4,24 -E 2,1 /home/callum/work/c7/b62e483f32f4ef969fae84ff3b15ac/barcode01.reconciled.fasta.mmi /home/callum/work/c7/b62e483f32f4ef969fae84ff3b15ac/barcode01.downsampled.fastq
[M::main] Real time: 0.253 sec; CPU: 0.321 sec; Peak RSS: 0.008 GB
Running medaka consensus
[15:15:42 - Predict] Reducing threads to 2, anymore is a waste.
[15:15:42 - Predict] It looks like you are running medaka without a GPU and attempted to set a high number of threads. We have scaled this down to an optimal number. If you wish to improve performance please see https://nanoporetech.github.io/medaka/installation.html#improving-parallelism.
[15:15:42 - Predict] Setting tensorflow inter/intra-op threads to 2/1.
[15:15:42 - Predict] Processing region(s): cluster_001_consensus:0-5382
[15:15:42 - Predict] Using model: /home/callum/work/conda/epi2melabs-wf-clone-validation-2feb60f4a8e4ee12789121e375386ea6/lib/python3.6/site-packages/medaka/data/r941_min_high_g360_model.hdf5.
[15:15:42 - Predict] Processing 1 long region(s) with batching.
[15:15:42 - ModelStore] filepath /home/callum/work/conda/epi2melabs-wf-clone-validation-2feb60f4a8e4ee12789121e375386ea6/lib/python3.6/site-packages/medaka/data/r941_min_high_g360_model.hdf5
[15:15:44 - ModelStore] ModelStore exception <class 'AttributeError'>
Traceback (most recent call last):
  File "/home/callum/work/conda/epi2melabs-wf-clone-validation-2feb60f4a8e4ee12789121e375386ea6/bin/medaka", line 11, in <module>
    sys.exit(main())
  File "/home/callum/work/conda/epi2melabs-wf-clone-validation-2feb60f4a8e4ee12789121e375386ea6/lib/python3.6/site-packages/medaka/medaka.py", line 684, in main
    args.func(args)
  File "/home/callum/work/conda/epi2melabs-wf-clone-validation-2feb60f4a8e4ee12789121e375386ea6/lib/python3.6/site-packages/medaka/prediction.py", line 157, in predict
    model = model_store.load_model(time_steps=args.chunk_len)
  File "/home/callum/work/conda/epi2melabs-wf-clone-validation-2feb60f4a8e4ee12789121e375386ea6/lib/python3.6/site-packages/medaka/datastore.py", line 78, in load_model
    model.load_weights(self.filepath)
  File "/home/callum/work/conda/epi2melabs-wf-clone-validation-2feb60f4a8e4ee12789121e375386ea6/lib/python3.6/site-packages/tensorflow/python/keras/engine/training.py", line 250, in load_weights
    return super(Model, self).load_weights(filepath, by_name, skip_mismatch)
  File "/home/callum/work/conda/epi2melabs-wf-clone-validation-2feb60f4a8e4ee12789121e375386ea6/lib/python3.6/site-packages/tensorflow/python/keras/engine/network.py", line 1266, in load_weights
    hdf5_format.load_weights_from_hdf5_group(f, self.layers)
  File "/home/callum/work/conda/epi2melabs-wf-clone-validation-2feb60f4a8e4ee12789121e375386ea6/lib/python3.6/site-packages/tensorflow/python/keras/saving/hdf5_format.py", line 659, in load_weights_from_hdf5_group
    original_keras_version = f.attrs['keras_version'].decode('utf8')
AttributeError: 'str' object has no attribute 'decode'
Failed to run medaka consensus.

[cjw85]

Hi Callum, I see you are using the conda profile. We are moving away toward not supporting this as we find it simply not reliable. (I believe the core developers of the nf-core project have similar views). The above error seems to be coming from a broken install of tensorflow and h5py.

I think it would be really worth working with your system administrators to figure out why things aren't working with docker for you, or failing that try to get things going with singularity.

[callumparr]

Hi Callum, I see you are using the conda profile. We are moving away toward not supporting this as we find it simply not reliable. (I believe the core developers of the nf-core project have similar views). The above error seems to be coming from a broken install of tensorflow and h5py.

I think it would be really worth working with your system administrators to figure out why things aren't working with docker for you, or failing that try to get things going with singularity.

Sorry I should have mentioned I used singularity as well but same error.

[cjw85]

Oh, that's interesting. We used this workflow ourselves quite heavily in our own research so I'd hope it was more robust.

@sarahjeeeze can you follow up here?

[callumparr]

I can post for singularity here.

I do not have any issue running this with conda/singularity or docker on my own ubuntu 20.04 system.

❯ nextflow run epi2me-labs/wf-clone-validation --fastq .nextflow/assets/epi2me-labs/wf-clone-validation/test_data/test --samples .nextflow/assets/epi2me-labs/wf-clone-validation/test_data/sample_sheet.txt --out_dir my_clone_validation_experiment -profile singularity --db_directory wf-clone-validation-db
zsh: correct 'singularity' to '.singularity' [nyae]? n

N E X T F L O W  ~  version 21.10.6
Launching `epi2me-labs/wf-clone-validation` [desperate_mahavira] - revision: b7fd9cd569 [master]

WARN: Found unexpected parameters:
* --samples: .nextflow/assets/epi2me-labs/wf-clone-validation/test_data/sample_sheet.txt
- Ignore this warning: params.schema_ignore_params = "samples" 

Core Nextflow options
  revision       : master
  runName        : desperate_mahavira
  containerEngine: singularity
  launchDir      : /home/callum
  workDir        : /home/callum/work
  projectDir     : /home/callum/.nextflow/assets/epi2me-labs/wf-clone-validation
  userName       : callum
  profile        : singularity
  configFiles    : /home/callum/.nextflow/assets/epi2me-labs/wf-clone-validation/nextflow.config

Input/output options
  fastq          : .nextflow/assets/epi2me-labs/wf-clone-validation/test_data/test
  out_dir        : my_clone_validation_experiment
  primers        : /home/callum/.nextflow/assets/epi2me-labs/wf-clone-validation/data/primers.tsv

Reference genome options
  db_directory   : wf-clone-validation-db

!! Only displaying parameters that differ from the pipeline defaults !!
------------------------------------------------------
If you use wf-clone-validation for your analysis please cite:

* The nf-core framework
  https://doi.org/10.1038/s41587-020-0439-x

[-        ] process > start_ping:pingMessage        -
[-        ] process > start_ping:pingMessage        -
executor >  local (1)
executor >  local (6)
executor >  local (10)
[e1/79e1c2] process > start_ping:pingMessage (1)        [100%] 1 of 1 ✔
[6d/8231ff] process > pipeline:combineFastq (2)         [100%] 3 of 3 ✔
[79/3ecc91] process > pipeline:assembleCore (1)         [100%] 1 of 1 ✔
[59/5210ed] process > pipeline:medakaPolishAssembly (1) [  0%] 0 of 1
[01/7e1f9d] process > pipeline:downsampledStats (1)     [100%] 1 of 1 ✔
[-        ] process > pipeline:assemblyMafs             -
[-        ] process > pipeline:findPrimers              -
[b2/dc2eb6] process > pipeline:getVersions              [100%] 1 of 1 ✔
[4a/62deac] process > pipeline:getParams                [100%] 1 of 1 ✔
[-        ] process > pipeline:report                   -
[-        ] process > output                            -
[d4/8557a6] process > end_ping:pingMessage              [100%] 1 of 1 ✔
Error executing process > 'pipeline:medakaPolishAssembly (1)'

Caused by:
  Process `pipeline:medakaPolishAssembly (1)` terminated with an error exit status (1)

Command executed:

  STATUS=barcode01",Failed to polish assembly with Medaka"
  medaka_consensus -i barcode01.downsampled.fastq -d barcode01.reconciled.fasta -m r941_min_high_g360 -o . -t 4 -f
  echo ">barcode01" >> barcode01.final.fasta
  sed "2q;d" consensus.fasta >> barcode01.final.fasta
  STATUS=barcode01",Completed successfully"

Command exit status:
  1

Command output:
  Checking program versions
  This is medaka 1.2.3
  Program    Version    Required   Pass     
  bcftools   1.11       1.9        True     
  bgzip      1.11       1.9        True     
  minimap2   2.24       2.11       True     
  samtools   1.11       1.9        True     
  tabix      1.11       1.9        True     
  Warning: Output will be overwritten (-f flag)
  Aligning basecalls to draft
  Removing previous index file /home/callum/work/79/3ecc913629fb8870b8e19ba3d07bf6/barcode01.reconciled.fasta.mmi
  Removing previous index file /home/callum/work/79/3ecc913629fb8870b8e19ba3d07bf6/barcode01.reconciled.fasta.fai
  Running medaka consensus
  Failed to run medaka consensus.

Command error:
  INFO:    Converting SIF file to temporary sandbox...
  WARNING: underlay of /etc/localtime required more than 50 (77) bind mounts
  Constructing minimap index.
executor >  local (10)
[e1/79e1c2] process > start_ping:pingMessage (1)        [100%] 1 of 1 ✔
[6d/8231ff] process > pipeline:combineFastq (2)         [100%] 3 of 3 ✔
[79/3ecc91] process > pipeline:assembleCore (1)         [100%] 1 of 1 ✔
[59/5210ed] process > pipeline:medakaPolishAssembly (1) [100%] 1 of 1, failed: 1 ✘
[01/7e1f9d] process > pipeline:downsampledStats (1)     [100%] 1 of 1 ✔
[-        ] process > pipeline:assemblyMafs             -
[-        ] process > pipeline:findPrimers              -
[b2/dc2eb6] process > pipeline:getVersions              [100%] 1 of 1 ✔
[4a/62deac] process > pipeline:getParams                [100%] 1 of 1 ✔
[-        ] process > pipeline:report                   -
[-        ] process > output                            -
[d4/8557a6] process > end_ping:pingMessage              [100%] 1 of 1 ✔
Error executing process > 'pipeline:medakaPolishAssembly (1)'

Caused by:
  Process `pipeline:medakaPolishAssembly (1)` terminated with an error exit status (1)

Command executed:

  STATUS=barcode01",Failed to polish assembly with Medaka"
  medaka_consensus -i barcode01.downsampled.fastq -d barcode01.reconciled.fasta -m r941_min_high_g360 -o . -t 4 -f
  echo ">barcode01" >> barcode01.final.fasta
  sed "2q;d" consensus.fasta >> barcode01.final.fasta
  STATUS=barcode01",Completed successfully"

Command exit status:
  1

Command output:
  Checking program versions
  This is medaka 1.2.3
  Program    Version    Required   Pass     
  bcftools   1.11       1.9        True     
  bgzip      1.11       1.9        True     
  minimap2   2.24       2.11       True     
  samtools   1.11       1.9        True     
  tabix      1.11       1.9        True     
  Warning: Output will be overwritten (-f flag)
  Aligning basecalls to draft
  Removing previous index file /home/callum/work/79/3ecc913629fb8870b8e19ba3d07bf6/barcode01.reconciled.fasta.mmi
  Removing previous index file /home/callum/work/79/3ecc913629fb8870b8e19ba3d07bf6/barcode01.reconciled.fasta.fai
  Running medaka consensus
  Failed to run medaka consensus.

Command error:
  INFO:    Converting SIF file to temporary sandbox...
  WARNING: underlay of /etc/localtime required more than 50 (77) bind mounts
  Constructing minimap index.
  [M::mm_idx_gen::0.002*1.25] collected minimizers
  [M::mm_idx_gen::0.003*1.48] sorted minimizers
  [M::main::0.004*1.34] loaded/built the index for 1 target sequence(s)
  [M::mm_idx_stat] kmer size: 15; skip: 10; is_hpc: 0; #seq: 1
  [M::mm_idx_stat::0.004*1.33] distinct minimizers: 1001 (100.00% are singletons); average occurrences: 1.000; average spacing: 5.377; total length: 5382
  [M::main] Version: 2.24-r1122
  [M::main] CMD: minimap2 -I 16G -x map-ont --MD -d /home/callum/work/79/3ecc913629fb8870b8e19ba3d07bf6/barcode01.reconciled.fasta.mmi /home/callum/work/79/3ecc913629fb8870b8e19ba3d07bf6/barcode01.reconciled.fasta
  [M::main] Real time: 0.006 sec; CPU: 0.006 sec; Peak RSS: 0.003 GB
  [M::main::0.003*0.95] loaded/built the index for 1 target sequence(s)
  [M::mm_mapopt_update::0.003*0.95] mid_occ = 10
  [M::mm_idx_stat] kmer size: 15; skip: 10; is_hpc: 0; #seq: 1
  [M::mm_idx_stat::0.003*0.95] distinct minimizers: 1001 (100.00% are singletons); average occurrences: 1.000; average spacing: 5.377; total length: 5382
  [M::worker_pipeline::0.098*3.42] mapped 113 sequences
  [M::main] Version: 2.24-r1122
  [M::main] CMD: minimap2 -x map-ont --MD -t 4 -a -A 2 -B 4 -O 4,24 -E 2,1 /home/callum/work/79/3ecc913629fb8870b8e19ba3d07bf6/barcode01.reconciled.fasta.mmi /home/callum/work/79/3ecc913629fb8870b8e19ba3d07bf6/barcode01.downsampled.fastq
  [M::main] Real time: 0.099 sec; CPU: 0.337 sec; Peak RSS: 0.008 GB
  [12:25:48 - Predict] Reducing threads to 2, anymore is a waste.
  [12:25:48 - Predict] It looks like you are running medaka without a GPU and attempted to set a high number of threads. We have scaled this down to an optimal number. If you wish to improve performance please see https://nanoporetech.github.io/medaka/installation.html#improving-parallelism.
  [12:25:48 - Predict] Setting tensorflow inter/intra-op threads to 2/1.
  [12:25:48 - Predict] Processing region(s): cluster_001_consensus:0-5382
  [12:25:48 - Predict] Using model: /home/epi2melabs/conda/lib/python3.6/site-packages/medaka/data/r941_min_high_g360_model.hdf5.
  [12:25:48 - Predict] Processing 1 long region(s) with batching.
  [12:25:48 - ModelStore] filepath /home/epi2melabs/conda/lib/python3.6/site-packages/medaka/data/r941_min_high_g360_model.hdf5
  [12:25:50 - ModelStore] ModelStore exception <class 'AttributeError'>
  Traceback (most recent call last):
    File "/home/epi2melabs/conda/bin/medaka", line 11, in <module>
      sys.exit(main())
    File "/home/epi2melabs/conda/lib/python3.6/site-packages/medaka/medaka.py", line 684, in main
      args.func(args)
    File "/home/epi2melabs/conda/lib/python3.6/site-packages/medaka/prediction.py", line 157, in predict
      model = model_store.load_model(time_steps=args.chunk_len)
    File "/home/epi2melabs/conda/lib/python3.6/site-packages/medaka/datastore.py", line 78, in load_model
      model.load_weights(self.filepath)
    File "/home/epi2melabs/conda/lib/python3.6/site-packages/tensorflow/python/keras/engine/training.py", line 250, in load_weights
      return super(Model, self).load_weights(filepath, by_name, skip_mismatch)
    File "/home/epi2melabs/conda/lib/python3.6/site-packages/tensorflow/python/keras/engine/network.py", line 1266, in load_weights
      hdf5_format.load_weights_from_hdf5_group(f, self.layers)
    File "/home/epi2melabs/conda/lib/python3.6/site-packages/tensorflow/python/keras/saving/hdf5_format.py", line 659, in load_weights_from_hdf5_group
      original_keras_version = f.attrs['keras_version'].decode('utf8')
  AttributeError: 'str' object has no attribute 'decode'
  INFO:    Cleaning up image...

Work dir:
  /home/callum/work/59/5210ed7868c4b23c108bc021fb56ed

Tip: view the complete command output by changing to the process work dir and entering the command `cat .command.out`

Unexpected error [ClosedByInterruptException]

[sarahjeeeze]

Hi, Thanks for the feedback. This should now be fixed in the latest version of the workflow.

[callumparr]

Hi, Thanks for the feedback. This should now be fixed in the latest version of the workflow.

Thank you for the update. We are in the process of updating our servers so will give this ago then.