search

a-ludi / dentist

Close assembly gaps using long-reads at high accuracy.
https://a-ludi.github.io/dentist/
MIT License
47 stars 6 forks source link

Failed to run with test data set: TypeError in line 45 of Snakefile #24

Open davidjstudholme opened 3 years ago

davidjstudholme commented 3 years ago 
(base) ubuntu@bio-xanthomonas:~/dentist-example$ snakemake --configfile=snakemake.yml --use-singularity --cores=all
Pre-fetching singularity image...
TypeError in line 45 of /home/djs217/dentist-example/Snakefile:
__init__() got an unexpected keyword argument 'is_containerized'
  File "/home/djs217/dentist-example/Snakefile", line 745, in <module>
  File "/home/djs217/dentist-example/Snakefile", line 45, in prefetch_singularity_image

(base) ubuntu@bio-xanthomonas:~/dentist-example$ snakemake --version
5.32.1
a-ludi commented 3 years ago

Oh man, I have fixed all these issues but not in that release. Please edit line 45 of Snakefile and remove , is_containerized=True. That fixes it.

a-ludi commented 3 years ago

@davidjstudholme Did this work? BTW you should consider updating to v2.0.0 of DENTIST. :wink:

davidjstudholme commented 3 years ago

I have not tried it yet. I thought it better to wait until the release of DENTIST that has these issues resolved.

davidjstudholme commented 3 years ago

This is what happened. I have not yet tried v2.0.0. Will try that next.

(base) ubuntu@bio-xanthomonas:~/dentist-example$ snakemake --configfile=snakemake.yml --use-singularity --cores=all
Pre-fetching singularity image...
Building DAG of jobs...
Updating job mask_dust.
Updating job tandem_alignment_block.
Updating job mask_tandem_block.
Updating job mask_tandem.
Updating job self_alignment_block.
Updating job self_alignment.
Updating job mask_self.
Updating job ref_vs_reads_alignment_block.
Updating job propagate_mask_to_reads_block.
Updating job propagate_mask_back_to_reference_block.
Updating job ref_vs_reads_alignment_block.
Updating job propagate_mask_to_reads_block.
Updating job propagate_mask_back_to_reference_block.
Updating job ref_vs_reads_alignment_block.
Updating job propagate_mask_to_reads_block.
Updating job propagate_mask_back_to_reference_block.
Updating job ref_vs_reads_alignment_block.
Updating job propagate_mask_to_reads_block.
Updating job propagate_mask_back_to_reference_block.
Updating job ref_vs_reads_alignment_block.
Updating job propagate_mask_to_reads_block.
Updating job propagate_mask_back_to_reference_block.
Updating job ref_vs_reads_alignment_block.
Updating job propagate_mask_to_reads_block.
Updating job propagate_mask_back_to_reference_block.
Updating job ref_vs_reads_alignment_block.
Updating job propagate_mask_to_reads_block.
Updating job propagate_mask_back_to_reference_block.
Updating job ref_vs_reads_alignment_block.
Updating job propagate_mask_to_reads_block.
Updating job propagate_mask_back_to_reference_block.
Updating job ref_vs_reads_alignment_block.
Updating job propagate_mask_to_reads_block.
Updating job propagate_mask_back_to_reference_block.
Updating job ref_vs_reads_alignment_block.
Updating job propagate_mask_to_reads_block.
Updating job propagate_mask_back_to_reference_block.
Updating job ref_vs_reads_alignment_block.
Updating job propagate_mask_to_reads_block.
Updating job propagate_mask_back_to_reference_block.
Updating job ref_vs_reads_alignment_block.
Updating job propagate_mask_to_reads_block.
Updating job propagate_mask_back_to_reference_block.
Updating job propagate_mask_batch.
Updating job propagate_mask_back_to_reference.
Updating job propagate_mask_to_reads_block.
Updating job propagate_mask_back_to_reference_block.
Updating job propagate_mask_to_reads_block.
Updating job propagate_mask_back_to_reference_block.
Updating job propagate_mask_to_reads_block.
Updating job propagate_mask_back_to_reference_block.
Updating job propagate_mask_to_reads_block.
Updating job propagate_mask_back_to_reference_block.
Updating job propagate_mask_to_reads_block.
Updating job propagate_mask_back_to_reference_block.
Updating job propagate_mask_to_reads_block.
Updating job propagate_mask_back_to_reference_block.
Updating job propagate_mask_to_reads_block.
Updating job propagate_mask_back_to_reference_block.
Updating job propagate_mask_to_reads_block.
Updating job propagate_mask_back_to_reference_block.
Updating job propagate_mask_to_reads_block.
Updating job propagate_mask_back_to_reference_block.
Updating job propagate_mask_to_reads_block.
Updating job propagate_mask_back_to_reference_block.
Updating job propagate_mask_to_reads_block.
Updating job propagate_mask_back_to_reference_block.
Updating job propagate_mask_to_reads_block.
Updating job propagate_mask_back_to_reference_block.
Updating job propagate_mask_batch.
Updating job propagate_mask_back_to_reference.
Updating job ref_vs_reads_alignment.
Updating job mask_reads.
Updating job propagate_mask_to_reads_block.
Updating job propagate_mask_back_to_reference_block.
Updating job propagate_mask_to_reads_block.
Updating job propagate_mask_back_to_reference_block.
Updating job propagate_mask_to_reads_block.
Updating job propagate_mask_back_to_reference_block.
Updating job propagate_mask_to_reads_block.
Updating job propagate_mask_back_to_reference_block.
Updating job propagate_mask_to_reads_block.
Updating job propagate_mask_back_to_reference_block.
Updating job propagate_mask_to_reads_block.
Updating job propagate_mask_back_to_reference_block.
Updating job propagate_mask_to_reads_block.
Updating job propagate_mask_back_to_reference_block.
Updating job propagate_mask_to_reads_block.
Updating job propagate_mask_back_to_reference_block.
Updating job propagate_mask_to_reads_block.
Updating job propagate_mask_back_to_reference_block.
Updating job propagate_mask_to_reads_block.
Updating job propagate_mask_back_to_reference_block.
Updating job propagate_mask_to_reads_block.
Updating job propagate_mask_back_to_reference_block.
Updating job propagate_mask_to_reads_block.
Updating job propagate_mask_back_to_reference_block.
Updating job propagate_mask_batch.
Updating job propagate_mask_back_to_reference.
Updating job collect.
Updating job process.
Updating job process.
Updating job process.
Updating job preliminary_output.
Updating job purged_output.
Updating job preliminary_gap_closed_vs_reads_alignment_block.
Updating job preliminary_gap_closed_vs_reads_alignment_block.
Updating job preliminary_gap_closed_vs_reads_alignment_block.
Updating job preliminary_gap_closed_vs_reads_alignment_block.
Updating job preliminary_gap_closed_vs_reads_alignment_block.
Updating job preliminary_gap_closed_vs_reads_alignment_block.
Updating job preliminary_gap_closed_vs_reads_alignment_block.
Updating job preliminary_gap_closed_vs_reads_alignment_block.
Updating job preliminary_gap_closed_vs_reads_alignment_block.
Updating job preliminary_gap_closed_vs_reads_alignment_block.
Updating job preliminary_gap_closed_vs_reads_alignment_block.
Updating job preliminary_gap_closed_vs_reads_alignment_block.
Updating job preliminary_gap_closed_vs_reads_alignment.
Updating job validate_regions_block.
Updating job validate_regions_block.
Updating job ref_vs_reads_alignment_block.
Updating job propagate_mask_to_reads_block.
Updating job propagate_mask_back_to_reference_block.
Updating job ref_vs_reads_alignment_block.
Updating job propagate_mask_to_reads_block.
Updating job propagate_mask_back_to_reference_block.
Updating job ref_vs_reads_alignment_block.
Updating job propagate_mask_to_reads_block.
Updating job propagate_mask_back_to_reference_block.
Updating job ref_vs_reads_alignment_block.
Updating job propagate_mask_to_reads_block.
Updating job propagate_mask_back_to_reference_block.
Updating job ref_vs_reads_alignment_block.
Updating job propagate_mask_to_reads_block.
Updating job propagate_mask_back_to_reference_block.
Updating job ref_vs_reads_alignment_block.
Updating job propagate_mask_to_reads_block.
Updating job propagate_mask_back_to_reference_block.
Updating job ref_vs_reads_alignment_block.
Updating job propagate_mask_to_reads_block.
Updating job propagate_mask_back_to_reference_block.
Updating job ref_vs_reads_alignment_block.
Updating job propagate_mask_to_reads_block.
Updating job propagate_mask_back_to_reference_block.
Updating job ref_vs_reads_alignment_block.
Updating job propagate_mask_to_reads_block.
Updating job propagate_mask_back_to_reference_block.
Updating job ref_vs_reads_alignment_block.
Updating job propagate_mask_to_reads_block.
Updating job propagate_mask_back_to_reference_block.
Updating job ref_vs_reads_alignment_block.
Updating job propagate_mask_to_reads_block.
Updating job propagate_mask_back_to_reference_block.
Updating job ref_vs_reads_alignment_block.
Updating job propagate_mask_to_reads_block.
Updating job propagate_mask_back_to_reference_block.
Updating job propagate_mask_batch.
Updating job propagate_mask_to_reads_block.
Updating job propagate_mask_back_to_reference_block.
Updating job propagate_mask_to_reads_block.
Updating job propagate_mask_back_to_reference_block.
Updating job propagate_mask_to_reads_block.
Updating job propagate_mask_back_to_reference_block.
Updating job propagate_mask_to_reads_block.
Updating job propagate_mask_back_to_reference_block.
Updating job propagate_mask_to_reads_block.
Updating job propagate_mask_back_to_reference_block.
Updating job propagate_mask_to_reads_block.
Updating job propagate_mask_back_to_reference_block.
Updating job propagate_mask_to_reads_block.
Updating job propagate_mask_back_to_reference_block.
Updating job propagate_mask_to_reads_block.
Updating job propagate_mask_back_to_reference_block.
Updating job propagate_mask_to_reads_block.
Updating job propagate_mask_back_to_reference_block.
Updating job propagate_mask_to_reads_block.
Updating job propagate_mask_back_to_reference_block.
Updating job propagate_mask_to_reads_block.
Updating job propagate_mask_back_to_reference_block.
Updating job propagate_mask_to_reads_block.
Updating job propagate_mask_back_to_reference_block.
Updating job propagate_mask_batch.
Updating job ref_vs_reads_alignment.
Updating job mask_reads.
Updating job propagate_mask_to_reads_block.
Updating job propagate_mask_back_to_reference_block.
Updating job propagate_mask_to_reads_block.
Updating job propagate_mask_back_to_reference_block.
Updating job propagate_mask_to_reads_block.
Updating job propagate_mask_back_to_reference_block.
Updating job propagate_mask_to_reads_block.
Updating job propagate_mask_back_to_reference_block.
Updating job propagate_mask_to_reads_block.
Updating job propagate_mask_back_to_reference_block.
Updating job propagate_mask_to_reads_block.
Updating job propagate_mask_back_to_reference_block.
Updating job propagate_mask_to_reads_block.
Updating job propagate_mask_back_to_reference_block.
Updating job propagate_mask_to_reads_block.
Updating job propagate_mask_back_to_reference_block.
Updating job propagate_mask_to_reads_block.
Updating job propagate_mask_back_to_reference_block.
Updating job propagate_mask_to_reads_block.
Updating job propagate_mask_back_to_reference_block.
Updating job propagate_mask_to_reads_block.
Updating job propagate_mask_back_to_reference_block.
Updating job propagate_mask_to_reads_block.
Updating job propagate_mask_back_to_reference_block.
Updating job propagate_mask_batch.
Updating job collect.
Updating job process.
Updating job process.
Updating job process.
Using shell: /bin/bash
Provided cores: 16
Rules claiming more threads will be scaled down.
Job counts:
    count   jobs
    1   ALL
    1   collect
    1   make_merge_config
    1   mark_assembly_preliminary_gap_closed
    1   mask_dust
    1   mask_self
    1   mask_tandem
    1   mask_tandem_block
    1   merge_insertions
    1   preliminary_closed_gaps_bed2mask
    1   preliminary_gap_closed2dam
    1   preliminary_gap_closed_vs_reads_alignment
    12  preliminary_gap_closed_vs_reads_alignment_block
    1   preliminary_output
    3   process
    1   purged_output
    1   self_alignment
    1   self_alignment_block
    1   skip_gaps
    1   split_preliminary_gap_closed_vs_reads_alignment
    1   tandem_alignment_block
    1   validate_regions
    2   validate_regions_block
    1   weak_coverage_mask
    38
Select jobs to execute...
ProtectedOutputException in line 1230 of /home/djs217/dentist-example/Snakefile:
Write-protected output files for rule collect:
workdir/pile-ups.db
  File "/home/djs217/mambaforge/lib/python3.9/site-packages/snakemake/executors/__init__.py", line 136, in run_jobs
  File "/home/djs217/mambaforge/lib/python3.9/site-packages/snakemake/executors/__init__.py", line 441, in run
  File "/home/djs217/mambaforge/lib/python3.9/site-packages/snakemake/executors/__init__.py", line 230, in _run
  File "/home/djs217/mambaforge/lib/python3.9/site-packages/snakemake/executors/__init__.py", line 155, in _run
davidjstudholme commented 3 years ago

I would like to try your suggestion of using v2.0.0. However, it is not clear how to achieve that. I first removed the cached version of DENTIST in order that the latest container would be used:

(base) ubuntu@bio-xanthomonas:~/dentist-example$ singularity cache clean
This will delete everything in your cache (containers from all sources and OCI blobs). 
Hint: You can see exactly what would be deleted by canceling and using the --dry-run option.
Do you want to continue? [N/y] y
INFO:    Removing blob cache entry: blobs
INFO:    Removing blob cache entry: index.json
INFO:    Removing blob cache entry: oci-layout
INFO:    No cached files to remove at /home/ubuntu/.singularity/cache/library
INFO:    Removing oci-tmp cache entry: 3c6fdfdea388b95d5940f2623c316f44dd1a27b990bb7c61a02d4f4363ca94ed
INFO:    No cached files to remove at /home/ubuntu/.singularity/cache/shub
INFO:    No cached files to remove at /home/ubuntu/.singularity/cache/oras
INFO:    No cached files to remove at /home/ubuntu/.singularity/cache/net

However, when I then executed a command to check DENTIST version via singularity, it reported version 1.0.2-1-gd85a86f: 

(base) ubuntu@bio-xanthomonas:~/dentist-example$ singularity exec docker://aludi/dentist:stable dentist --version
INFO:    Converting OCI blobs to SIF format
INFO:    Starting build...
Getting image source signatures
Copying blob 596ba82af5aa done  
Copying blob 7b86bff36c77 done  
Copying blob 331ea36ef737 done  
Copying blob cdc4901af31b done  
Copying blob c0d6090ea70a done  
Copying blob 0c84452243d1 done  
Copying blob 54ecc0ee587b done  
Copying blob 611103b2d057 done  
Copying blob 4845adf8dffa done  
Copying blob 46c445d1f136 done  
Copying blob 97e58e63575a done  
Copying blob 7c3c3027a611 done  
Copying blob 6043df05ca09 done  
Copying blob a27fdc5c6b08 done  
Copying config 9a16d63ba7 done  
Writing manifest to image destination
Storing signatures
2021/06/23 09:51:51  info unpack layer: sha256:596ba82af5aaa3e2fd9d6f955b8b94f0744a2b60710e3c243ba3e4a467f051d1
2021/06/23 09:51:51  info unpack layer: sha256:7b86bff36c77cb27274c28d3f6b84b0538151a48459bad5389408e4cb903907a
2021/06/23 09:51:55  info unpack layer: sha256:331ea36ef737b546dffca18072cd6b29f4ee279c124a83fe6b26bba09918183b
2021/06/23 09:51:55  info unpack layer: sha256:cdc4901af31bd626b25a72ec2128ef282ff91a36b47dd9f7162f923434e4cf0b
2021/06/23 09:51:56  info unpack layer: sha256:c0d6090ea70a95526c02c71409d2f6602c23d1a080fe0e52735d3434f08fc31f
2021/06/23 09:51:58  info unpack layer: sha256:0c84452243d175f3725f6507c9d88d0ca4883ed5ecfdabf4941303dcbcab1ddb
2021/06/23 09:51:58  info unpack layer: sha256:54ecc0ee587bdec75f64c1681b923604ddd94214856607d75cfaedc8080f931e
2021/06/23 09:51:58  info unpack layer: sha256:611103b2d0579911f1ae34d6815d121b2efde4aeadb11f73224bca3dccda583c
2021/06/23 09:51:58  info unpack layer: sha256:4845adf8dffa298b5ad21cb242b65e6a28d970dcd4040a88776d050a14ba3a05
2021/06/23 09:51:58  info unpack layer: sha256:46c445d1f1367c357c4fcce9d0f8d6987033b61ad23e113aac7b03da12ba3e46
2021/06/23 09:51:58  info unpack layer: sha256:97e58e63575a7469dd7b2ca7773b4338b7141525584238b8b1d7368cb1bf211d
2021/06/23 09:51:59  info unpack layer: sha256:7c3c3027a611f72c5b153bf7dc1640a42f3a932ac0a66b7fdc77cf442fdfaa63
2021/06/23 09:51:59  info unpack layer: sha256:6043df05ca095f9aa28479565659ad11086a6b249b2f19efdbed0fb674547069
2021/06/23 09:51:59  info unpack layer: sha256:a27fdc5c6b08ac9b2a9a216bc5a54fd3421ed7b8722fc93f532c28113d0f6a82
INFO:    Creating SIF file...
dentist v1.0.2-1-gd85a86f (commit d85a86fda8da241b0de3d3b8d3b02cf9e3405302)

Copyright © 2018, Arne Ludwig <arne.ludwig@posteo.de>

Subject to the terms of the MIT license, as written in the included LICENSE file
a-ludi commented 3 years ago

Interesting. The same happens when I retrace your actions. I am no Singularity expert and have no idea why this happens. The image in the Docker registry is up-to-date:

$ docker image rm aludi/dentist:latest
Untagged: aludi/dentist:latest
$ docker pull aludi/dentist:latest
latest: Pulling from aludi/dentist
Digest: sha256:e2c2b3dfbbd08e2234e2df8ec24282a4af1b025292abc906ba751d4306f92567
Status: Downloaded newer image for aludi/dentist:latest
docker.io/aludi/dentist:latest
$ docker run --rm aludi/dentist:latest dentist --version
dentist v2.0.0 (commit e7659a9f23178c952b21da14da72f1aa6a812675)

Copyright © 2018, Arne Ludwig <arne.ludwig@posteo.de>

Subject to the terms of the MIT license, as written in the included LICENSE file

However, there is a simple solution. Just download the SIF file from the releases section and instruct Snakemake to use it by providing the path in snakemake.yml with dentist_container: /path/to/dentist_v2.0.0.sif.