search

oschwengers / asap

A scalable bacterial genome assembly, annotation and analysis pipeline
https://doi.org/10.1371/journal.pcbi.1007134
GNU General Public License v3.0
71 stars 19 forks source link
amr annotation assembly bacteria bioinformatics ngs

readme

DOI:10.1371/journal.pcbi.1007134 License: GPL v3 Don't judge me GitHub release Docker Pulls DOI

ASA³P - Automatic Bacterial Isolate Assembly, Annotation and Analyses Pipeline

ASA³P Overview

Contents

Description

ASA³P is an automatic and highly scalable assembly, annotation and higher-level analyses pipeline for closely related bacterial isolates.

ASA³P is a fully automatic, locally executable and scalable assembly, annotation and higher-level analysis pipeline creating results in standard bioinformatics file formats as well as sophisticated HTML5 documents. Its main purpose is the automatic processing of NGS WGS data of multiple closely related isolates, thus transforming raw reads into assembled and annotated genomes and finally gathering as much information on every single bacterial genome as possible. Per-isolate analyses are complemented by comparative insights. Therefore, the pipeline incorporates many best-in-class open source bioinformatics tools and thus minimizes the burden of ever-repeating tasks. Envisaged as a preprocessing tool it provides comprehensive insights as well as a general overview and comparison of analysed genomes along with all necessary result files for subsequent deeper analyses. All results are presented via modern HTML5 documents comprising interactive visualizations.

Features

Per isolate

Comparative

Availability

All necessary files are hosted at Zenodo: DOI

Targeting different project sizes, i.e. number of genomes which should be analysed as a single project, we distribute ASA³P in two versions:

For both the following files are necessary:

Note: As the ASA³P tarball contains all databases and 3rd party executables necessary, it is rather huge (23 Gb zipped, 29 Gb unzipped) and thus, download times may be quit long. To unzip the tarball a deflating tool supporting multithreadding might be beneficial, e.g. pigz on linux (sudo apt install pigz for Ubuntu).

Additional files:

Additional example and benchmark projects are hostet in a distinct repository at Zenodo: DOI

Docker

For small to medium projects (up to ~200 isolates) but also for the sake of simplicity, reproducibility and easy distribution, we offer ASA³P as a Docker image hosted at: Docker Hub (https://hub.docker.com/r/oschwengers/asap/). Please, follow the official instructions (https://docs.docker.com/install) to install Docker.

Setup:

$ sudo docker pull oschwengers/asap
$ wget https://zenodo.org/record/3780003/files/asap.tar.gz
$ tar -xzf asap.tar.gz
$ rm asap.tar.gz

Running an ASA³P Container using the asap-docker.sh shell wrapper script:

$ #<ASAP_DIR>/asap-docker.sh -p <PROJECT_DIR> [-s <SCRATCH_DIR>] [-a ASAP_DIR] [-z] [-c] [-d]
$ asap/asap-docker.sh -p example-lmonocytogenes -s /tmp

Parameters & Options:

Note

  1. This shell wrapper script should remain within the ASA³P directory in order to correctly extract related paths. In case the script was moved/copied somewhere else, you have to provide the path via -a <ASAP_DIR>.
  2. The script gathers user:group ids and passes these to the Docker container thus, files created by ASA³P automatically have the correct user ownerships instead of sudo ones.
  3. The script will ask for the sudo password as Docker containers can currently only be executed as sudo. This is pure technical necessity unrelated to ASA³P itself.

Complete example:

$ sudo docker pull oschwengers/asap
$ wget https://zenodo.org/record/3780003/files/asap.tar.gz
$ tar -xzf asap.tar.gz
$ rm asap.tar.gz
$ wget https://zenodo.org/record/3606761/files/example-lmonocytogenes-4.tar.gz?download=1
$ tar -xzf example-lmonocytogenes-4.tar.gz
$ rm example-lmonocytogenes-4.tar.gz
$ asap/asap-docker.sh -p example-lmonocytogenes-4/

For further information have a look at the Docker readme (docker/DOCKER.md ).

Cloud OpenStack

ASA³P's OpenStack based cloud version targets the analysis of hundreds to even thousands of bacterial isolates. Therefore, it features automatic creation, setup and orchestration of an SGE based compute cluster and its entire underlying infrastructure. Therefore, the OpenStack cloud version internally takes advantage of the BiBiGrid (https://github.com/BiBiServ/bibigrid) framework. Hence, analysis of thousands of genomes can be achieved in a highly parallel manner and adequate amount of time. ASA³P takes care of all setup and orchestration aspects and thus hides away as much technical complexity as possible. For further information please have a look at our user manual.

In order to trigger an OpenStack based cloud project, you need the following additional cloud related files:

Once ASA³P is properly setup you can start it by executing a single shell script:

$ ~/asap-cloud/asap-cloud.sh -i <INSTANCE_ID> -o <OPEN_STACK_RC_FILE> -p <PROJECT_DIR>

Parameters:

For a comprehensive and detailed description of how to setup an OpenStack project and ASA³P therein, please have a look at our manual stored here: DOI

Input/Output

Input

ASA³P is able to process raw sequencing reads from Illumina (SE/PE), PacBio (bax.h5/ubam) and ONT (basecalled as fastq) as well as assembled contigs (fasta) and annotated genomes (GenBank/EMBL/GFF).

ASA³P expects all input files and information regarding a single execution (i.e. a "project") within a dedicated directory. All necessary information (meta information, reference genomes, isolate/sample names and files) are provided via an Excel config file named config.xls. A corresponding template can be downloaded here. For further details on how to fill out a proper configuration file, please have a look at the manual and the exemplary projects listed above. All input files referenced in a configuration spreadsheet need to be placed in a subdirectory called data.

Example:

project-dir
├── config.xls
├── data
│   ├── reference-genome-1.gbk
│   ├── reference-genome-2.fasta
│   ├── isolate-1-1.fastq.gz
│   ├── isolate-1-2.fastq.gz
│   ├── isolate-2-1.fastq.gz
│   ├── isolate-2-2.fastq.gz
│   ├── isolate-3.1.bax.h5
│   ├── isolate-3.2.bax.h5
│   ├── isolate-3.3.bax.h5
│   ├── ...

Output

tl; dr Just open your browser and open the index.html file located at:

project-dir
├── reports   (HTML5 reports)
│   ├── index.html

In order to provide a first glimpse into the results of the pipeline, we configured a public login to a static web server for demonstration purposes only at: https://www.computational.bio.uni-giessen.de/asap/

$ login: asap-test
$ password: asap-test

ASA³P stores all output files within the specified project directory leaving input files untouched:

Furthermore, for each analysis ASA³P creates a corresponding subdirectory containing result files such as:

Where necessary ASA³P creates subdirectories for each isolate within an analysis directory.

Example:

project-dir
├── [state.running | state.finished | state.failed]
├── asap.log   (global logging file)
├── config.xls   (config spreadsheet)
├── config.json   (internal config)
├── reports   (HTML5 reports)
│   ├── index.html
│   ├── ...
├── reads_qc   (quality clipped read files)
│   ├── <sample-name>
│   ├── ├── [state.finished | state.failed]
│   ├── ├── isolate-1-1.fastq.gz
│   ├── ├── isolate-1-2.fastq.gz
│   ├── ├── info.json
│   ├── ...
├── assembly   (assemblies)
│   ├── <sample-name>
│   ├── ├── [state.finished | state.failed]
│   ├── ├── <sample-name>.fasta
│   ├── ├── <sample-name>-discarded.fasta
│   ├── ├── info.json
│   ├── ...
├── scaffolds   (scaffolded contigs)
│   ├── <sample-name>
│   ├── ├── [state.finished | state.failed]
│   ├── ├── <sample-name>.fasta   (scaffolds)
│   ├── ├── <sample-name>-pseudo.fasta   (pseudo genome)
│   ├── ├── info.json
│   ├── ...
├── annotations
│   ├── <sample-name>
│   ├── ├── [state.finished | state.failed]
│   ├── ├── <sample-name>.gbk   (Genbank)
│   ├── ├── <sample-name>.gff   (GFF3)
│   ├── ├── <sample-name>.ffn   (gene sequences)
│   ├── ├── <sample-name>.faa   (protein sequences)
│   ├── ├── info.json
│   ├── ...
├── taxonomy   (taxonomic classfication results)
│   ├── [<sample-name>.finished | <sample-name>.failed]
│   ├── <sample-name>.json
│   ├── ...
├── mlst   (multi-locus sequence typing results)
│   ├── [<sample-name>.finished | <sample-name>.failed]
│   ├── <sample-name>.json
│   ├── ...
├── abr   (antibiotic resistance genes detection)
│   ├── [<sample-name>.finished | <sample-name>.failed]
│   ├── <sample-name>.json
│   ├── ...
├── vf   (virulence factor detection results)
│   ├── [<sample-name>.finished | <sample-name>.failed]
│   ├── <sample-name>.json
│   ├── ...
├── mappings   (reference mappings)
│   ├── [<sample-name>.finished | <sample-name>.failed]
│   ├── <sample-name>.json
│   ├── <sample-name>.bam
│   ├── <sample-name>.bam.bai
│   ├── ...
├── snps   (called single nucleotide polymorphisms)
│   ├── [<sample-name>.finished | <sample-name>.failed]
│   ├── <sample-name>.json
│   ├── <sample-name>.consensus.fasta   (mpileup consensus file)
│   ├── <sample-name>.vcf.gz   (SNPs in variant calling format file)
│   ├── <sample-name>.vcf.gz.tbi
│   ├── <sample-name>.chk   (bcftools stats)
│   ├── <sample-name>.csv   (SNPeff per gene statisics)
│   ├── ...
├── corepan
│   ├── [state.finished | state.failed]
│   ├── info.json
│   ├── core.fasta   (core genome sequences)
│   ├── pan.fasta   (pan genome sequences)
│   ├── pan-matrix.tsv   (pan genome matrix)
│   ├── <sample-name>.json
│   ├── ...
├── phylogeny
│   ├── [state.finished | state.failed]
│   ├── info.json
│   ├── tree.nwk   (phylogenetic tree in newick file)
│   ├── consensus.fasta   (global consensus file)
├── data

Citation

Schwengers et al. (2020). ASA³P: An automatic and scalable pipeline for the assembly, annotation and higher level analysis of closely related bacterial isolates. PLOS Computational Biology 16(3): e1007134. https://doi.org/10.1371/journal.pcbi.1007134

License

ASA³P itself is published and distributed under GPL3 license. In contradiction, some of its dependencies bundled within the ASA³P tarball (asap.tar.gz file) are published under different licenses, e.g. GPL2, BSD, MIT, LGPL, etc. A file (README.md) within the ASA³P directory contains a list of all dependencies and related licenses.

NOTE Please, notice that some bundled dependencies are published under a free-for-academic or free-for-non-commercial usage license model. To our best knowledge this is true for at least the following databases:

FAQ