search

AmpliconSuite / AmpliconSuite-pipeline

A quickstart tool for AmpliconArchitect. Performs all preliminary steps (alignment, CNV calling, seed interval detection) required prior to running AmpliconArchitect. Previously called PrepareAA.
Other
48 stars 25 forks source link
cancer cancer-genomics ecdna

readme

AmpliconSuite-pipeline logo GitHub release (latest by date) Docker Image Version (latest by date) Docker pulls Singularity Conda

A multithread-enabled end-to-end wrapper for AmpliconArchitect and AmpliconClassifier to enable analysis of focal copy number amplifications such as ecDNA or BFBs from paired-end whole genome sequencing data.

AmpliconSuite-pipeline can be invoked to begin at any intermediate stage of the data preparation process and can itself invoke both AmpliconArchitect and the downstream tool AmpliconClassifier. AmpliconSuite-pipeline was formerly called "PrepareAA".

We recommend browsing our detailed guide to learn about best practices and to see some FAQs.

AmpliconSuite-pipeline supports hg19, GRCh37, GRCh38 (hg38), and mouse genome mm10 (GRCm38). It also supports analysis with a human-viral hybrid reference genome we provide, "GRCh38_viral", which can be used to detect oncoviral hybrid focal amplifications in oncoviral cancers.

Licenses

The modules wrapped in AmpliconSuite-pipeline use the following licenses. Please note that the AmpliconArchitect license specifies that AmpliconArchitect is for research use and does not give license for commerical for-profit use.

Other dependencies used by these modules (e.g. Mosek, samtools, etc.) have their own set of licensing requirements which users should make themselves aware of as needed. The Mosek license requires that users obtain a copy (which is free for academic use) from the Mosek website. More information is available in the installation section.

Installation

Option A: Installation-free platforms

The most convenient option, but not suitable for analysis of large collections of samples or protected health information (PHI), and may not support more advanced command-line options. An excellent option for most users with small numbers of non-PHI samples.

GenePattern:

AmpliconSuite-pipeline can be run using the web interface at GenePattern Web Interface. Search the module list for AmpliconSuite. Constructed in collaboration with members of the GenePattern team (Edwin Huang, Ted Liefeld, Michael Reich).

Nextflow:

AmpliconSuite-pipeline can also be run through Nextflow, using the nf-core/circdna pipeline constructed by Daniel Schreyer.

Option B: Conda or Mamba

conda create -n ampsuite && conda activate ampsuite
conda install -c bioconda -c conda-forge ampliconsuite 
conda install -c mosek mosek

# then run the installer script to finalize the locations of the data repo and mosek license 
wget https://raw.githubusercontent.com/AmpliconSuite/AmpliconSuite-pipeline/master/install.sh
source install.sh --finalize_only  # -h to see options

Then obtain the Mosek license (free for academic use) and place it in $HOME/mosek/. AA will not work without it.

Option C: Standalone installation using the installer script

Can be used on recent Unix systems (e.g. Ubuntu 18.04+, CentOS 7+, macOS). Requires python>=3.7.

  1. Pull source code and run install script (skip if installed via Conda):

    # first install some dependencies (BWA, R, samtools) if you don't already have them
# for ubuntu:
sudo apt install bwa r-base samtools
# or for macOS: brew install bwa r samtools  

git clone https://github.com/AmpliconSuite/AmpliconSuite-pipeline
cd AmpliconSuite-pipeline
# To see install  options, consider first doing 
# source ./install -h
# The install.sh script will install python dependencies using 'python3 -m pip install' 
source ./install.sh

  2. Obtain the Mosek license (free for academic use) and place it in $HOME/mosek/. AA will not work without it.

  3. (Optional) If you want the Arial font in your AA figures (helpful for publication-quality fonts), but do not have Arial on your Linux system, please see these instructions for making it available to Matplotlib.

Option D: Singularity & Docker images

Containerized versions of AmpliconSuite-pipeline are available for Singularity and Docker.

  1. Obtain the AmpliconSuite-pipeline image from the options below:

    • Singularity:

    • Docker:

      • Docker installation: https://docs.docker.com/install/

      • Pull the docker image: docker pull jluebeck/prepareaa

      • (Optional): Add user to the docker group: sudo usermod -a -G docker $USER (log out and back in after performing).

  2. Obtain the execution script and configure the data repo location

    git clone https://github.com/AmpliconSuite/AmpliconSuite-pipeline
cd AmpliconSuite-pipeline
# Can use ./install.sh -h to see help before installing
source ./install.sh --finalize_only

  3. License for Mosek dependency:

    • Obtain Mosek license file mosek.lic. The license is free for academic use.
    • Place the file in $HOME/mosek/ (the mosek/ folder that now exists in your home directory).
    • If you are not able to place the license in the default location, you can set a custom location by exporting the bash variable MOSEKLM_LICENSE_FILE=/custom/path/.

  4. (Recommended) Pre-download AA data repositories and set environment variable AA_DATA_REPO:

    • See the instructions in the section below on obtaining required reference annotations.
    • If you do not do this process, the container runscript will attempt to download the files into the container before running. This can add to your compute time, especially if you are running many samples.

Launching the execution script for the container:

These scripts use most of the same arguments are the main driver script AmpliconSuite-pipeline.py

An example command might look like:

AmpliconSuite-pipeline/singularity/run_paa_singularity.py --sif /path/to/ampliconsuite-pipeline.sif -o /path/to/output_dir -s name_of_run -t 8 --bam bamfile.bam --run_AA --run_AC

Option E: Installation without installer script

Try this if you are going to use python2. Please see the instructions here.

Downloading required reference annotations (AA data repo)

Before running AmpliconSuite-pipeline, populate the data repo with required annotations for the reference genomes of interest. This can be done using the example command below. Specifying [ref]_indexed will download a version that includes the BWA index, which is useful for alignment.

AmpliconSuite-pipeline.py --download_repo [GRCh38|hg19|mm10|... or GRCh38_indexed|hg19_indexed...]

Running AmpliconSuite-pipeline

The main driver script for the standalone pipeline is called AmpliconSuite-pipeline.py.

Example 1: Starting from .fastq files, using CNVkit for seed generation.

AmpliconSuite-pipeline.py -s sample_name -t number_of_threads --cnvkit_dir /path/to/cnvkit.py --fastqs sample_r1.fq.gz sample_r2.fq.gz --ref hg38 [--run_AA] [--run_AC]

Example 2: Starting from .bam, using CNVkit for seed generation

AmpliconSuite-pipeline.py -s sample_name -t n_threads --bam sample.bam [--run_AA] [--run_AC]

Example 3: Starting from .bam and your own whole-genome CNV calls, or an existing AA_CNV_SEEDS.bed

AmpliconSuite-pipeline.py -s sample_name -t number_of_threads --cnv_bed your_cnvs.bed --bam sample.bam [--run_AA] [--run_AC]

Where the CNV bed file reports the following four fields:

chr start end copy_number

Additional fields between end and copy_number may exist, but copy_number must always be the last column.

Example 4: Analyzing a collection of related samples (replicates or multi-region sampling)

If you have multiple samples from the same patient, cell line, etc., these should be run as a group to ensure that the same regions are studied across those samples. Please see the GroupedAnalysisAmpSuite.py example below for instructions.

Example 5: Analyzing an oncoviral sample for human-viral hybrid ecDNA detection

Note that users must start with fastq files and --ref GRCh38_viral or a bam file aligned to the AA_DATA_REPO/GRCh38_viral reference.

AmpliconSuite-pipeline.py -s sample_name -t n_threads --fastqs sample_r1.fq.gz sample_r2.fq.gz --ref GRCh38_viral --cnsize_min 10000 [--run_AA] [--run_AC]

Example 6: Starting from completed AA results

If the user has one or more AA results directories inside a directory, the user can use AmpliconSuite-pipeline to call AmpliconClassifier with default settings.

AmpliconSuite-pipeline.py -s project_name --completed_AA_runs /path/to/location_of_all_AA_results/ --completed_run_metadata run_metadata_file.json -t 1 --ref hg38

Note that when this mode is used all AA results must have been generated with respect to the same reference genome version.

Command line arguments to AmpliconSuite-pipeline

Downloading data repo files:

Otherwise, you will instead need these arguments below:

Required

Optional

Interpreting classification outputs

Testing your installation

To ensure your local installation of, we provide a small test dataset (~3Gb), which users can download from SRA. After obtaining either the BAM or FASTQ files for GBM39_FF-8 provided on SRA, users can run the method and compare their results against the files in the test_outputs/ directory.

FAQ

Please check out our guide document.

Packaging outputs for AmpliconRepository

We will soon release an online platform for storing and sharing your AmpliconSuite-pipeline outputs.

To package a collection of AA outputs for AmpliconRepository, you will need to do the following steps.

  1. (Recommended) Before running AmpliconSuite-pipeline, using the file sample_metadata_skeleton.json as a template, please create a copy of the file and fill out the JSON file for each sample. Provide this to AmpliconSuite-pipeline.py using --sample_metadata {sample_metadata.json}

  2. Create a tar.gz file from your AA outputs tar -czf my_collection.tar.gz /path/to/AA_outputs/ (creating a .zip also works)

    • If you used the docker or singularity option, step 2 is already done.
    • Be sure to exclude bam & fastqs when you zip up your files (e.g. tar -czf --exclude='*.bam')

  3. If you have not already, create an account at GenePattern.

  4. Upload your compressed collection of AA output files (one or more samples) to the AmpliconSuiteAggregator GenePattern module.

  5. Run the aggregator and download the aggregated .tar.gz result file.

  6. Sign up on AmpliconRepository.org and upload your aggregated file.

Citing

If using AmpliconSuite-pipeline in your publication, please cite all the relevant modules used in the analysis, which are summarized in CITATIONS.md.

Additional analysis tools and scripts

- Grouped analysis of related samples GroupedAnalysisAmpSuite.py

For samples derived from a common origin (longitudinal, multiregional sampling from the same source material), it is advised that the seed intervals be unified before running AA in order to provide the best comparability between runs. We provide a script GroupedAnalysisAmpSuite.py which automates this analysis. GroupedAnalysisAmpSuite.py takes almost all the same arguments as PrepareAA.py, however it requires an additional input file, listing the inputs. This file is to be formatted as follows

sample_name bamfile "tumor"/"normal" [CNV_calls] [sample_metadata_json]

Where CNV_calls and sample_metadata_json are optional. All samples listed in each file should be uniquely named and from the same group of related samples. Do not include different collections of related samples in the same table - make different tables. However, they are positional, so if CNV_calls is skipped, it should be set as either NA or None.

AA and AC will be run by default, but can be disabled with --no_AA.

Example command:

GroupedAnalysisAmpSuite.py -i {inputs.txt} -o {output_dir} -t {num_threads}

- Candidate AMplicon Path EnumeratoR CAMPER.py

Exahustively search an AA graph file for longest paths (cyclic and non-cyclic). A median amplicon copy number must be specified, or the script will attempt to estimate on its own. CAMPER.py rescales the copy numbers by the median to estimate the multiplicity of each segment within the amplicon, and then searches for plausible longest paths explaining the copy number multiplicities. This is useful for identifiying some candidate ecDNA structures. The output will be an AA-formatted cycles file with additional annotations for length and quality control filter status. The quality filters take into account root-mean-square residual of copy numbers ("RMSR", lower score is better), as well as "DBI" representing the Davies-Bouldin index of copy-number to multiplicity clustering. More information on the method can be found in the methods section of this publication. The first entry (Cycle1) will be a cyclic path, while the second entry (Cycle2) will be a non-cyclic path. A full explanation of arguments is available with -h. Note that this should only be applied to AA amplicons with at most 1 ecDNA present in the AA amplicon (multiple-species reconstruction not supported).

AmpliconSuite-pipeline/scripts/plausible_paths.py -g sample_amplicon1_graph.txt [--scaling_factor (CN estimate value)] [--remove_short_jumps] [--keep_all_LC] [--max_length (value in kbp)]

- breakpoints_to_bed.py

Requires intervaltree python package. Write discordant edges (breakpoint junctions) from an AA graph into a pseudo-bed file. The .input file is automatically produced by AC, but is formatted like so

samplename /path/to/sample_amplicon1_cycles.txt /path/to/sample_amplicon1_graph.txt

Usage:

scripts/breakpoints_to_bed.py -i (AC.input) [--regions chrA:start-stop chrB:start-stop ...]

- convert_cns_to_bed.py

Many users will choose to run CNVkit outside AmpliconSuite-pipeline and then want to use the CNVkit calls in AA. We recommend using the .cns file as a source for the seeds. Note the .call.cns file is different and contains more aggressively merged CNV calls, which we do not recommend as a source of seeds. As the .cns file specifies a log2 ratio, we provide the following script to reformat the .cns file from CNVkit into a .bed file.

This script should not be needed by most users, as the *_cnvkit_output/ directory will already contain a .bed of genome-wide CNV calls produced by CNVkit.

Usage:

scripts/convert_cns_to_bed.py sample.cns

- cycles_to_bed.py

Requires intervaltree python package. Write an AA cycles file as a series of bed files, one for each decomposition. Writes two types of bed files, an unordered_cycle file, where segments are merged and sorted, and order and orientation of segments is lost, and also writes an ordered file where the order and orientation of the genome segments comprising the cycle is maintained.

Usage:

scripts/cycles_to_bed.py -c sample_amplicon1_cycles.txt

- graph_cleaner.py

Requires intervaltree python package. Poorly controlled insert size can lead to numerous spurious short breakpoint edges. This script attempts to remove SV edges which conform to small artificial indels. Namely, very short everted (inside-out read pair) orientation edges. These will appear as numerous short brown 'spikes' in the AA amplicon image. This script removes them from the graph file. The filtering of these artifacts is done automatically by AA in modern releases, so this script is for legacy purposes.

Usage:

scripts/graph_cleaner.py -g /path/to/sample_ampliconx_graph.txt [--max_hop_size 4000]

or

scripts/graph_cleaner.py --graph_list /path/to/list_of_graphfiles.txt [--max_hop_size 4000]

This will output an AA graph file(s) /path/to/my_sample_ampliconX_cleaned_graph.txt.

- graph_to_bed.py

Requires intervaltree python package. Create a bed file of the graph segments and a bedpe file of the disordant graph edges. Can also filter to only get segments with CN above --min_cn. Setting --unmerged will not merge adjacent graph segments and will print the graph segment CN in the last column.

Usage:

scripts/graph_to_bed.py -g sample_amplicon_graph.txt [--unmerged] [--min_cn 0] [--add_chr_tag]

//: # ()

//: # ()

//: # ()

//: # ()

//: # ()