HOWARD

Highly Open Workflow for Annotation & Ranking toward genomic variant Discovery

HOWARD annotates and prioritizes genetic variations, calculates and normalizes annotations, translates files in multiple formats (e.g. vcf, tsv, parquet) and generates variants statistics.

HOWARD annotation is mainly based on a build-in Parquet annotation method, and external tools such as BCFTOOLS, ANNOVAR, snpEff, Exomiser and Splice (see docs, automatically downloaded if needed). Parquet annotation uses annotation database in VCF or BED format, in mutliple file format: Parquet/duckdb, VCF, BED, TSV, CSV, TBL, JSON.

HOWARD calculation processes variants information to calculate new information, such as: harmonizes allele frequency (VAF), extracts Nomen (transcript, cNomen, pNomen...) from HGVS fields with an optional list of personalized transcripts, generates VaRank format barcode.

HOWARD prioritization algorithm uses profiles to flag variants (as passed or filtered), calculate a prioritization score, and automatically generate a comment for each variants (example: 'polymorphism identified in dbSNP. associated to Lung Cancer. Found in ClinVar database').Prioritization profiles are defined in a configuration file. A profile is defined as a list of annotation/value, using wildcards and comparison options (contains, lower than, greater than, equal...). Annotations fields may be quality values (usually from callers, such as 'GQ', 'DP') or other annotations fields provided by annotations tools, such as HOWARD itself (example: COSMIC, Clinvar, 1000genomes, PolyPhen, SIFT). Multiple profiles can be used simultaneously, which is useful to define multiple validation/prioritization levels (example: 'standard', 'stringent', 'rare variants', 'low allele frequency').

HOWARD translates VCF format into multiple formats (e.g. VCF, TSV, Parquet), by sorting variants using specific fields (example : 'prioritization score', 'allele frequency', 'gene symbol'), including/excluding annotations/fields, including/excluding variants, adding fixed columns.

HOWARD generates statistics files with a specific algorithm, snpEff and BCFTOOLS.

HOWARD is multithreaded through the number of variants and by database (data-scaling).

HOWARD is able to add plugins for further analyses.

Table of contents

• Installation
  • Python
  • Docker
  • Databases
  • Configuration
• Tools
  • Parameters
  • Stats
  • Convert
  • Query
  • Annotation
  • Calculation
  • Prioritization
  • HGVS annotation
  • Process
• Documentation
• Contact

Installation

HOWARD can be installed using Python, and a Docker installation provides a CLI (Command Line Interface) with all external tools and useful databases. Databases can be automatically downloaded, or home-made generated (created or downloaded).

Python

Install HOWARD using Python Pip tool, and run HOWARD for help options:

conda create --name=howard python=3.10
conda activate howard
python -m pip install -e .
howard --help

usage: howard [-h] {query,stats,convert,hgvs,annotation,calculation,prioritization,process,databases,gui,help,genebe,minimalize} ...

HOWARD:0.11.0
Highly Open Workflow for Annotation & Ranking toward genomic variant Discovery
HOWARD annotates and prioritizes genetic variations, calculates and normalizes annotations, convert on multiple formats, query variations and generates statistics

Shared arguments:
  -h, --help            show this help message and exit

Tools:
  {query,stats,convert,hgvs,annotation,calculation,prioritization,process,databases,gui,help,genebe,minimalize}
    query               Query genetic variations file in SQL format.
    stats               Statistics on genetic variations file.
    convert             Convert genetic variations file to another format.
    hgvs                HGVS annotation (HUGO internation nomenclature) using refGene, genome and transcripts list.
    annotation          Annotation of genetic variations file using databases/files and tools.
    calculation         Calculation operations on genetic variations file and genotype information.
    prioritization      Prioritization of genetic variations based on annotations criteria (profiles).
    process             Full genetic variations process: annotation, calculation, prioritization, format, query, filter...
    databases           Download databases and needed files for howard and associated tools
    gui                 Graphical User Interface tools
    help                Help tools
    genebe              GeneBe annotation using REST API
    minimalize          Minimalize a VCF file, such as removing INFO/Tags or samples

Usage examples:
   howard process --input=tests/data/example.vcf.gz --output=/tmp/example.annotated.vcf.gz --param=config/param.json 
   howard annotation --input=tests/data/example.vcf.gz --output=/tmp/example.howard.vcf.gz --annotations='tests/databases/annotations/current/hg19/dbnsfp42a.parquet,tests/databases/annotations/current/hg19/gnomad211_genome.parquet' 
   howard calculation --input=tests/data/example.full.vcf --output=/tmp/example.calculation.tsv --calculations='vartype' 
   howard prioritization --input=tests/data/example.vcf.gz --output=/tmp/example.prioritized.vcf.gz --prioritization_config=config/prioritization_profiles.json --prioritizations='default,GERMLINE' 
   howard query --input=tests/data/example.vcf.gz --explode_infos --query='SELECT "#CHROM", POS, REF, ALT, "DP", "CLNSIG", sample2, sample3 FROM variants WHERE "DP" >= 50 OR "CLNSIG" NOT NULL ORDER BY "CLNSIG" DESC, "DP" DESC' 
   howard stats --input=tests/data/example.vcf.gz 
   howard convert --input=tests/data/example.vcf.gz --output=/tmp/example.tsv --explode_infos && cat /tmp/example.tsv 

Install HOWARD Graphical User Interface using Python Pip tool with supplementary packages, and run as a tool:

python -m pip install -r requirements-gui.txt
howard gui

Docker

In order to build, setup and create a persitent CLI (running container with all useful external tools such as BCFTools, snpEff, Annovar, Exomiser), docker-compose command build images and launch services as containers.

docker-compose up -d

A setup container (HOWARD-setup) will download useful databases (take a while). To avoid databases download (see Databases section to download manually), just start:

docker-compose up -d HOWARD-CLI

A Command Line Interface container (HOWARD-CLI) is started with host data and databases folders mounted (by default in ~/howard folder, i.e. ~/howard/data:/data and ~/howard/databases:/databases). Let's play within Docker HOWARD-CLI service!

docker exec -ti HOWARD-CLI bash
howard --help

Databases

Multiple databases can be automatically downloaded with databases tool, such as:

Configuration

HOWARD Configuration JSON file defined default configuration regarding resources (e.g. threads, memory), settings (e.g. verbosity, temporary files), default folders (e.g. for databases) and paths to external tools.

See HOWARD Configuration JSON for more information.

Tools

HOWARD annotates and prioritizes genetic variations, calculates and normalizes annotations, convert on multiple formats, query variations and generates statistics. These tools require options or a Parameters JSON file.

Parameters

HOWARD Parameters JSON file defined parameters to process annotations, prioritization, calculations, convertions and queries. Use this parameters file to configure tools, instead of options or as a main configuration (options will replace parameters in JSON file).

See HOWARD Parameters JSON for more information.

Stats

Statistics on genetic variations, such as: number of variants, number of samples, statistics by chromosome, genotypes by samples, annotations. Theses statsitics can be applied to VCF files and all database annotation files.

Convert

Convert genetic variations file to another format. Multiple format are available, such as usual and official VCF format, but also other formats such as TSV, CSV, TBL, JSON and Parquet/duckDB. These formats need a header '.hdr' file to take advantage of the power of howard (especially through INFO/tag definition), and using howard convert tool automatically generate header file fo futher use (otherwise, an default '.hdr' file is generated).

Query

Query genetic variations in SQL format. Data can be loaded into 'variants' table from various formats (e.g. VCF, TSV, Parquet...). Using 'explode' option allows querying on INFO/tag annotations. SQL query can also use external data within the request, such as a Parquet file(s).

Annotation

Annotation is mainly based on a build-in Parquet annotation method, using database format such as Parquet, duckdb, VCF, BED, TSV, JSON. External annotation tools are also available, such as BCFTOOLS, Annovar, snpEff, Exomiser and Splice. It uses available databases and homemade databases. Annovar and snpEff databases are automatically downloaded (see HOWARD Help Databases tool). All annotation parameters are defined in HOWARD Parameters JSON file.

Quick annotation allows to annotates by simply listing annotation databases, or defining external tools keywords. These annotations can be combined.

Calculation

Calculation processes variants information to generate new information, such as: identify variation type (VarType), harmonizes allele frequency (VAF) and calculate sttistics (VAF_stats), extracts Nomen (transcript, cNomen, pNomen...) from an HGVS field (e.g. snpEff, Annovar) with an optional list of personalized transcripts, generates VaRank format barcode, identify trio inheritance.

Prioritization

Prioritization algorithm uses profiles to flag variants (as passed or filtered), calculate a prioritization score, and automatically generate a comment for each variants (example: 'polymorphism identified in dbSNP. associated to Lung Cancer. Found in ClinVar database'). Prioritization profiles are defined in a configuration file in JSON format. A profile is defined as a list of annotation/value, using wildcards and comparison options (contains, lower than, greater than, equal...). Annotations fields may be quality values (usually from callers, such as 'DP') or other annotations fields provided by annotations tools, such as HOWARD itself (example: COSMIC, Clinvar, 1000genomes, PolyPhen, SIFT).

Multiple profiles can be used simultaneously, which is useful to define multiple validation/prioritization levels (e.g. 'standard', 'stringent', 'rare variants'). Prioritization score can be calculated following multiple mode, either 'HOWARD' (incremental) or 'VaRank' (maximum). Prioritization fields can be selected (PZScore, PZFlag, PZComment, PZTags, PZInfos).

HGVS Annotation

HOWARD annotates variants with HGVS annotation using HUGO HGVS internation Sequence Variant Nomenclature (http://varnomen.hgvs.org/). Annotation refere to refGene and genome to generate HGVS nomenclature for all available transcripts. This annotation add 'hgvs' field into VCF INFO column of a VCF file. Several options are available, to add gene, exon and protein information, to generate a "full format" detailed annotation, to choose codon format.

See HOWARD Help HGVS tool for more options.

Process

HOWARD process tool manage genetic variations to:

• annotates genetic variants with multiple annotation databases/files and tools
• calculates and normalizes annotations
• prioritizes variants with profiles (list of citeria) to calculate scores and flags
• annotates genetic variants with HGVS nomenclature
• translates into various formats
• query genetic variants and annotations
• generates variants statistics

This process tool combines all other tools to pipe them in a uniq command, through available options or a parameters file in JSON format (see HOWARD Parameters JSON file).

See HOWARD Help Process tool tool for more information.

Documentation

HOWARD User Guide is available to assist users for particular commands, such as software installation, databases download, annotation command, and so on.

HOWARD Tips proposes some additional advices to handle HOWARD for particular use cases.

HOWARD Help describes options of all HOWARD tools. All information are also available for each tool using --help option.

HOWARD Configuration JSON describes configuration JSON file structure and options.

HOWARD Parameters JSON describes parameters JSON file structure and options.

HOWARD Parameters Databases JSON describes configuration JSON file for databases download and convert.

HOWARD Plugins describes how to create HOWARD plugins.

HOWARD Package describes HOWARD Package, Classes and Functions.

Contact

Medical Bioinformatics applied to Diagnosis Lab @ Strasbourg Univerty Hospital

bioinfo@chru-strasbourg.fr

GitHub