Echtvar: Really, truly rapid variant annotation and filtering

Echtvar efficiently encodes variant allele frequency and other information from huge population datasets to enable rapid (1M variants/second) annotation of genetic variants. It chunks the genome into 1<<20 (~1 million) bases, encodes each variant into a 32 bit integer (with a supplemental table for those that can't fit due to large REF and/or ALT alleles). It uses the zip format, delta encoding and integer compression to create a compact and searchable format of any integer, float, or low-cardinality string columns selected from the population file.

Getting started.

Get a static binary and pre-encoded echtvar files for gnomad v3.1.2 (hg38) here: https://github.com/brentp/echtvar/releases/latest That page contains exact instructions to get started with the static binary.

:arrow_down:Download or Build instructions for linux

The linux binary is available via: ``` wget -O ~/bin/echtvar https://github.com/brentp/echtvar/releases/latest/download/echtvar \ && chmod +x ~/bin/echtvar \ && ~/bin/echtvar # show help ``` Users can make their own *echtvar* archives with `echtvar encode`, and pre-made archives for gnomAD version 3.1.2 are [here](https://github.com/brentp/echtvar/release) Rust users can build on linux with: ``` cargo build --release --target x86_64-unknown-linux-gnu ```

To run echtvar with an existing archive (we have several available in releases) is as simple as

echtvar anno -e gnomad.echtvar.zip -e other.echtvar.zip input.vcf output.annotated.bcf

an optional filter that utilizes fields available any of the zip files can be added like:

-i "gnomad_popmax_af < 0.01"

echtvar can also accept input from stdin using "-" or "/dev/stdin" for the input argument.

usage

encode

make (encode) a new echtvar file. This is usually done once (or download from those provided in the Release pages) and then the file can be re-used for the annotation (echtvar anno) step with each new query file. Note that input VCFs must be decomposed.

echtvar \
   encode \
   gnomad.v3.1.2.echtvar.zip \
   conf.json # this defines the columns to pull from $input_vcf, and how to
   $input_population_vcf[s] \ can be split by chromosome or all in a single file.
name and encode them

See below for a description of the json file that defines which columns are pulled from the population VCF.

annotate

Annotate a decomposed (and normalized) VCF with an echtvar file and only output variants where gnomad_af from the echtvar file is < 0.01. Note that multiple echtvar files can be specified and the -i expression is optional and can be elided to output all variants.

echtvar anno \
   -e gnomad.v3.1.2.echtvar.v2.zip \
   -e dbsnp.echtvar.zip \
   -i 'gnomad_popmax_af < 0.01' \
   $cohort.input.bcf \
   $cohort.echtvar-annotated.filtered.bcf

Configuration File for Encode

When running echtvar encode, a json5 (json with comments and other nice features) determines which columns are pulled from the input VCF and how they are stored.

A simple example is to pull a single integer field and give it a new name (alias):

[{"field": "AC", "alias": "gnomad_AC"}]

This will extract the "AC" field from the INFO and labeled as "gnomadAC" when later used to annotate a VCF. Note that it's important to give a description/unique prefix lke "`gnomad`" so as not to collide with fields already in the query VCF.

:arrow_down:Expand this section for detail on additional fields, including float and string types

``` [ {"field": "AC", "alias": "gnomad_AC"}, // this JSON file is json 5 and so can have comments // the missing value will default to -1, but the value: -2147483648 will // result in '.' as it is the missing value for VCF. {"field": "AN", "alias":, gnomad_AN", missing_value: -2147483648}, { field: "AF", alias: "gnomad_AF", missing_value: -1, // since all values (including floats) are stored as integers, echtvar internally converts // any float to an integer by multiplying by `multiplier`. // higher values give better precision and worse compression. // upon annotation, the score is divided by multiplier to give a number close to the original float. multiplier: 2000000, // set zigzag to true if your data has negative values zigzag: true, } // echtvar will save strings as integers along with a lookup. this can work for fields with a low cardinality. {"field": "string_field", "alias":, gnomad_string_field", missing_string: "UNKNOWN"}, // "FILTER" is a special case that indicates that echtvar should extract the FILTER column from the annotation vcf. {"field": "FILTER", "alias": "gnomad_filter"}, ] ``` The above file will extract 5 fields, but the user can chooose as many as they like when encoding. All fields in an `echtvar` file will be added (with the given alias) to any VCF it is used to annotate.

Other examples are available here

And full examples are in the wiki

Expressions

An optional expression will determine which variants are written. It can utilize any (and only) integer or float fields present in the echtvar file (not those present in the query VCF). An example could be:

-i 'gnomad_af < 0.01 && gnomad_nhomalts < 10'

The expressions are enabled by fasteval with supported syntax detailed here.

In brief, the normal operators: (&&, ||, +, -, *, /, <, <=, >, >= and groupings (, ), etc) are supported and can be used to craft an expression that returns true or false as above.

References and Acknowledgements

Without these (and other) critical libraries, echtvar would not exist.

htslib is used for reading and writing BCF and VCF via rust-htslib
stream-vbyte is used for integer compression via the excellent rust bindings
fasteval is used for the expressions. It is fast and simple and awesome.
bincode is used for rapid serialization of large variants.

echtvar is developed in the Jeroen De Ridder lab

brentp / echtvar

readme