Freeze 6

[ch-kr]

PR with code to re-export 300K VCFs and prepare them for return to the UKBB.

New scripts:

• Script to reformat VCF headers (from UKBB): ukbb_header_reformat.sh
• Script to get VCF shard start/end positions: get_shard_positions.py

VCF script updates:

• Added code block to VCF HT (to save some time when preparing VCF annotations)
• Added code to drop frequencies stratified by subpop (using gnomad pop as pop label and hybrid pop as subpop)
• Added code to flag segdup regoins in region_flag annotation
• Added code to re-add rsid annotation as set, not string (to allow for multiple rsids per site)
• Added code to remove duplicate sample IDs present in 300K raw MT
• Added code to remove samples that withdrew consent after 300K QC completion
• Added code to write VCF MT to the same number of partitions as number of desired VCF shards
• Added tabix index and append to header file
• Removed code used to export subpop information to VCF and VCF header

[ch-kr]

Desired VCF schema:

FORMAT fields:

GT, GQ, DP, AD, MIN_DP, PGT, PID, PL, SB

FILTER fields:

AC0, InbreedingCoeff, MonoAllelic, PASS, RF

INFO fields:

Frequency fields:

AC, AN, AF, nhomalt, popmax, faf95, faf99

• adj, raw
• Stratified by population, sex, and pop + sex

gnomAD frequency fields:

AC, AN, AF, nhomalt, popmax, faf95, faf99

• adj, raw
• Stratified by population, sex, and pop + sex
• Stratified by subpop (gnomAD exomes only)

RF fields:

rf_tp_probability, rf_positive_label, rf_negative_label, rf_label, rf_train

Region type fields:

lcr, segdup, nonpar, fail_interval_qc, in_capture_region

VEP

Allele info fields

allele_type, has_star, n_alt_alleles, original_alleles, variant_type, was_mixed

VQSR and info HT fields

• AS_VQSLOD, AS_culprit, NEGATIVE_TRAIN_SITE, POSITIVE_TRAIN_SITE
• FS, MQ, MQRankSum, QD, ReadPosRankSum, SOR, VarDP, InbreedingCoeff, sibling_singleton
• AS_FS, AS_MQ, AS_MQRankSum, AS_pab_max, AS_QD, AS_ReadPosRankSum, AS_SOR, AS_VarDP

Hist fields

• gq_hist_alt_bin_freq, gq_hist_alt_raw_bin_freq, gq_hist_all_bin_freq, gq_hist_all_raw_bin_freq
• dp_hist_alt_bin_freq, dp_hist_alt_n_larger, dp_hist_alt_raw_bin_freq, dp_hist_alt_raw_n_larger, dp_hist_all_bin_freq, dp_hist_all_n_larger, dp_hist_all_raw_bin_freq, dp_hist_all_raw_n_larger
• ab_hist_alt_bin_freq, ab_hist_alt_raw_bin_freq
• age_hist_het_bin_freq, age_hist_het_n_smaller, age_hist_het_n_larger, age_hist_hom_bin_freq, age_hist_hom_n_smaller, age_hist_hom_n_larger

[ch-kr]

I just added a batch script to repackage the VCF shard headers (required for ROR). note that the script isn't fully tested -- I tried testing it using one of the old VCF shards but forgot that this wouldn't work (bcftools complains about the duplicated sample ID)