Missing data issue

[peterzt]

Greetings,

I was checking over resulted tsv file and found out that at least some variants included in clinvar VCF file ( ftp://ftp.ncbi.nlm.nih.gov/pub/clinvar/vcf_GRCh37/ ) are not being shown there.

Can you please explain why ? For example, the following variant appears in the ClinVar VCF file but doesn't appear in the TSV file included in the codebase for this repo AND also doesn't appear in regenerated TSV file using the latest ClinVar dataset:

7 92130876 rs61750420 C T

Thanks in advance, Peter

[kristjaneerik]

There are a few reasons why certain variants are skipped.

• The reference file has a mismatch with what's in the ClinVar data
• The reference file used has the reference sequence soft-masked in lower-case, causing a pseudo-mismatch. Because of this line https://github.com/ericminikel/minimal_representation/blob/master/normalize.py#L80 this case results in an "Incorrect REF value" warning being generated and the variant skipped. I have a fork of normalize.py which makes the reference bases uppercase: https://github.com/kristjaneerik/minimal_representation/blob/uppercase-ref/normalize.py#L80
• If the REF and ALT alleles are the same or an invalid nucleotide is seen
• There's no SequenceLocation information in the ClinVar XML, so we don't know where the variant is

However, I see this particular variant in clinvar_with_exac.tsv.gz (line 46777), clinvar.vcf.gz (line 46822) and clinvar.tsv.gz (line 46777) when zgrep-ing for 92130876 in these files that are currently in the repo.

[peterzt]

Hello kristjaneerik !

Thank you for your response, I was checking several variants and here is the one that I have not found there (I have searched on ready built TSV files):

7 99382096 rs2740574 C T

Another problem is that when I was trying to run the pipeline (inregarldess of using your normalize.py fork or original some) I have got the errors (listed below).

Can you please tell me what is wrong with my environment? Or probably FASTA selection (I've tried different reference fastas from UCSC and 1000g, but no success).

[Dec 8 19:23:53]: --> Exec 1.4: Rscript join_data.R variant_summary.txt.gz [Dec 8 19:23:53]: Output (last mod N/A): clinvar_combined.tsv [doesn't exist yet] [Dec 8 19:23:59]: [1] 184803 14 [Dec 8 19:24:12]: Warning message: [Dec 8 19:24:12]: In scan(file = file, what = what, sep = sep, quote = quote, dec = dec, : [Dec 8 19:24:12]: embedded nul(s) found in input [Dec 8 19:24:12]: [1] 354889 30 [Dec 8 19:24:12]: Error in [.data.frame(x, r, vars, drop = drop) : [Dec 8 19:24:12]: undefined columns selected [Dec 8 19:24:12]: Calls: subset -> subset.data.frame -> Dec 8 19:24:34: $subset(txt_download, assembly == "GRCh37", select = desired_columns) Dec 8 19:24:34: <environment: 0x222f190> Dec 8 19:24:34: Dec 8 19:24:34: $subset.data.frame(txt_download, assembly == "GRCh37", select = desired_colu Dec 8 19:24:34: <environment: 0x222de20> Dec 8 19:24:34: Dec 8 19:24:34: $x[r, vars, drop = drop] Dec 8 19:24:34: <environment: 0x2396f20> Dec 8 19:24:34: Dec 8 19:24:34: $`[.data.frame(x, r, vars, drop = drop) [Dec 8 19:24:34]: <environment: 0x2396f58> [Dec 8 19:24:34]: [Dec 8 19:24:34]: $stop("undefined columns selected")` Dec 8 19:24:34: <environment: 0x2352348> Dec 8 19:24:34:

Dec 8 19:24:34: Finished 1.4. Running time: 0:00:41.048258 sec. Dec 8 19:24:34: WARNING: unable to rename tmp.2016-12-08_19.23.51.clinvar_combined.tsv to clinvar_combined.tsv. tmp.2016-12-08_19.23.51.clinvar_combined.tsv is not readable Dec 8 19:24:34: --> Skipping 1.5: (cat clinvar_combined.tsv | head -1 > tmp.2016-12-08_19.23.51.clinvar_combined_sorted.tsv ) && (cat clinvar_combined.tsv | tail -n +2 | egrep -v "^[XYM]" | sort -k1,1n -k2,2n -k3,3 -k4,4 >> tmp.2016-12-08_19.23.51.clinvar_combined_sorted.tsv ) && (cat clinvar_combined.tsv | tail -n +2 | egrep "^[XYM]" | sort -k1,1 -k2,2n -k3,3 -k4,4 >> tmp.2016-12-08_19.23.51.clinvar_combined_sorted.tsv ) Dec 8 19:24:34: Input (last mod N/A): clinvar_combined.tsv Error - input file not found: Traceback (most recent call last): Dec 8 19:24:34: File "/root/.local/lib/python2.7/site-packages/pypez.py", line 769, in does_command_need_to_run Dec 8 19:24:34: try: raise Exception("File not found: " + str(input_filename)) Dec 8 19:24:34: Exception: File not found: clinvar_combined.tsv Dec 8 19:24:34:

Thanks in advance, Peter

[kristjaneerik]

With regards to 7 99382096 rs2740574 C T: yes, it is not in any of the posted files. When I fetch from the genome I used, that position shows up at c -- so this variant was (most likely) skipped due to softmasking.

The issue you are seeing (join_data.R failing, which triggers other failures downstream) is because ClinVar changed their data format and this codebase hasn't been updated to deal with it as the change is not a simple one. See #18 , #19. The solution for now, unfortunately, is to use older files as outlined in those issues. There's nothing wrong with your environment as such, you just need to specify an older XML and summary table as in #19 .

I don't know if/when the original authors will get to address this join_data.R issue; I have some thoughts on what could be done, but might not have the time to do it.

[bw2]

21 from @XiaoleiZ has fixed join_data.R so we'll be making up-to-date releases again.