0xTCG / geny

1 stars 2 forks source link

About T1K's evaluation results #1

Open mourisl opened 6 months ago

mourisl commented 6 months ago

Greetings. Thank you for developing Geny to expand the potential of the KIR genotyping with next-generation sequencing data!

In your manuscript, you also compared the accuracy with T1K, and I appreciate the sharing of the genotyping results of the HPRC samples on the github repository. As the developer, I can't help to check how T1K behaves in these experiments :). I noticed that many errors T1K made are the missing of alleles marked as "partial" or the alleles with no intron data in the IPD-KIR database. For example, for the KIR2DL3 in the HG00438 sample, T1K misses the 023 allele, which has no intronic sequence in the IPD-KIR v2.12.0. In T1K's older versions, these alleles are excluded when creating the reference sequence for WGS data. This constraint is addressed in T1K v1.0.4 and a relevant bug is fixed in v1.05 later. In our current test on the HPRC sample, I can get the KIR2DL3*023 in the outcome. I have uploaded the results for T1K v1.0.5 on the IPD-KIR v2.12.0 database at: https://github.com/mourisl/T1K_manuscript_evaluation/tree/master/T1K_1.0.5 for you convenience. By comparing the results with the excel file in your github repo, I found several missing/wrong predictions in T1K have been resolved. Though in the manuscript, you mentioned that the T1K was tested with v1.0.5, I guess the extracted reference sequence was still generated with an older version (t1k-build). Could you please check whether this is the case and maybe give T1K another round of test? Since the default output from T1K does not infer copy numbers, I expect Geny will still be more sensitive than T1K considering the copy number. But the new test may reflect the true precision of T1K v1.0.5.

Thank you for considering this.

inumanag commented 5 months ago

Hi @mourisl

Thanks for the comment!

IIRC we've run everything (including the database prep) w/ 1.0.5 or 1.0.4. I will check this again for sure---might reach out if we get stuck. The commands are available here: https://github.com/0xTCG/geny/blob/master/paper/experiments.ipynb

Cheers, Ibrahim