search

samtools / hts-specs

Specifications of SAM/BAM and related high-throughput sequencing file formats
http://samtools.github.io/hts-specs/
647 stars 174 forks source link

Ambiguity in rounding specification #253

Closed eyherabh closed 6 years ago

eyherabh commented 7 years ago

Table 2 of VCFv4.3.pdf describes the field PL as "Phred-scaled genotype likelihoods rounded to the closest integer", without specifying the tie-breaking rule. Could you please clarify me whether ties (e.g. 3.5) are rounded away from zero, towards zero, or to the nearest even / odd value or something else? Thank you very much. Best regards.

cyenyxe commented 6 years ago

Nobody seems to have a strong preference, so I suggest we use existing tools as reference. @lbergelson @pd3 what is the GATK and bcftools behavior?

pd3 commented 6 years ago

BCFtools use integer truncation, realized by casting the floats to int.

However, I'd say this is not something to be enforced by the specification. The differences between different methods are bigger than rounding errors.

cyenyxe commented 6 years ago

Agreed offline by the maintainers that this must be implementation dependent, not defined by the specification.