search

tvondra / tdigest

PostgreSQL extension for estimating percentiles using t-digest
PostgreSQL License
88 stars 13 forks source link

Invalid percentiles calculated #4

Closed sporty81 closed 4 years ago

sporty81 commented 4 years ago

For some reason with this tdigest, some percentiles are wrong.

flags 0 count 14 compression 100 centroids 14 (3.249696, 1) (8.566236, 1) (10.157393, 1) (12.757881, 1) (12.882931, 1) (38.585662, 1) (48.822837, 1) (55.881150, 1) (60.400374, 1) (69.476553, 1) (77.267742, 1) (83.499431, 1) (86.896817, 1) (91.439196, 1)

For some reason the 93-96 percentiles report higher than 97-99

92% 90.8941102381796 93% 91.5300433030352 94% 92.1659763678908 95% 92.8019094327465 96% 93.4378424976021 97% 91.4391957223415 98% 91.4391957223415 99% 91.4391957223415

tvondra commented 4 years ago

Thanks for the report. I can reproduce it and yeah - that seems like a bug.

tvondra commented 4 years ago

Should be fixed by 4eb5bff2222f39553e120bc82ebd32130721dee9. A stupid rounding issue with int64 and double arithmetics.

tvondra commented 4 years ago

Should be fixed by 4eb5bff2222f39553e120bc82ebd32130721dee9, and I've added a simple regression test in 0be29a29ea7dd0a561ff3fbf210323821669007b.