FeatureBaseDB / featurebase

A crazy fast analytical database, built on bitmaps. Perfect for ML applications. Learn more at: http://docs.featurebase.com/. Start a Docker instance: https://hub.docker.com/r/featurebasedb/featurebase
https://www.featurebase.com
Apache License 2.0
2.53k stars 230 forks source link

make percentile work with where clauses and on decimal fields #2362

Closed seebs closed 1 year ago

seebs commented 1 year ago

no worries this one should be a simple one-liner, right?

actually it's four distinct bugs.

fbsql=# select percentile(x, 50) from i where x < 13;
------
 5.00

fbsql=# select percentile(x, 50) from i where x < 13.0;
-------
 10.00

In this, we see the following bugs:

  1. With where x < 13, we actually use an SQL3-internal implementation of Percentile, rather than the PQL implementation, because 13 is an int64 and not a float64, and we don't allow int64 in Decimal conditions.
  2. With where x < 13.0, the WHERE clause is not actually exported to the PQL Percentile call as a filter.
  3. With or without the WHERE clause, PQL Percentile produces nonsense results (always the minimum, basically) for decimal fields.
  4. The SQL3-internal implementation of Percentile is actually just a copy of Count, rather than actually trying to implement percentile.

This took some sorting.

seebs commented 1 year ago

Ugh. Those failing tests are actually real errors -- in that the tests are wrong, which means I need to revert the fourth bug fix, fix the answers in the tests, and then debug why they were passing, because they shouldn't have been.

seebs commented 1 year ago

Talked to Pat, got more feedback, abandoned my plan to complain about every failure to convert to PQL, because the actual reason the filters were getting lost was coincidental to that -- or rather, required both that and that the filter in question was using an aliased name. Anyway, this should actually be done now.

sonarcloud[bot] commented 1 year ago

Kudos, SonarCloud Quality Gate passed!    Quality Gate passed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 2 Code Smells

No Coverage information No Coverage information
48.5% 48.5% Duplication