Since Javascript uses 58-bit numbers (Number), using 32-bit floating point values will sometimes (often) result in rounding errors being introduced.
This has been happening to the numbers written to osm-p2p-db's kdb-tree-store. As a result, the value in the hyperlog doesn't actually necessarily match the value stored in the kdb index. This is usually fine: broad bounding box queries will return points within it even if there are small errors.
However, in the case of removing a point from the kdb index, the kdb tree relies on a certain amount of exactness: if the rounding error so happens to result in a value that is on the wrong side of a kdb tree split, the remove logic will recurse down the wrong branch and not delete the point. Such an error is idempotent: the rounding error will be the same each time and the point will essentially be undeletable.
These commits do two things:
Use 64-bit floats for the value of points in the kdb index. This is a more precise data type than Javascript's 58-bit Number type, so that error cannot occur.
Coerce all points from the hyperlog entering the kdb index into the Number type. Strange behaviour can occur if Strings are inserted.
I did some manual testing by looking over the Siekopai data with either index. Left is float32, right is float64. (Disregard the satellite tile that didn't load.)
Since Javascript uses 58-bit numbers (
Number), using 32-bit floating point values will sometimes (often) result in rounding errors being introduced.This has been happening to the numbers written to
osm-p2p-db'skdb-tree-store. As a result, the value in the hyperlog doesn't actually necessarily match the value stored in the kdb index. This is usually fine: broad bounding box queries will return points within it even if there are small errors.However, in the case of removing a point from the kdb index, the kdb tree relies on a certain amount of exactness: if the rounding error so happens to result in a value that is on the wrong side of a kdb tree split, the
removelogic will recurse down the wrong branch and not delete the point. Such an error is idempotent: the rounding error will be the same each time and the point will essentially be undeletable.These commits do two things:
Use 64-bit floats for the value of points in the kdb index. This is a more precise data type than Javascript's 58-bit
Numbertype, so that error cannot occur.Coerce all points from the hyperlog entering the kdb index into the
Numbertype. Strange behaviour can occur ifStrings are inserted.I did some manual testing by looking over the Siekopai data with either index. Left is float32, right is float64. (Disregard the satellite tile that didn't load.)