astrolabsoftware / spark3D

Spark extension for processing large-scale 3D data sets: Astrophysics, High Energy Physics, Meteorology, …
https://astrolabsoftware.github.io/spark3D/
Apache License 2.0
30 stars 16 forks source link

Fix wrong ordering when using unique in KNN. #111

Closed JulienPeloton closed 5 years ago

JulienPeloton commented 5 years ago

We are using the wrong convention for scala.Ordering.get. From the doc we have

def gt(x: T, y: T): Boolean
  // Return true if x > y in the ordering.

while in the code we had:

// in BoundedUniquePriorityQueue.scala
if (head != null && ord.gt(a, head)) {
  // replace head by a
}

leading to wrong result!

without the fix:

knn_with_octree_repartitioning

With the fix:

knn_with_octree_repartitioning

JulienPeloton commented 5 years ago

Note that:

JulienPeloton commented 5 years ago

Note also that it was not seen without data repartitioning.

codecov-io commented 5 years ago

Codecov Report

:exclamation: No coverage uploaded for pull request base (queries@ad37750). Click here to learn what that means. The diff coverage is 90%.

Impacted file tree graph

@@            Coverage Diff             @@
##             queries     #111   +/-   ##
==========================================
  Coverage           ?   89.64%           
==========================================
  Files              ?       32           
  Lines              ?     1178           
  Branches           ?      194           
==========================================
  Hits               ?     1056           
  Misses             ?      122           
  Partials           ?        0
Flag Coverage Δ
#python 91.24% <ø> (?)
#scala 88.95% <90%> (?)
Impacted Files Coverage Δ
...com/spark3d/utils/BoundedUniquePriorityQueue.scala 90.47% <100%> (ø)
src/main/scala/com/spark3d/utils/Utils.scala 94.11% <88.88%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update ad37750...82bc7ae. Read the comment docs.