Closed jbrea closed 2 months ago
NearestNeighbors uses StaticArrays which is built on top of tuples which has compilation time issues when they get very big.
adapt the claim "perform high performance nearest neighbor searches in arbitrarily high dimensions."?
I removed this note in the README. We could add some blurb warning about this scenario but I am not sure it is worth doing something about.
I'm running into this issue too. I've got ~1600 dimensional features. There's not really a good reason to use StaticArrays past certain size IMO.
Using 1000 dimensions, compilation times aren't too bad. The cutover for me is somewhere between that and 1600. My guess would be it's around 1024, which for 32 bit floats means a SVector would be more than a 4k page, and that's where things get wonky. Honestly, there's probably no performance benefit to static arrays beyond like 64 dimensions.
I'm running into this issue too. I've got ~1600 dimensional features. There's not really a good reason to use StaticArrays past certain size IMO.
True but there is probably no reason to use this package for these type of dimensions either since it won't really accelerate your search.
Not that it would make a lot of sense to use KNN in high dimensions... but for demonstration purposes I was running KNN in high dimensions and ran into excessively long compilation times. To illustrate this I compared it to a very simple custom implementation of KNN.
simple_knn
)The performance of
simple_knn
becomes better already atnd = 100
and fornd > 1000
the time it takes to runknn
increases a lot.Would it make sense to
simpe_knn
for largend
?nd
?