Motivated by trying to fit a DBSCAN model to a large dataset. Problems were:
indistinct clustering criterion; this was very strict (separation between within and between strain cluster on both axes required), rejecting some sensible fits; now relaxed (separation only required on one axis) - let me know if you think the stricter option should still be available though a flag, or if you're happy with change across the board
slow fitting to large datasets; implemented GPU version of DBSCAN, which is fast; the problem is then assigning all distances, which is slow, because the model takes up a lot of GPU memory, and copying over batches of distances into the variable amount of remaining GPU memory (customisable with the new --assign-subsample option) negates the speed up of the initial fit
slow assignment of distances to model fit; this is inefficient, as we typically don't use the assignments of points to the initial model fit, and it takes ages on a large dataset. Instead I have added a --no-assign flag, which skips the assignment, labels the model appropriately, and allows a refined model fit that then assigns all points
If you approve these changes conceptually, then I'll add tests and docs. At the moment, local tests fail on the mandrake clustering step - I don't know if these are related to the failing tests for mandrake, or a local installation problem - will see what the CI outcomes are. Hence the slightly early-stage PR, sorry!
Motivated by trying to fit a DBSCAN model to a large dataset. Problems were:
--assign-subsampleoption) negates the speed up of the initial fit--no-assignflag, which skips the assignment, labels the model appropriately, and allows a refined model fit that then assigns all pointsIf you approve these changes conceptually, then I'll add tests and docs. At the moment, local tests fail on the mandrake clustering step - I don't know if these are related to the failing tests for mandrake, or a local installation problem - will see what the CI outcomes are. Hence the slightly early-stage PR, sorry!