#227: Utils: Integrate RAPIDS library to speed up dimensionality reduction.

[nikolay-alemasov]

• [ ] Enable selection of a machine where a Python-script is executed.
• [ ] Validate the performance gain from the RAPIDS use compared to the native dimensionality reduction implementation.
• [ ] Add this new feature into standard dimensionality reduction pipeline.

[nikolay-alemasov]

@skalkin , could you, please, correct the issue if needed?

[skalkin]

1. Get an EC2 machine with the GPU and test the UMAP performance against the 40k dataset
2. If we like what we see, integrate it as a package (we will discuss our options with Alexander and Sofia)

[dnillovna]

This issue has been mirrored in Jira: https://reddata.atlassian.net/browse/GROK-14924

[StLeonidas]

Consider to close

[dnillovna]

This issue has been mirrored in Jira: https://reddata.atlassian.net/browse/GROK-16034

[drizhina]

Update: We have the working code for dimensionality reduction using the RAPIDS library. the constraint is that given library only supports primitive distance functions, such as Euclidean, Minkowsky, Hamming etc. and therefore, is much less useful for diverse data.

The library also supports using precalculated KNN for dimensionality reduction, but as this pre-calculating KNN is very heavy opperation, it can become a serious bottleneck.

We have already sped up the dimensionality reduction using webGPU, which enables all the aforementioned calculations (KNN generation, UMAP) on your gpu. even on integrated graphics card it can be tens of times faster. you can check out the post about it here