Migrate to use cuVS for vector search

[benfred]

This PR updates to use cuVS instead of RAFT for vector search, pairwise distances and clustering. This is required for us to deprecate the vector search functionality in RAFT, in favour of the code in cuVS.

Because some code hasn't been migrated over to cuvs yet, we will continue to use the version in RAFT - but with RAFT in header only mode. In particular this functionality will be used in RAFT header only mode:

• Random Ball Cover (see https://github.com/rapidsai/cuvs/pull/218)
• Sparse KNN
• nn-descent https://github.com/rapidsai/cuvs/issues/364
• MetricProcessor
• knn_merge_parts
• build_dendrogram_host
• build_sorted_mst
• raft DistanceType

Because sparse KNN in RAFT uses the DistanceType in RAFT, we can't fully move over to use the DistanceType code in cuVS with this PR. (Also the DistanceType code in RAFT has a Precomputed option that isn't available in cuvs - but is needed by cuml for dbscan.) This means that we have both the raft and cuvs DistanceType enum's in use with this change, with conversions between them.

[benfred]

@bdice I'm seeing an error during the auditwheel step in the wheel builds saying:

ValueError: Cannot repair wheel, because required library "libcuvs.so" could not be located

Do you have any ideas what I'm doing wrong here =) ?

[bdice]

@benfred I pushed a commit with a fix -- I'm pretty sure we need to list libcuvs.so in the exclusion list for cuml, because we expect that library to be provided by another wheel that cuml depends on. This is how we handle cuspatial's dependency on cudf, among other examples.

[dantegd]

It sems like the wheel jobs are still having trouble finding libcuvs.so:

ImportError while loading conftest '/__w/cuml/cuml/python/cuml/cuml/tests/conftest.py'.
conftest.py:17: in <module>
    from cuml.testing.utils import create_synthetic_dataset
/pyenv/versions/3.10.15/lib/python3.10/site-packages/cuml/__init__.py:17: in <module>
    from cuml.internals.base import Base, UniversalBase
/pyenv/versions/3.10.15/lib/python3.10/site-packages/cuml/internals/__init__.py:17: in <module>
    from cuml.internals.available_devices import is_cuda_available
/pyenv/versions/3.10.15/lib/python3.10/site-packages/cuml/internals/available_devices.py:16: in <module>
    from cuml.internals.device_support import GPU_ENABLED
E   ImportError: libcuvs.so: cannot open shared object file: No such file or directory

[benfred]

It sems like the wheel jobs are still having trouble finding libcuvs.so:

ImportError while loading conftest '/__w/cuml/cuml/python/cuml/cuml/tests/conftest.py'.
conftest.py:17: in <module>
    from cuml.testing.utils import create_synthetic_dataset
/pyenv/versions/3.10.15/lib/python3.10/site-packages/cuml/__init__.py:17: in <module>
    from cuml.internals.base import Base, UniversalBase
/pyenv/versions/3.10.15/lib/python3.10/site-packages/cuml/internals/__init__.py:17: in <module>
    from cuml.internals.available_devices import is_cuda_available
/pyenv/versions/3.10.15/lib/python3.10/site-packages/cuml/internals/available_devices.py:16: in <module>
    from cuml.internals.device_support import GPU_ENABLED
E   ImportError: libcuvs.so: cannot open shared object file: No such file or directory

Yeah - I tried fixing this by adding a python dependency to cuvs here https://github.com/rapidsai/cuml/pull/6085/commits/257898e94355a816623fb936264a56aacc6246b5 , but it seemed like this didn't work =(

[dantegd]

It is installing cuvs in the environment, but the cuml wheel is not finding the libcuvs.so even then :/

[bdice]

@KyleFromNVIDIA @jameslamb may be able to help here, I suspect we need to do some work to set RPATHs on the cuml wheel libraries.

[cjnolet]

/merge