[BUG] HDBSCAN precomputed metric AttributeError

RyanWalden20 commented 2 years ago

Describe the bug cuml.cluster.HDBSCAN does not appear to support precomputed distance matrix as the documentation suggests

Steps/Code to reproduce bug

from cuml.cluster import HDBSCAN
model = HDBSCAN(metric='precomputed')
model.fit(np.ones((3,3)))

AttributeError                            Traceback (most recent call last)
<command-3589287876064841> in <module>
      1 from cuml.cluster import HDBSCAN
      2 model = HDBSCAN(metric='precomputed')
----> 3 model.fit(np.ones((3,3)))

/databricks/python/lib/python3.8/site-packages/cuml/internals/api_decorators.py in inner_with_setters(*args, **kwargs)
    407                                 target_val=target_val)
    408 
--> 409                 return func(*args, **kwargs)
    410 
    411         @wraps(func)

cuml/cluster/hdbscan.pyx in cuml.cluster.hdbscan.HDBSCAN.fit()

cuml/common/base.pyx in cuml.common.base.Base.__getattr__()

Expected behavior The model to fit using the precomputed distance matrix

Environment details (please complete the following information):

Environment location: Databricks GPU Cluster Databricks Runtime Version 10.1 (includes Apache Spark 3.2.0, Scala 2.12) Public docker image: apfmcommunitysearch/rapids_databricks:latest Custom Docker Container with base image databricksruntime/gpu-conda:cuda11 Docker Container contains conda environment from rapidsai/rapidsai-core:21.12-cuda11.0-base-ubuntu18.04-py3.8
Linux Distro/Architecture: [Ubuntu 18.04 amd64]
GPU Model/Driver: g4dn.xlarge AWS instance type NVIDIA T4
CUDA: 11.0
Method of cuDF & cuML install: conda inside Docker
docker pull apfmcommunitysearch/rapids_databricks:latest
docker run apfmcommunitysearch/rapids_databricks:latest

Conda environment.yaml:

name: rapids
channels:
- gpuci
- rapidsai
- nvidia
- pytorch
- conda-forge
- anaconda
dependencies:
- _libgcc_mutex=0.1=conda_forge
- _openmp_mutex=4.5=1_llvm
- abseil-cpp=20210324.2=h9c3ff4c_0
- aiohttp=3.8.1=py38h497a2fe_0
- aiosignal=1.2.0=pyhd8ed1ab_0
- anyio=3.4.0=py38h578d9bd_0
- appdirs=1.4.4=pyh9f0ad1d_0
- argon2-cffi=21.1.0=py38h497a2fe_2
- arrow-cpp=5.0.0=py38h87d5a9a_15_cuda
- arrow-cpp-proc=3.0.0=cuda
- async-timeout=4.0.1=pyhd8ed1ab_0
- async_generator=1.10=py_0
- attrs=21.2.0=pyhd8ed1ab_0
- aws-c-auth=0.6.7=hfef2836_0
- aws-c-cal=0.5.12=h70efedd_7
- aws-c-common=0.6.17=h7f98852_0
- aws-c-compression=0.2.14=h7c7754b_7
- aws-c-event-stream=0.2.7=hb80ed28_31
- aws-c-http=0.6.10=h58a30cf_2
- aws-c-io=0.10.13=he836878_5
- aws-c-mqtt=0.7.9=h042a236_0
- aws-c-s3=0.1.27=hae5f17b_11
- aws-c-sdkutils=0.1.1=h7c7754b_4
- aws-checksums=0.1.12=h7c7754b_6
- aws-crt-cpp=0.17.8=h82bac0c_1
- aws-sdk-cpp=1.9.145=hfe59705_2
- backcall=0.2.0=pyh9f0ad1d_0
- backports=1.0=py_2
- backports.functools_lru_cache=1.6.4=pyhd8ed1ab_0
- bleach=4.1.0=pyhd8ed1ab_0
- blosc=1.21.0=h9c3ff4c_0
- bokeh=2.4.0=py38h578d9bd_0
- boost=1.74.0=py38h2b96118_4
- boost-cpp=1.74.0=h312852a_4
- brotli=1.0.9=h7f98852_6
- brotli-bin=1.0.9=h7f98852_6
- brotlipy=0.7.0=py38h497a2fe_1003
- brunsli=0.1=h9c3ff4c_0
- bzip2=1.0.8=h7f98852_4
- c-ares=1.18.1=h7f98852_0
- c-blosc2=2.0.4=h5f21a17_1
- ca-certificates=2021.10.8=ha878542_0
- cachetools=4.2.4=pyhd8ed1ab_0
- cairo=1.16.0=h6cf1ce9_1008
- certifi=2021.10.8=py38h578d9bd_1
- cffi=1.15.0=py38h3931269_0
- cfitsio=3.470=hb418390_7
- charls=2.2.0=h9c3ff4c_0
- charset-normalizer=2.0.9=pyhd8ed1ab_0
- click=8.0.3=py38h578d9bd_1
- click-plugins=1.1.1=py_0
- cligj=0.7.2=pyhd8ed1ab_1
- cloudpickle=2.0.0=pyhd8ed1ab_0
- colorama=0.4.4=pyh9f0ad1d_0
- colorcet=3.0.0=pyhd8ed1ab_0
- cryptography=36.0.1=py38h3e25421_0
- cucim=21.12.00=cuda_11_py38_g6d1f082_0
- cudatoolkit=11.0.221=h6bb024c_0
- cudf=21.12.02=cuda_11_py38_g06540b9b37_0
- cudf_kafka=21.12.02=py38_g06540b9b37_0
- cugraph=21.12.00=cuda11_py38_g3a43e9d0_0
- cuml=21.12.00=cuda11_py38_g04c4927f3_0
- cupy=9.6.0=py38h57017e2_0
- curl=7.80.0=h2574ce0_0
- cusignal=21.12.00=py37_g2bf865c_0
- cuspatial=21.12.00=py38_gab6748f_0
- custreamz=21.12.02=py38_g06540b9b37_0
- cuxfilter=21.12.00=py38_g2e0fb5a_0
- cycler=0.11.0=pyhd8ed1ab_0
- cyrus-sasl=2.1.27=h230043b_5
- cytoolz=0.11.2=py38h497a2fe_1
- dask=2021.11.2=pyhd8ed1ab_0
- dask-core=2021.11.2=pyhd8ed1ab_0
- dask-cuda=21.12.00=py38_0
- dask-cudf=21.12.02=cuda_11_py38_g06540b9b37_0
- datashader=0.11.1=pyh9f0ad1d_0
- datashape=0.5.4=py_1
- debugpy=1.5.1=py38h709712a_0
- decorator=5.1.0=pyhd8ed1ab_0
- defusedxml=0.7.1=pyhd8ed1ab_0
- distributed=2021.11.2=py38h578d9bd_0
- dlpack=0.5=h9c3ff4c_0
- entrypoints=0.3=py38h32f6830_1002
- expat=2.4.1=h9c3ff4c_0
- faiss-proc=1.0.0=cuda
- fastavro=1.4.7=py38h497a2fe_1
- fastrlock=0.8=py38h709712a_1
- fiona=1.8.20=py38hbb147eb_2
- font-ttf-dejavu-sans-mono=2.37=hab24e00_0
- font-ttf-inconsolata=3.000=h77eed37_0
- font-ttf-source-code-pro=2.038=h77eed37_0
- font-ttf-ubuntu=0.83=hab24e00_0
- fontconfig=2.13.1=hba837de_1005
- fonts-conda-ecosystem=1=0
- fonts-conda-forge=1=0
- fonttools=4.28.4=py38h497a2fe_0
- freetype=2.10.4=h0708190_1
- freexl=1.0.6=h7f98852_0
- frozenlist=1.2.0=py38h497a2fe_1
- fsspec=2021.11.1=pyhd8ed1ab_0
- gdal=3.3.2=py38h81a01a0_3
- geopandas=0.9.0=pyhd8ed1ab_1
- geopandas-base=0.9.0=pyhd8ed1ab_1
- geos=3.9.1=h9c3ff4c_2
- geotiff=1.7.0=h08e826d_2
- gettext=0.19.8.1=h73d1719_1008
- gflags=2.2.2=he1b5a44_1004
- giflib=5.2.1=h36c2ea0_2
- git=2.34.1=pl5321hc30692c_0
- glog=0.5.0=h48cff8f_0
- gpuci-tools=0.3.1=12
- grpc-cpp=1.41.1=h75e9d12_2
- hdf4=4.2.15=h10796ff_3
- hdf5=1.12.1=nompi_h2750804_103
- heapdict=1.0.1=py_0
- icu=68.2=h9c3ff4c_0
- idna=3.1=pyhd3deb0d_0
- imagecodecs=2021.8.26=py38hb5ce8f7_1
- imageio=2.13.3=pyh239f2a4_0
- importlib-metadata=4.8.3=py38h578d9bd_0
- importlib_metadata=4.8.3=hd8ed1ab_0
- importlib_resources=5.4.0=pyhd8ed1ab_0
- ipykernel=6.6.0=py38he5a9106_0
- ipython=7.30.1=py38h578d9bd_0
- ipython_genutils=0.2.0=py_1
- ipywidgets=7.6.5=pyhd8ed1ab_0
- jbig=2.1=h7f98852_2003
- jedi=0.18.1=py38h578d9bd_0
- jinja2=3.0.3=pyhd8ed1ab_0
- joblib=1.1.0=pyhd8ed1ab_0
- jpeg=9d=h36c2ea0_0
- json-c=0.15=h98cffda_0
- jsonschema=4.3.1=pyhd8ed1ab_0
- jupyter-server-proxy=3.2.0=pyhd8ed1ab_0
- jupyter_client=7.1.0=pyhd8ed1ab_0
- jupyter_core=4.9.1=py38h578d9bd_1
- jupyter_server=1.13.1=pyhd8ed1ab_0
- jupyterlab_pygments=0.1.2=pyh9f0ad1d_0
- jupyterlab_widgets=1.0.2=pyhd8ed1ab_0
- jxrlib=1.1=h7f98852_2
- kealib=1.4.14=h87e4c3c_3
- kiwisolver=1.3.2=py38h1fd1430_1
- krb5=1.19.2=hcc1bbae_3
- lcms2=2.12=hddcbb42_0
- ld_impl_linux-64=2.36.1=hea4e1c9_2
- lerc=3.0=h9c3ff4c_0
- libaec=1.0.6=h9c3ff4c_0
- libblas=3.9.0=12_linux64_openblas
- libbrotlicommon=1.0.9=h7f98852_6
- libbrotlidec=1.0.9=h7f98852_6
- libbrotlienc=1.0.9=h7f98852_6
- libcblas=3.9.0=12_linux64_openblas
- libcucim=21.12.00=cuda11_g6d1f082_0
- libcudf=21.12.02=cuda11_g06540b9b37_0
- libcudf_kafka=21.12.02=g06540b9b37_0
- libcugraph=21.12.00=cuda11_g3a43e9d0_0
- libcuml=21.12.00=cuda11_g04c4927f3_0
- libcumlprims=21.12.00=cuda11_g0a7f19f_0
- libcurl=7.80.0=h2574ce0_0
- libcusolver=11.3.2.107=hc875929_0
- libcuspatial=21.12.00=cuda11_gab6748f_0
- libdap4=3.20.6=hd7c4107_2
- libdeflate=1.8=h7f98852_0
- libedit=3.1.20191231=he28a2e2_2
- libev=4.33=h516909a_1
- libevent=2.1.10=h9b69904_4
- libfaiss=1.7.0=cuda110h8045045_8_cuda
- libffi=3.4.2=h7f98852_5
- libgcc-ng=9.4.0=hfa6338b_11
- libgcrypt=1.9.4=h7f98852_0
- libgdal=3.3.2=h6acdded_3
- libgfortran-ng=11.2.0=h69a702a_11
- libgfortran5=11.2.0=h5c6108e_11
- libglib=2.70.2=h174f98d_0
- libgpg-error=1.42=h9c3ff4c_0
- libgsasl=1.10.0=h5b4c23d_0
- libhwloc=2.3.0=h5e5b7d1_1
- libiconv=1.16=h516909a_0
- libkml=1.3.0=h238a007_1014
- liblapack=3.9.0=12_linux64_openblas
- libllvm11=11.1.0=hf817b99_2
- libnetcdf=4.8.1=nompi_hb3fd0d9_101
- libnghttp2=1.43.0=h812cca2_1
- libnsl=2.0.0=h7f98852_0
- libntlm=1.4=h7f98852_1002
- libopenblas=0.3.18=pthreads_h8fe5266_0
- libpng=1.6.37=h21135ba_2
- libpq=13.5=hd57d9b9_1
- libprotobuf=3.18.1=h780b84a_0
- librdkafka=1.6.1=hc49e61c_1
- librmm=21.12.00=cuda11_g957ad04_0
- librttopo=1.1.0=h1185371_6
- libsodium=1.0.18=h36c2ea0_1
- libspatialindex=1.9.3=h9c3ff4c_4
- libspatialite=5.0.1=h5cf074c_8
- libssh2=1.10.0=ha56f1ee_2
- libstdcxx-ng=9.4.0=h79bfe98_11
- libthrift=0.15.0=he6d91bd_1
- libtiff=4.3.0=h6f004c6_2
- libutf8proc=2.7.0=h7f98852_0
- libuuid=2.32.1=h7f98852_1000
- libuv=1.42.0=h7f98852_0
- libwebp=1.2.1=h3452ae3_0
- libwebp-base=1.2.1=h7f98852_0
- libxcb=1.13=h7f98852_1004
- libxgboost=1.5.0dev.rapidsai21.12=cuda11.0_0
- libxml2=2.9.12=h72842e0_0
- libzip=1.8.0=h4de3113_1
- libzlib=1.2.11=h36c2ea0_1013
- libzopfli=1.0.3=h9c3ff4c_0
- llvm-openmp=12.0.1=h4bd325d_1
- llvmlite=0.37.0=py38h4630a5e_1
- locket=0.2.0=py_2
- lz4-c=1.9.3=h9c3ff4c_1
- mapclassify=2.4.3=pyhd8ed1ab_0
- markdown=3.3.6=pyhd8ed1ab_0
- markupsafe=2.0.1=py38h497a2fe_1
- matplotlib-base=3.5.1=py38hf4fb855_0
- matplotlib-inline=0.1.3=pyhd8ed1ab_0
- mistune=0.8.4=py38h497a2fe_1005
- msgpack-python=1.0.3=py38h1fd1430_0
- multidict=5.2.0=py38h497a2fe_1
- multipledispatch=0.6.0=py_0
- munch=2.5.0=py_0
- munkres=1.1.4=pyh9f0ad1d_0
- nbclient=0.5.9=pyhd8ed1ab_0
- nbconvert=6.3.0=py38h578d9bd_1
- nbformat=5.1.3=pyhd8ed1ab_0
- nccl=2.11.4.1=h96e36e3_0
- ncurses=6.2=h58526e2_4
- nest-asyncio=1.5.4=pyhd8ed1ab_0
- networkx=2.6.3=pyhd8ed1ab_1
- nodejs=14.17.4=h92b4a50_0
- notebook=6.4.6=pyha770c72_0
- nspr=4.32=h9c3ff4c_1
- nss=3.73=hb5efdd6_0
- numba=0.54.1=py38h4bf6c61_0
- numpy=1.20.3=py38h9894fe3_1
- nvtx=0.2.3=py38h497a2fe_1
- olefile=0.46=pyh9f0ad1d_1
- openjpeg=2.4.0=hb52868f_1
- openssl=1.1.1l=h7f98852_0
- orc=1.7.1=h68e2c4e_0
- packaging=21.3=pyhd8ed1ab_0
- pandas=1.3.5=py38h43a58ef_0
- pandoc=2.16.2=h7f98852_0
- pandocfilters=1.5.0=pyhd8ed1ab_0
- panel=0.12.4=pyhd8ed1ab_0
- param=1.12.0=pyh6c4a22f_0
- parquet-cpp=1.5.1=1
- parso=0.8.3=pyhd8ed1ab_0
- partd=1.2.0=pyhd8ed1ab_0
- pcre=8.45=h9c3ff4c_0
- pcre2=10.37=h032f7d1_0
- perl=5.32.1=1_h7f98852_perl5
- pexpect=4.8.0=py38h32f6830_1
- pickleshare=0.7.5=py38h32f6830_1002
- pillow=8.4.0=py38h8e6f84c_0
- pip=21.3.1=pyhd8ed1ab_0
- pixman=0.40.0=h36c2ea0_0
- pooch=1.5.2=pyhd8ed1ab_0
- poppler=21.09.0=ha39eefc_3
- poppler-data=0.4.11=hd8ed1ab_0
- postgresql=13.5=h2510834_1
- proj=8.1.0=h277dcde_1
- prometheus_client=0.12.0=pyhd8ed1ab_0
- prompt-toolkit=3.0.24=pyha770c72_0
- protobuf=3.18.1=py38h709712a_0
- psutil=5.8.0=py38h497a2fe_2
- pthread-stubs=0.4=h36c2ea0_1001
- ptxcompiler=0.2.0=py38hb739d79_0
- ptyprocess=0.7.0=pyhd3deb0d_0
- py-xgboost=1.5.0dev.rapidsai21.12=cuda11.0py38_0
- pyarrow=5.0.0=py38hb90f683_15_cuda
- pycparser=2.21=pyhd8ed1ab_0
- pyct=0.4.6=py_0
- pyct-core=0.4.6=py_0
- pydeck=0.5.0=pyh9f0ad1d_0
- pyee=8.1.0=pyh9f0ad1d_0
- pygments=2.10.0=pyhd8ed1ab_0
- pynvml=11.4.1=pyhd8ed1ab_0
- pyopenssl=21.0.0=pyhd8ed1ab_0
- pyparsing=3.0.6=pyhd8ed1ab_0
- pyppeteer=0.2.6=pyhd8ed1ab_0
- pyproj=3.1.0=py38h3701b11_4
- pyrsistent=0.18.0=py38h497a2fe_0
- pysocks=1.7.1=py38h578d9bd_4
- python=3.8.12=hb7a2778_2_cpython
- python-confluent-kafka=1.6.0=py38h497a2fe_1
- python-dateutil=2.8.2=pyhd8ed1ab_0
- python_abi=3.8=2_cp38
- pytz=2021.3=pyhd8ed1ab_0
- pyviz_comms=2.1.0=pyhd8ed1ab_0
- pywavelets=1.2.0=py38h6c62de6_1
- pyyaml=6.0=py38h497a2fe_3
- pyzmq=22.3.0=py38h2035c66_1
- rapids=21.12.00=cuda11.0_py38_gc46440c_94
- rapids-xgboost=21.12.00=cuda11.0_py38_gc46440c_94
- re2=2021.11.01=h9c3ff4c_0
- readline=8.1=h46c0cb4_0
- requests=2.26.0=pyhd8ed1ab_1
- rmm=21.12.00=cuda11_py38_g957ad04_0_no_cma
- rtree=0.9.7=py38h02d302b_3
- s2n=1.3.0=h9b69904_0
- scikit-image=0.18.1=py38h51da96c_0
- scikit-learn=1.0.1=py38h1561384_3
- scipy=1.7.3=py38h56a6a73_0
- send2trash=1.8.0=pyhd8ed1ab_0
- setuptools=59.6.0=py38h578d9bd_0
- shapely=1.8.0=py38hb7fe4a8_0
- simpervisor=0.4=pyhd8ed1ab_0
- six=1.16.0=pyh6c4a22f_0
- snappy=1.1.8=he1b5a44_3
- sniffio=1.2.0=py38h578d9bd_2
- sortedcontainers=2.4.0=pyhd8ed1ab_0
- spdlog=1.8.5=h4bd325d_0
- sqlite=3.37.0=h9cd32fc_0
- streamz=0.6.3=pyh6c4a22f_0
- tblib=1.7.0=pyhd8ed1ab_0
- terminado=0.12.1=py38h578d9bd_1
- testpath=0.5.0=pyhd8ed1ab_0
- threadpoolctl=3.0.0=pyh8a188c0_0
- tifffile=2021.11.2=pyhd8ed1ab_0
- tiledb=2.3.4=he87e0bf_0
- tk=8.6.11=h27826a3_1
- toolz=0.11.2=pyhd8ed1ab_0
- tornado=6.1=py38h497a2fe_2
- tqdm=4.62.3=pyhd8ed1ab_0
- traitlets=5.1.1=pyhd8ed1ab_0
- treelite=2.1.0=py38hdd725b4_0
- typing-extensions=4.0.1=hd8ed1ab_0
- typing_extensions=4.0.1=pyha770c72_0
- tzcode=2021e=h7f98852_0
- tzdata=2021e=he74cb21_0
- ucx=1.11.2+gef2bbcf=cuda11.0_0
- ucx-proc=1.0.0=gpu
- ucx-py=0.23.0=py38_gef2bbcf_0
- unicodedata2=13.0.0.post2=py38h497a2fe_4
- urllib3=1.26.7=pyhd8ed1ab_0
- wcwidth=0.2.5=pyh9f0ad1d_2
- webencodings=0.5.1=py_1
- websocket-client=1.2.3=pyhd8ed1ab_0
- websockets=9.1=py38h497a2fe_0
- wheel=0.37.0=pyhd8ed1ab_1
- widgetsnbextension=3.5.2=py38h578d9bd_1
- xarray=0.20.2=pyhd8ed1ab_0
- xerces-c=3.2.3=h9d8b166_3
- xgboost=1.5.0dev.rapidsai21.12=cuda11.0py38_0
- xorg-kbproto=1.0.7=h7f98852_1002
- xorg-libice=1.0.10=h7f98852_0
- xorg-libsm=1.2.3=hd9c2040_1000
- xorg-libx11=1.7.2=h7f98852_0
- xorg-libxau=1.0.9=h7f98852_0
- xorg-libxdmcp=1.1.3=h7f98852_0
- xorg-libxext=1.3.4=h7f98852_1
- xorg-libxrender=0.9.10=h7f98852_1003
- xorg-renderproto=0.11.1=h7f98852_1002
- xorg-xextproto=7.3.0=h7f98852_1002
- xorg-xproto=7.0.31=h7f98852_1007
- xz=5.2.5=h516909a_1
- yaml=0.2.5=h516909a_0
- yarl=1.7.2=py38h497a2fe_1
- zeromq=4.3.4=h9c3ff4c_1
- zfp=0.5.5=h9c3ff4c_8
- zict=2.0.0=py_0
- zipp=3.6.0=pyhd8ed1ab_0
- zlib=1.2.11=h36c2ea0_1013
- zstd=1.5.0=ha95c52a_0
- pip:
- treelite-runtime==2.1.0
prefix: /opt/conda/envs/rapids

RyanWalden20 commented 2 years ago

Verified this error is reproducible in the environment of the Rapids Getting Started Colab notebook at https://rapids.ai/start.html

RichieHakim commented 2 years ago

I'm getting this error too. I accidentally made a duplicate issue (though slightly different). https://github.com/rapidsai/cuml/issues/4475 Also, using metric = 'manhattan' also crashes my kernel.

cjnolet commented 2 years ago

As mentioned in #4475, the current documentation is indeed misleading as it was copied directly from the hdbscan library even though the only supported metric currently is euclidean. We need to update the documentation for now but there's a pending item in #3879 to expose more metrics through hdbscan (and single-linkage hierarchical clustering).

github-actions[bot] commented 2 years ago

This issue has been labeled inactive-30d due to no recent activity in the past 30 days. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed. This issue will be labeled inactive-90d if there is no activity in the next 60 days.

github-actions[bot] commented 2 years ago

This issue has been labeled inactive-90d due to no recent activity in the past 90 days. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed.

CholoTook commented 5 months ago

Can I somehow make a euclidean space from a cosign space and feed it to hdbscan?

rapidsai / cuml

[BUG] HDBSCAN precomputed metric AttributeError #4460