search

rapidsai / cuml

cuML - RAPIDS Machine Learning Library
https://docs.rapids.ai/api/cuml/stable/
Apache License 2.0
4.2k stars 527 forks source link

[BUG] SGE, multi-gpu K-means & bad_alloc - the program hangs indefinitely #4876

Open mkvakic-srce opened 2 years ago

mkvakic-srce commented 2 years ago

Describe the bug

The bug concerns the usage of the multi-gpu K-means algorithm using the dask-scheduler/dask-cuda-worker API on a cluster with a SGE job scheduling system.

In case of a bad_alloc error (eg. too much data being processed) the program hangs indefinitely with 0% GPU usage on the master and 100% on the slave nodes, and requires stopping it manually (using qdel)

On the other hand, when not being submitted through SGE, the job ends gracefully after the inactivity period as defined by the dask-scheduler (in spite of the bad_alloc error).

Running the code normally over SGE (when not using too much data) is executed ok.

Any help would be much appreciated on how to reconcile the dask-scheduler API with SGE in case of a bad_alloc error

Steps/Code to reproduce bug

kmeans.sge:

#$ -cwd
#$ -o output/
#$ -e output/

module load rapids
env NTHREADS=4 ./dask_start.sh
env NSAMPLES=5e8 \
  taskset -c 0,2,4,6,8,10,12,14,16,18 \
  env CUDA_VISIBLE_DEVICES=$( cat $TMPDIR/gpu ) \
  python kmeans.py
dask_start.sh ```bash #$!/bin/bash scheduler_log="$( dirname $SGE_STDOUT_PATH )/scheduler.log.$JOB_ID" taskset -c 0,2,4,6,8,10,12,14,16,18 \ env CUDA_VISIBLE_DEVICES=$( cat $TMPDIR/gpu ) \ env DASK_DISTRIBUTED__COMM__UCX__CREATE_CUDA_CONTEXT=True \ dask-scheduler \ --no-show \ --no-dashboard \ --idle-timeout "60 seconds" \ --scheduler-file scheduler.info \ --protocol ucx \ --interface ib0 > $scheduler_log 2>&1 & # dask-cuda-worker while IFS=" " read -r machine number gpufull bla do gpu=${gpufull//".q@$machine"/} gpu=${gpu//"gpu."/} ssh $machine "cd $PWD; export JOB_ID=$JOB_ID; export SGE_STDOUT_PATH=$SGE_STDOUT_PATH; export CUDA_VISIBLE_DEVICES=$gpu; export NTHREADS=${NTHREADS:-1}; ./dask_worker.sh" & done < $PE_HOSTFILE ```
dask_worker.sh ```bash #!/bin/bash module load rapids workers_log="$( dirname $SGE_STDOUT_PATH )/${HOSTNAME}_${CUDA_VISIBLE_DEVICES}_workers.log.$JOB_ID" taskset -c 0,2,4,6,8,10,12,14,16,18 \ env UCX_MEMTYPE_REG_WHOLE_ALLOC_TYPES=cuda \ dask-cuda-worker \ --no-dashboard \ --nthreads ${NTHREADS:-1} \ --scheduler-file scheduler.info \ --interface ib0 > $workers_log 2>&1 & ```
kmeans.py ```python import os import json import time from cuml.dask.cluster.kmeans import KMeans from cuml.dask.datasets import make_blobs from dask.distributed import Client if __name__ == '__main__': # client with open('./scheduler.info') as f: dictionary = json.load(f) address = dictionary['address'] client = Client(address) tries = 0 while len(client.has_what()) == 0: print( 'kmeans_dask_demo.py: trying to connect to scheduler (try %i)' % tries ) tries += 1 time.sleep(5) if tries == 4: print( 'kmeans_dask_demo.py: workers not initiated, exiting' ) sys.exit() # parameters n_samples = int(float(os.environ['NSAMPLES'])) n_features = 2 n_clusters = 5 random_state = 0 # make_blobs device_data, _ = make_blobs( n_samples=n_samples, n_features=n_features, centers=n_clusters, client=client, ) # kmeans kmeans = KMeans( init="k-means||", n_clusters=n_clusters, oversampling_factor=40, random_state=random_state, ) kmeans.fit(device_data) # shutdown client.shutdown() ```

Expected behavior

After the bad_alloc error happens, for the dask-scheduler to stop after the designated period of inactivity and for the SGE job to finish, instead of needing to manually shut it down (via qdel)

Environment details (please complete the following information):

conda list output ```bash _libgcc_mutex 0.1 conda_forge conda-forge _openmp_mutex 4.5 2_gnu conda-forge abseil-cpp 20210324.2 h9c3ff4c_0 conda-forge aiohttp 3.8.1 py38h0a891b7_1 conda-forge aiosignal 1.2.0 pyhd8ed1ab_0 conda-forge anyio 3.6.1 pyhd8ed1ab_1 conda-forge aom 3.4.0 h27087fc_1 conda-forge appdirs 1.4.4 pyh9f0ad1d_0 conda-forge argon2-cffi 21.3.0 pyhd8ed1ab_0 conda-forge argon2-cffi-bindings 21.2.0 py38h0a891b7_2 conda-forge arrow-cpp 8.0.1 py38ha7276ea_0_cpu conda-forge asttokens 2.0.8 pyhd8ed1ab_0 conda-forge async-timeout 4.0.2 pyhd8ed1ab_0 conda-forge attrs 22.1.0 pyh71513ae_1 conda-forge aws-c-cal 0.5.11 h95a6274_0 conda-forge aws-c-common 0.6.2 h7f98852_0 conda-forge aws-c-event-stream 0.2.7 h3541f99_13 conda-forge aws-c-io 0.10.5 hfb6a706_0 conda-forge aws-checksums 0.1.11 ha31a3da_7 conda-forge aws-sdk-cpp 1.8.186 hb4091e7_3 conda-forge backcall 0.2.0 pyh9f0ad1d_0 conda-forge backports 1.0 py_2 conda-forge backports.functools_lru_cache 1.6.4 pyhd8ed1ab_0 conda-forge backports.zoneinfo 0.2.1 py38h0a891b7_5 conda-forge beautifulsoup4 4.11.1 pyha770c72_0 conda-forge bleach 5.0.1 pyhd8ed1ab_0 conda-forge blosc 1.21.1 h83bc5f7_3 conda-forge bokeh 2.4.3 pyhd8ed1ab_3 conda-forge boost 1.74.0 py38h2b96118_5 conda-forge boost-cpp 1.74.0 h6cacc03_7 conda-forge branca 0.5.0 pyhd8ed1ab_0 conda-forge brotli 1.0.9 h166bdaf_7 conda-forge brotli-bin 1.0.9 h166bdaf_7 conda-forge brotlipy 0.7.0 py38h0a891b7_1004 conda-forge brunsli 0.1 h9c3ff4c_0 conda-forge bzip2 1.0.8 h7f98852_4 conda-forge c-ares 1.18.1 h7f98852_0 conda-forge c-blosc2 2.2.0 h7a311fb_0 conda-forge ca-certificates 2022.6.15 ha878542_0 conda-forge cachetools 5.2.0 pyhd8ed1ab_0 conda-forge cairo 1.16.0 ha12eb4b_1010 conda-forge certifi 2022.6.15 py38h578d9bd_0 conda-forge cffi 1.15.1 py38h4a40e3a_0 conda-forge cfitsio 4.1.0 hd9d235c_0 conda-forge charls 2.3.4 h9c3ff4c_0 conda-forge charset-normalizer 2.1.0 pyhd8ed1ab_0 conda-forge click 8.1.3 py38h578d9bd_0 conda-forge click-plugins 1.1.1 py_0 conda-forge cligj 0.7.2 pyhd8ed1ab_1 conda-forge cloudpickle 2.1.0 pyhd8ed1ab_0 conda-forge colorama 0.4.5 pyhd8ed1ab_0 conda-forge colorcet 3.0.0 pyhd8ed1ab_0 conda-forge cryptography 37.0.4 py38h2b5fc30_0 conda-forge cucim 22.08.00a220817 cuda_11_py38_g8136f07_62 rapidsai-nightly cuda-python 11.7.0 py38h3fd9d12_0 nvidia cudatoolkit 11.2.72 h2bc3f7f_0 nvidia cudf 22.08.00a220817 cuda_11_py38_gdccb586111_303 rapidsai-nightly cudf_kafka 22.08.00a220817 py38_gdccb586111_303 rapidsai-nightly cugraph 22.08.00a220817 cuda11_py38_gb0280511_131 rapidsai-nightly cuml 22.08.00a220817 cuda11_py38_g63e880778_51 rapidsai-nightly cupy 10.6.0 py38h405e1b6_0 conda-forge curl 7.83.1 h7bff187_0 conda-forge cusignal 22.08.00a220817 py39_gb1de59c_17 rapidsai-nightly cuspatial 22.08.00a220817 py38_g66d4739_51 rapidsai-nightly custreamz 22.08.00a220817 py38_gdccb586111_303 rapidsai-nightly cuxfilter 22.08.00a220817 py38_gdc8653f_19 rapidsai-nightly cycler 0.11.0 pyhd8ed1ab_0 conda-forge cyrus-sasl 2.1.27 h230043b_5 conda-forge cytoolz 0.12.0 py38h0a891b7_0 conda-forge dask 2022.7.1 pyhd8ed1ab_0 conda-forge dask-core 2022.7.1 pyhd8ed1ab_0 conda-forge dask-cuda 22.08.00a220817 py38_g9860cad_36 rapidsai-nightly dask-cudf 22.08.00a220817 cuda_11_py38_gdccb586111_303 rapidsai-nightly dask-glm 0.2.0 py_1 conda-forge dask-ml 2022.5.27 pyhd8ed1ab_0 conda-forge dask-sql 2022.8.0 pyhd8ed1ab_0 conda-forge datashader 0.13.1a py_0 rapidsai-nightly datashape 0.5.4 py_1 conda-forge dav1d 1.0.0 h166bdaf_1 conda-forge debugpy 1.6.3 py38hfa26641_0 conda-forge decorator 5.1.1 pyhd8ed1ab_0 conda-forge defusedxml 0.7.1 pyhd8ed1ab_0 conda-forge distributed 2022.7.1 pyhd8ed1ab_0 conda-forge dlpack 0.5 h9c3ff4c_0 conda-forge entrypoints 0.4 pyhd8ed1ab_0 conda-forge executing 0.10.0 pyhd8ed1ab_0 conda-forge expat 2.4.8 h27087fc_0 conda-forge faiss-proc 1.0.0 cuda conda-forge fastapi 0.79.0 pyhd8ed1ab_0 conda-forge fastavro 1.6.0 py38h0a891b7_0 conda-forge fastrlock 0.8 py38hfa26641_2 conda-forge fiona 1.8.21 py38ha349e16_1 conda-forge flit-core 3.7.1 pyhd8ed1ab_0 conda-forge folium 0.12.1.post1 pyhd8ed1ab_1 conda-forge font-ttf-dejavu-sans-mono 2.37 hab24e00_0 conda-forge font-ttf-inconsolata 3.000 h77eed37_0 conda-forge font-ttf-source-code-pro 2.038 h77eed37_0 conda-forge font-ttf-ubuntu 0.83 hab24e00_0 conda-forge fontconfig 2.14.0 h8e229c2_0 conda-forge fonts-conda-ecosystem 1 0 conda-forge fonts-conda-forge 1 0 conda-forge fonttools 4.36.0 py38h0a891b7_0 conda-forge freetype 2.12.1 hca18f0e_0 conda-forge freexl 1.0.6 h7f98852_0 conda-forge frozenlist 1.3.1 py38h0a891b7_0 conda-forge fsspec 2022.7.1 pyhd8ed1ab_0 conda-forge gdal 3.4.3 py38hdd69c9e_0 conda-forge geopandas 0.11.1 pyhd8ed1ab_0 conda-forge geopandas-base 0.11.1 pyha770c72_0 conda-forge geos 3.10.2 h9c3ff4c_0 conda-forge geotiff 1.7.1 h509b78c_1 conda-forge gettext 0.19.8.1 h73d1719_1008 conda-forge gflags 2.2.2 he1b5a44_1004 conda-forge giflib 5.2.1 h36c2ea0_2 conda-forge glog 0.6.0 h6f12383_0 conda-forge grpc-cpp 1.45.2 h9d3bbbb_5 conda-forge h11 0.13.0 pyhd8ed1ab_1 conda-forge hdf4 4.2.15 h9772cbc_4 conda-forge hdf5 1.12.1 nompi_h2386368_104 conda-forge heapdict 1.0.1 py_0 conda-forge holoviews 1.14.6 pyhd8ed1ab_0 conda-forge icu 69.1 h9c3ff4c_0 conda-forge idna 3.3 pyhd8ed1ab_0 conda-forge imagecodecs 2022.8.8 py38h2b96160_3 conda-forge imageio 2.21.1 pyhfa7a67d_0 conda-forge importlib-metadata 4.11.4 py38h578d9bd_0 conda-forge importlib_resources 5.9.0 pyhd8ed1ab_0 conda-forge ipykernel 6.15.1 pyh210e3f2_0 conda-forge ipython 8.4.0 py38h578d9bd_0 conda-forge ipython_genutils 0.2.0 py_1 conda-forge ipywidgets 7.7.1 pyhd8ed1ab_0 conda-forge jbig 2.1 h7f98852_2003 conda-forge jedi 0.18.1 pyhd8ed1ab_2 conda-forge jinja2 3.1.2 pyhd8ed1ab_1 conda-forge joblib 1.1.0 pyhd8ed1ab_0 conda-forge jpeg 9e h166bdaf_2 conda-forge jpype1 1.4.0 py38h43d8883_0 conda-forge json-c 0.16 hc379101_0 conda-forge jsonschema 4.9.1 pyhd8ed1ab_0 conda-forge jupyter-server-proxy 3.2.1 pyhd8ed1ab_0 conda-forge jupyter_client 7.3.4 pyhd8ed1ab_0 conda-forge jupyter_core 4.11.1 py38h578d9bd_0 conda-forge jupyter_server 1.18.1 pyhd8ed1ab_0 conda-forge jupyterlab_pygments 0.2.2 pyhd8ed1ab_0 conda-forge jupyterlab_widgets 1.1.1 pyhd8ed1ab_0 conda-forge jxrlib 1.1 h7f98852_2 conda-forge kealib 1.4.15 hfe1a663_0 conda-forge keyutils 1.6.1 h166bdaf_0 conda-forge kiwisolver 1.4.4 py38h43d8883_0 conda-forge krb5 1.19.3 h3790be6_0 conda-forge lcms2 2.12 hddcbb42_0 conda-forge ld_impl_linux-64 2.36.1 hea4e1c9_2 conda-forge lerc 4.0.0 h27087fc_0 conda-forge libaec 1.0.6 h9c3ff4c_0 conda-forge libavif 0.10.1 h166bdaf_1 conda-forge libblas 3.9.0 16_linux64_openblas conda-forge libbrotlicommon 1.0.9 h166bdaf_7 conda-forge libbrotlidec 1.0.9 h166bdaf_7 conda-forge libbrotlienc 1.0.9 h166bdaf_7 conda-forge libcblas 3.9.0 16_linux64_openblas conda-forge libcrc32c 1.1.2 h9c3ff4c_0 conda-forge libcucim 22.08.00a220817 cuda11_g8136f07_62 rapidsai-nightly libcudf 22.08.00a220817 cuda11_gdccb586111_303 rapidsai-nightly libcudf_kafka 22.08.00a220817 gdccb586111_303 rapidsai-nightly libcugraph 22.08.00a220817 cuda11_gb0280511_131 rapidsai-nightly libcugraph_etl 22.08.00a220817 cuda11_gb0280511_131 rapidsai-nightly libcugraphops 22.08.00 cuda11_g7f764d0_0 nvidia libcuml 22.08.00a220817 cuda11_g73b8d00d0_52 rapidsai-nightly libcumlprims 22.08.00 cuda11_g1770e60_0 nvidia libcurl 7.83.1 h7bff187_0 conda-forge libcusolver 11.4.0.1 0 nvidia libcusparse 11.7.4.91 0 nvidia libcuspatial 22.08.00a220817 cuda11_g66d4739_51 rapidsai-nightly libdap4 3.20.6 hd7c4107_2 conda-forge libdeflate 1.13 h166bdaf_0 conda-forge libedit 3.1.20191231 he28a2e2_2 conda-forge libev 4.33 h516909a_1 conda-forge libevent 2.1.10 h9b69904_4 conda-forge libfaiss 1.7.0 cuda112h5bea7ad_8_cuda conda-forge libffi 3.4.2 h7f98852_5 conda-forge libgcc-ng 12.1.0 h8d9b700_16 conda-forge libgcrypt 1.10.1 h166bdaf_0 conda-forge libgdal 3.4.3 h56144a5_0 conda-forge libgfortran-ng 12.1.0 h69a702a_16 conda-forge libgfortran5 12.1.0 hdcd56e2_16 conda-forge libglib 2.72.1 h2d90d5f_0 conda-forge libgomp 12.1.0 h8d9b700_16 conda-forge libgoogle-cloud 1.40.2 habd0e3a_0 conda-forge libgpg-error 1.45 hc0c96e0_0 conda-forge libgsasl 1.10.0 h5b4c23d_0 conda-forge libiconv 1.16 h516909a_0 conda-forge libkml 1.3.0 h238a007_1014 conda-forge liblapack 3.9.0 16_linux64_openblas conda-forge libllvm11 11.1.0 hf817b99_3 conda-forge libnetcdf 4.8.1 nompi_h329d8a1_102 conda-forge libnghttp2 1.47.0 hdcd2b5c_1 conda-forge libnsl 2.0.0 h7f98852_0 conda-forge libntlm 1.4 h7f98852_1002 conda-forge libopenblas 0.3.21 pthreads_h78a6416_1 conda-forge libpng 1.6.37 h753d276_4 conda-forge libpq 14.2 hd57d9b9_0 conda-forge libprotobuf 3.20.1 h6239696_1 conda-forge libraft-distance 22.08.00a220817 cuda11_g9070c305_70 rapidsai-nightly libraft-headers 22.08.00a220817 cuda11_g9070c305_70 rapidsai-nightly libraft-nn 22.08.00a220817 cuda11_g9070c305_70 rapidsai-nightly librdkafka 1.7.0 hc49e61c_1 conda-forge librmm 22.08.00 h65003ff_0 conda-forge librttopo 1.1.0 hf69c175_9 conda-forge libsodium 1.0.18 h36c2ea0_1 conda-forge libspatialindex 1.9.3 h9c3ff4c_4 conda-forge libspatialite 5.0.1 ha867d66_15 conda-forge libsqlite 3.39.2 h753d276_1 conda-forge libssh2 1.10.0 haa6b8db_3 conda-forge libstdcxx-ng 12.1.0 ha89aaad_16 conda-forge libthrift 0.16.0 h519c5ea_1 conda-forge libtiff 4.4.0 h0e0dad5_3 conda-forge libutf8proc 2.7.0 h7f98852_0 conda-forge libuuid 2.32.1 h7f98852_1000 conda-forge libuv 1.43.0 h7f98852_0 conda-forge libwebp 1.2.4 h522a892_0 conda-forge libwebp-base 1.2.4 h166bdaf_0 conda-forge libxcb 1.13 h7f98852_1004 conda-forge libxgboost 1.6.0dev.rapidsai22.08 cuda_11_0 rapidsai-nightly libxml2 2.9.12 h885dcf4_1 conda-forge libxslt 1.1.33 h0ef7038_3 conda-forge libzip 1.9.2 hc869a4a_1 conda-forge libzlib 1.2.12 h166bdaf_2 conda-forge libzopfli 1.0.3 h9c3ff4c_0 conda-forge llvmlite 0.38.1 py38h38d86a4_0 conda-forge locket 1.0.0 pyhd8ed1ab_0 conda-forge lxml 4.8.0 py38h0a891b7_2 conda-forge lz4 4.0.0 py38h1bf946c_2 conda-forge lz4-c 1.9.3 h9c3ff4c_1 conda-forge mapclassify 2.4.3 pyhd8ed1ab_0 conda-forge markdown 3.4.1 pyhd8ed1ab_0 conda-forge markupsafe 2.1.1 py38h0a891b7_1 conda-forge matplotlib-base 3.5.3 py38h826bfd8_1 conda-forge matplotlib-inline 0.1.5 pyhd8ed1ab_0 conda-forge mistune 0.8.4 py38h497a2fe_1005 conda-forge msgpack-python 1.0.4 py38h43d8883_0 conda-forge multidict 6.0.2 py38h0a891b7_1 conda-forge multipledispatch 0.6.0 py_0 conda-forge munch 2.5.0 py_0 conda-forge munkres 1.1.4 pyh9f0ad1d_0 conda-forge nbclient 0.6.6 pyhd8ed1ab_0 conda-forge nbconvert 6.5.3 pyhd8ed1ab_0 conda-forge nbconvert-core 6.5.3 pyhd8ed1ab_0 conda-forge nbconvert-pandoc 6.5.3 pyhd8ed1ab_0 conda-forge nbformat 5.4.0 pyhd8ed1ab_0 conda-forge nccl 2.13.4.1 h0800d71_0 conda-forge ncurses 6.3 h27087fc_1 conda-forge nest-asyncio 1.5.5 pyhd8ed1ab_0 conda-forge networkx 2.6.3 pyhd8ed1ab_1 conda-forge nodejs 17.8.0 h784f1fd_0 conda-forge notebook 6.4.12 pyha770c72_0 conda-forge nspr 4.32 h9c3ff4c_1 conda-forge nss 3.78 h2350873_0 conda-forge numba 0.55.2 py38hdc3674a_0 conda-forge numpy 1.22.0 py38h6ae9a64_1 conda-forge nvtx 0.2.3 py38h497a2fe_1 conda-forge openjdk 8.0.332 h166bdaf_0 conda-forge openjpeg 2.4.0 hb52868f_1 conda-forge openssl 1.1.1q h166bdaf_0 conda-forge orc 1.7.5 h6c59b99_0 conda-forge packaging 21.3 pyhd8ed1ab_0 conda-forge pandas 1.4.3 py38h47df419_0 conda-forge pandoc 2.19 ha770c72_0 conda-forge pandocfilters 1.5.0 pyhd8ed1ab_0 conda-forge panel 0.12.7 pyhd8ed1ab_0 conda-forge param 1.12.2 pyh6c4a22f_0 conda-forge parquet-cpp 1.5.1 2 conda-forge parso 0.8.3 pyhd8ed1ab_0 conda-forge partd 1.3.0 pyhd8ed1ab_0 conda-forge pcre 8.45 h9c3ff4c_0 conda-forge pexpect 4.8.0 pyh9f0ad1d_2 conda-forge pickleshare 0.7.5 py_1003 conda-forge pillow 9.2.0 py38h0ee0e06_1 conda-forge pip 22.2.2 pyhd8ed1ab_0 conda-forge pixman 0.40.0 h36c2ea0_0 conda-forge pkgutil-resolve-name 1.3.10 pyhd8ed1ab_0 conda-forge poppler 22.04.0 h1434ded_1 conda-forge poppler-data 0.4.11 hd8ed1ab_0 conda-forge postgresql 14.2 h2510834_0 conda-forge proj 9.0.0 h93bde94_1 conda-forge prometheus_client 0.14.1 pyhd8ed1ab_0 conda-forge prompt-toolkit 3.0.30 pyha770c72_0 conda-forge protobuf 3.20.1 py38hfa26641_0 conda-forge psutil 5.9.1 py38h0a891b7_0 conda-forge pthread-stubs 0.4 h36c2ea0_1001 conda-forge ptxcompiler 0.4.0 py38h7525318_0 conda-forge ptyprocess 0.7.0 pyhd3deb0d_0 conda-forge pure_eval 0.2.2 pyhd8ed1ab_0 conda-forge py-xgboost 1.6.0dev.rapidsai22.08 cuda_11_py38_0 rapidsai-nightly pyarrow 8.0.1 py38h9f6a473_0_cpu conda-forge pycparser 2.21 pyhd8ed1ab_0 conda-forge pyct 0.4.6 py_0 conda-forge pyct-core 0.4.6 py_0 conda-forge pydantic 1.9.2 py38h0a891b7_0 conda-forge pydeck 0.5.0 pyh9f0ad1d_0 conda-forge pyee 8.1.0 pyhd8ed1ab_0 conda-forge pygments 2.13.0 pyhd8ed1ab_0 conda-forge pylibcugraph 22.08.00a220817 cuda11_py38_gb0280511_131 rapidsai-nightly pynvml 11.4.1 pyhd8ed1ab_0 conda-forge pyopenssl 22.0.0 pyhd8ed1ab_0 conda-forge pyparsing 3.0.9 pyhd8ed1ab_0 conda-forge pyppeteer 1.0.2 pyhd8ed1ab_0 conda-forge pyproj 3.3.1 py38h5b5ac8f_0 conda-forge pyraft 22.08.00a220817 cuda11_py38_gf974c7bd_69 rapidsai-nightly pyrsistent 0.18.1 py38h0a891b7_1 conda-forge pysocks 1.7.1 py38h578d9bd_5 conda-forge python 3.8.13 h582c2e5_0_cpython conda-forge python-confluent-kafka 1.7.0 py38h497a2fe_2 conda-forge python-dateutil 2.8.2 pyhd8ed1ab_0 conda-forge python-fastjsonschema 2.16.1 pyhd8ed1ab_0 conda-forge python-tzdata 2022.2 pyhd8ed1ab_0 conda-forge python_abi 3.8 2_cp38 conda-forge pytz 2022.2.1 pyhd8ed1ab_0 conda-forge pytz-deprecation-shim 0.1.0.post0 py38h578d9bd_2 conda-forge pyviz_comms 2.2.0 pyhd8ed1ab_0 conda-forge pywavelets 1.3.0 py38h71d37f0_1 conda-forge pyyaml 6.0 py38h0a891b7_4 conda-forge pyzmq 23.2.1 py38hfc09fa9_0 conda-forge rapids 22.08.00a220817 cuda11_py38_g0b0e275_81 rapidsai-nightly rapids-xgboost 22.08.00a220817 cuda11_py38_g0b0e275_81 rapidsai-nightly re2 2022.06.01 h27087fc_0 conda-forge readline 8.1.2 h0f457ee_0 conda-forge requests 2.28.1 pyhd8ed1ab_0 conda-forge rmm 22.08.00a220817 cuda11_py38_gf6bf0471_62 rapidsai-nightly rtree 1.0.0 py38h02d302b_1 conda-forge s2n 1.0.10 h9b69904_0 conda-forge scikit-image 0.19.3 py38h47df419_1 conda-forge scikit-learn 1.1.2 py38h0b08f9b_0 conda-forge scipy 1.9.0 py38hea3f02b_0 conda-forge send2trash 1.8.0 pyhd8ed1ab_0 conda-forge setuptools 60.10.0 py38h578d9bd_0 conda-forge shapely 1.8.2 py38h97f7145_1 conda-forge simpervisor 0.4 pyhd8ed1ab_0 conda-forge six 1.16.0 pyh6c4a22f_0 conda-forge snappy 1.1.9 hbd366e4_1 conda-forge sniffio 1.2.0 py38h578d9bd_3 conda-forge sortedcontainers 2.4.0 pyhd8ed1ab_0 conda-forge soupsieve 2.3.2.post1 pyhd8ed1ab_0 conda-forge spdlog 1.10.0 h924138e_0 conda-forge sqlite 3.39.2 h4ff8645_1 conda-forge stack_data 0.4.0 pyhd8ed1ab_0 conda-forge starlette 0.19.1 pyhd8ed1ab_0 conda-forge streamz 0.6.4 pyh6c4a22f_0 conda-forge tabulate 0.8.10 pyhd8ed1ab_0 conda-forge tblib 1.7.0 pyhd8ed1ab_0 conda-forge terminado 0.15.0 py38h578d9bd_0 conda-forge threadpoolctl 3.1.0 pyh8a188c0_0 conda-forge thrust 1.16.0 h0800d71_1 conda-forge tifffile 2022.8.12 pyhd8ed1ab_0 conda-forge tiledb 2.8.3 h1e4a385_1 conda-forge tinycss2 1.1.1 pyhd8ed1ab_0 conda-forge tk 8.6.12 h27826a3_0 conda-forge toolz 0.12.0 pyhd8ed1ab_0 conda-forge tornado 6.1 py38h0a891b7_3 conda-forge tqdm 4.64.0 pyhd8ed1ab_0 conda-forge traitlets 5.3.0 pyhd8ed1ab_0 conda-forge treelite 2.4.0 py38hdd725b4_0 conda-forge treelite-runtime 2.4.0 pypi_0 pypi typing-extensions 4.3.0 hd8ed1ab_0 conda-forge typing_extensions 4.3.0 pyha770c72_0 conda-forge tzcode 2022c h166bdaf_0 conda-forge tzdata 2022c h191b570_0 conda-forge tzlocal 4.2 py38h578d9bd_1 conda-forge ucx 1.13.0 h538f049_0 conda-forge ucx-proc 1.0.0 gpu conda-forge ucx-py 0.27.00a220810 py38_ge9e81f8_29 rapidsai-nightly unicodedata2 14.0.0 py38h0a891b7_1 conda-forge urllib3 1.26.11 pyhd8ed1ab_0 conda-forge uvicorn 0.18.2 py38h578d9bd_2 conda-forge wcwidth 0.2.5 pyh9f0ad1d_2 conda-forge webencodings 0.5.1 py_1 conda-forge websocket-client 1.3.3 pyhd8ed1ab_0 conda-forge websockets 10.3 py38h0a891b7_0 conda-forge wheel 0.37.1 pyhd8ed1ab_0 conda-forge widgetsnbextension 3.6.1 pyha770c72_0 conda-forge xarray 2022.6.0 pyhd8ed1ab_1 conda-forge xerces-c 3.2.3 h8ce2273_4 conda-forge xgboost 1.6.0dev.rapidsai22.08 cuda_11_py38_0 rapidsai-nightly xorg-kbproto 1.0.7 h7f98852_1002 conda-forge xorg-libice 1.0.10 h7f98852_0 conda-forge xorg-libsm 1.2.3 hd9c2040_1000 conda-forge xorg-libx11 1.7.2 h7f98852_0 conda-forge xorg-libxau 1.0.9 h7f98852_0 conda-forge xorg-libxdmcp 1.1.3 h7f98852_0 conda-forge xorg-libxext 1.3.4 h7f98852_1 conda-forge xorg-libxrender 0.9.10 h7f98852_1003 conda-forge xorg-renderproto 0.11.1 h7f98852_1002 conda-forge xorg-xextproto 7.3.0 h7f98852_1002 conda-forge xorg-xproto 7.0.31 h7f98852_1007 conda-forge xyzservices 2022.6.0 pyhd8ed1ab_0 conda-forge xz 5.2.6 h166bdaf_0 conda-forge yaml 0.2.5 h7f98852_2 conda-forge yarl 1.7.2 py38h0a891b7_2 conda-forge zeromq 4.3.4 h9c3ff4c_1 conda-forge zfp 1.0.0 h27087fc_1 conda-forge zict 2.2.0 pyhd8ed1ab_0 conda-forge zipp 3.8.1 pyhd8ed1ab_0 conda-forge zlib 1.2.12 h166bdaf_2 conda-forge zlib-ng 2.0.6 h166bdaf_0 conda-forge zstd 1.5.2 h6239696_4 conda-forge ```
github-actions[bot] commented 2 years ago

This issue has been labeled inactive-30d due to no recent activity in the past 30 days. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed. This issue will be labeled inactive-90d if there is no activity in the next 60 days.