search

rapidsai / cudf

cuDF - GPU DataFrame Library
https://docs.rapids.ai/api/cudf/stable/
Apache License 2.0
8.35k stars 890 forks source link

[BUG] Starting a LocalCUDACluster causes "no space left on device" error in CUDA 11.8 on ARM #12683

Closed taureandyernv closed 1 year ago

taureandyernv commented 1 year ago

Describe the bug If you try to start a LocalCUDACluster, it throws an OSError: [Errno 28] No space left on device, despite there being plenty of storage space on the host device, and an mostly empty GPU.

Steps/Code to reproduce bug

from dask.distributed import Client, wait
from dask_cuda import LocalCUDACluster

cluster = LocalCUDACluster(threads_per_worker=1)
client = Client(cluster)

outputs: 

<<truncated for space>>
Task exception was never retrieved
future: <Task finished name='Task-68' coro=<_wrap_awaitable() done, defined at /opt/conda/envs/rapids/lib/python3.8/site-packages/distributed/deploy/spec.py:115> exception=RuntimeError('Nanny failed to start.')>
Traceback (most recent call last):
  File "/opt/conda/envs/rapids/lib/python3.8/site-packages/distributed/core.py", line 528, in start
    await asyncio.wait_for(self.start_unsafe(), timeout=timeout)
  File "/opt/conda/envs/rapids/lib/python3.8/asyncio/tasks.py", line 455, in wait_for
    return await fut
  File "/opt/conda/envs/rapids/lib/python3.8/site-packages/distributed/nanny.py", line 366, in start_unsafe
    response = await self.instantiate()
  File "/opt/conda/envs/rapids/lib/python3.8/site-packages/distributed/nanny.py", line 442, in instantiate
    result = await self.process.start()
  File "/opt/conda/envs/rapids/lib/python3.8/site-packages/distributed/nanny.py", line 678, in start
    self.init_result_q = init_q = get_mp_context().Queue()
  File "/opt/conda/envs/rapids/lib/python3.8/multiprocessing/context.py", line 103, in Queue
    return Queue(maxsize, ctx=self.get_context())
  File "/opt/conda/envs/rapids/lib/python3.8/multiprocessing/queues.py", line 42, in __init__
    self._rlock = ctx.Lock()
  File "/opt/conda/envs/rapids/lib/python3.8/multiprocessing/context.py", line 68, in Lock
    return Lock(ctx=self.get_context())
  File "/opt/conda/envs/rapids/lib/python3.8/multiprocessing/synchronize.py", line 162, in __init__
    SemLock.__init__(self, SEMAPHORE, 1, 1, ctx=ctx)
  File "/opt/conda/envs/rapids/lib/python3.8/multiprocessing/synchronize.py", line 57, in __init__
    sl = self._semlock = _multiprocessing.SemLock(
OSError: [Errno 28] No space left on device

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/conda/envs/rapids/lib/python3.8/site-packages/distributed/deploy/spec.py", line 116, in _wrap_awaitable
    return await aw
  File "/opt/conda/envs/rapids/lib/python3.8/site-packages/distributed/core.py", line 536, in start
    raise RuntimeError(f"{type(self).__name__} failed to start.") from exc
RuntimeError: Nanny failed to start.
Task exception was never retrieved
future: <Task finished name='Task-67' coro=<_wrap_awaitable() done, defined at /opt/conda/envs/rapids/lib/python3.8/site-packages/distributed/deploy/spec.py:115> exception=RuntimeError('Nanny failed to start.')>
Traceback (most recent call last):
  File "/opt/conda/envs/rapids/lib/python3.8/site-packages/distributed/core.py", line 528, in start
    await asyncio.wait_for(self.start_unsafe(), timeout=timeout)
  File "/opt/conda/envs/rapids/lib/python3.8/asyncio/tasks.py", line 455, in wait_for
    return await fut
  File "/opt/conda/envs/rapids/lib/python3.8/site-packages/distributed/nanny.py", line 366, in start_unsafe
    response = await self.instantiate()
  File "/opt/conda/envs/rapids/lib/python3.8/site-packages/distributed/nanny.py", line 442, in instantiate
    result = await self.process.start()
  File "/opt/conda/envs/rapids/lib/python3.8/site-packages/distributed/nanny.py", line 678, in start
    self.init_result_q = init_q = get_mp_context().Queue()
  File "/opt/conda/envs/rapids/lib/python3.8/multiprocessing/context.py", line 103, in Queue
    return Queue(maxsize, ctx=self.get_context())
  File "/opt/conda/envs/rapids/lib/python3.8/multiprocessing/queues.py", line 42, in __init__
    self._rlock = ctx.Lock()
  File "/opt/conda/envs/rapids/lib/python3.8/multiprocessing/context.py", line 68, in Lock
    return Lock(ctx=self.get_context())
  File "/opt/conda/envs/rapids/lib/python3.8/multiprocessing/synchronize.py", line 162, in __init__
    SemLock.__init__(self, SEMAPHORE, 1, 1, ctx=ctx)
  File "/opt/conda/envs/rapids/lib/python3.8/multiprocessing/synchronize.py", line 57, in __init__
    sl = self._semlock = _multiprocessing.SemLock(
OSError: [Errno 28] No space left on device

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/conda/envs/rapids/lib/python3.8/site-packages/distributed/deploy/spec.py", line 116, in _wrap_awaitable
    return await aw
  File "/opt/conda/envs/rapids/lib/python3.8/site-packages/distributed/core.py", line 536, in start
    raise RuntimeError(f"{type(self).__name__} failed to start.") from exc
RuntimeError: Nanny failed to start.
---------------------------------------------------------------------------
OSError                                   Traceback (most recent call last)
/rapids/notebooks/cudf/Untitled.ipynb in <module>
      2 from dask_cuda import LocalCUDACluster
      3 
----> 4 cluster = LocalCUDACluster(threads_per_worker=1)
      5 client = Client(cluster)

/opt/conda/envs/rapids/lib/python3.8/site-packages/dask_cuda/local_cuda_cluster.py in __init__(self, CUDA_VISIBLE_DEVICES, n_workers, threads_per_worker, memory_limit, device_memory_limit, data, local_directory, shared_filesystem, protocol, enable_tcp_over_ucx, enable_infiniband, enable_nvlink, enable_rdmacm, rmm_pool_size, rmm_maximum_pool_size, rmm_managed_memory, rmm_async, rmm_log_directory, rmm_track_allocations, jit_unspill, log_spilling, worker_class, pre_import, **kwargs)
    363         self.cuda_visible_devices = CUDA_VISIBLE_DEVICES
    364         self.scale(n_workers)
--> 365         self.sync(self._correct_state)
    366 
    367     def new_worker_spec(self):

/opt/conda/envs/rapids/lib/python3.8/site-packages/distributed/utils.py in sync(self, func, asynchronous, callback_timeout, *args, **kwargs)
    336             return future
    337         else:
--> 338             return sync(
    339                 self.loop, func, *args, callback_timeout=callback_timeout, **kwargs
    340             )

/opt/conda/envs/rapids/lib/python3.8/site-packages/distributed/utils.py in sync(loop, func, callback_timeout, *args, **kwargs)
    403     if error:
    404         typ, exc, tb = error
--> 405         raise exc.with_traceback(tb)
    406     else:
    407         return result

/opt/conda/envs/rapids/lib/python3.8/site-packages/distributed/utils.py in f()
    376                 future = asyncio.wait_for(future, callback_timeout)
    377             future = asyncio.ensure_future(future)
--> 378             result = yield future
    379         except Exception:
    380             error = sys.exc_info()

/opt/conda/envs/rapids/lib/python3.8/site-packages/tornado/gen.py in run(self)
    767 
    768                     try:
--> 769                         value = future.result()
    770                     except Exception:
    771                         exc_info = sys.exc_info()

/opt/conda/envs/rapids/lib/python3.8/site-packages/distributed/deploy/spec.py in _correct_state_internal(self)
    380                 for w in workers:
    381                     w._cluster = weakref.ref(self)
--> 382                     await w  # for tornado gen.coroutine support
    383             self.workers.update(dict(zip(to_open, workers)))
    384 

/opt/conda/envs/rapids/lib/python3.8/site-packages/distributed/core.py in start(self)
    515             if self.status == Status.failed:
    516                 assert self.__startup_exc is not None
--> 517                 raise self.__startup_exc
    518             elif self.status != Status.init:
    519                 return self

/opt/conda/envs/rapids/lib/python3.8/site-packages/distributed/core.py in start(self)
    526 
    527             try:
--> 528                 await asyncio.wait_for(self.start_unsafe(), timeout=timeout)
    529             except asyncio.TimeoutError as exc:
    530                 await _close_on_failure(exc)

/opt/conda/envs/rapids/lib/python3.8/asyncio/tasks.py in wait_for(fut, timeout, loop)
    453 
    454     if timeout is None:
--> 455         return await fut
    456 
    457     if timeout <= 0:

/opt/conda/envs/rapids/lib/python3.8/site-packages/distributed/nanny.py in start_unsafe(self)
    364 
    365         logger.info("        Start Nanny at: %r", self.address)
--> 366         response = await self.instantiate()
    367 
    368         if response != Status.running:

/opt/conda/envs/rapids/lib/python3.8/site-packages/distributed/nanny.py in instantiate(self)
    440         else:
    441             try:
--> 442                 result = await self.process.start()
    443             except Exception:
    444                 logger.error("Failed to start process", exc_info=True)

/opt/conda/envs/rapids/lib/python3.8/site-packages/distributed/nanny.py in start(self)
    676             return self.status
    677 
--> 678         self.init_result_q = init_q = get_mp_context().Queue()
    679         self.child_stop_q = get_mp_context().Queue()
    680         uid = uuid.uuid4().hex

/opt/conda/envs/rapids/lib/python3.8/multiprocessing/context.py in Queue(self, maxsize)
    101         '''Returns a queue object'''
    102         from .queues import Queue
--> 103         return Queue(maxsize, ctx=self.get_context())
    104 
    105     def JoinableQueue(self, maxsize=0):

/opt/conda/envs/rapids/lib/python3.8/multiprocessing/queues.py in __init__(self, maxsize, ctx)
     40         self._maxsize = maxsize
     41         self._reader, self._writer = connection.Pipe(duplex=False)
---> 42         self._rlock = ctx.Lock()
     43         self._opid = os.getpid()
     44         if sys.platform == 'win32':

/opt/conda/envs/rapids/lib/python3.8/multiprocessing/context.py in Lock(self)
     66         '''Returns a non-recursive lock object'''
     67         from .synchronize import Lock
---> 68         return Lock(ctx=self.get_context())
     69 
     70     def RLock(self):

/opt/conda/envs/rapids/lib/python3.8/multiprocessing/synchronize.py in __init__(self, ctx)
    160 
    161     def __init__(self, *, ctx):
--> 162         SemLock.__init__(self, SEMAPHORE, 1, 1, ctx=ctx)
    163 
    164     def __repr__(self):

/opt/conda/envs/rapids/lib/python3.8/multiprocessing/synchronize.py in __init__(self, kind, value, maxvalue, ctx)
     55         for i in range(100):
     56             try:
---> 57                 sl = self._semlock = _multiprocessing.SemLock(
     58                     kind, value, maxvalue, self._make_name(),
     59                     unlink_now)

OSError: [Errno 28] No space left on device

nvidia-smi outputs:

Thu Feb  2 20:13:32 2023       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 520.61.05    Driver Version: 520.61.05    CUDA Version: 11.8     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA A100 80G...  On   | 00000000:01:00.0 Off |                    0 |
| N/A   44C    P0    67W / 300W |   2694MiB / 81920MiB |      0%      Default |
|                               |                      |             Disabled |
+-------------------------------+----------------------+----------------------+
|   1  NVIDIA A100 80G...  On   | 00000001:01:00.0 Off |                    0 |
| N/A   37C    P0    63W / 300W |    414MiB / 81920MiB |      0%      Default |
|                               |                      |             Disabled |
+-------------------------------+----------------------+----------------------+
|   2  NVIDIA A100 80G...  On   | 0000000C:01:00.0 Off |                    0 |
| N/A   36C    P0    61W / 300W |    414MiB / 81920MiB |      0%      Default |
|                               |                      |             Disabled |
+-------------------------------+----------------------+----------------------+
|   3  NVIDIA A100 80G...  On   | 0000000D:01:00.0 Off |                    0 |
| N/A   40C    P0    63W / 300W |    414MiB / 81920MiB |      0%      Default |
|                               |                      |             Disabled |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
+-----------------------------------------------------------------------------+

Expected behavior It creates the local cuda cluster Environment overview (please complete the following information)

Environment details Please run and paste the output of the cudf/print_env.sh script here, to gather any other relevant environment details

<details><summary>Click here to see environment details</summary><pre>

     **git***
     commit 48df508e3650c161310c5c7306e8e78e59ccb60c (grafted, HEAD -> branch-23.02, origin/branch-23.02, origin/HEAD)
     Author: AJ Schmidt <ajschmidt8@users.noreply.github.com>
     Date:   Tue Jan 17 12:37:09 2023 -0500

     Merge pull request #369 from bdice/update-cudf-notebooks

     Update cudf notebooks.
     **git submodules***
     +3f2e30669549e7d2cd6dad8f6a03a995315f584b clx (heads/branch-23.02)
     +3c39be5a9d7d69adaad47121359e0c084b76decf cudf (remotes/origin/branch-23.02)
     +64dabc361caa5088945fddc82c0c05ed72e5cb7c cugraph (heads/branch-23.02)
     +85b33dfd9d77d3ee38785bd3d5e1720c89fb5ebe cuml (heads/branch-23.02)
     +cbdfc6964597800080eae4a3c9139a2a535b22b0 cusignal (heads/branch-23.02)
     +e4e79a5606ccec06f2e001bde052e93257d13530 cuspatial (heads/branch-23.02)
     +0ffe93c62e0587b7553cb7dc079c19c275a3fee8 cuxfilter (heads/branch-23.02)
     +14cad33a6a9c8e2a89351f2ba880fbb69249c58d xgboost (heads/branch-23.02)

     ***OS Information***
     DISTRIB_ID=Ubuntu
     DISTRIB_RELEASE=22.04
     DISTRIB_CODENAME=jammy
     DISTRIB_DESCRIPTION="Ubuntu 22.04.1 LTS"
     PRETTY_NAME="Ubuntu 22.04.1 LTS"
     NAME="Ubuntu"
     VERSION_ID="22.04"
     VERSION="22.04.1 LTS (Jammy Jellyfish)"
     VERSION_CODENAME=jammy
     ID=ubuntu
     ID_LIKE=debian
     HOME_URL="https://www.ubuntu.com/"
     SUPPORT_URL="https://help.ubuntu.com/"
     BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
     PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
     UBUNTU_CODENAME=jammy
     Linux 848351ff0f5f 5.15.0-56-generic #62-Ubuntu SMP Tue Nov 22 19:56:13 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux

     ***GPU Information***
     Thu Feb  2 20:16:28 2023
     +-----------------------------------------------------------------------------+
     | NVIDIA-SMI 520.61.05    Driver Version: 520.61.05    CUDA Version: 11.8     |
     |-------------------------------+----------------------+----------------------+
     | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
     | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
     |                               |                      |               MIG M. |
     |===============================+======================+======================|
     |   0  NVIDIA A100 80G...  On   | 00000000:01:00.0 Off |                    0 |
     | N/A   43C    P0    67W / 300W |   2694MiB / 81920MiB |      0%      Default |
     |                               |                      |             Disabled |
     +-------------------------------+----------------------+----------------------+
     |   1  NVIDIA A100 80G...  On   | 00000001:01:00.0 Off |                    0 |
     | N/A   37C    P0    63W / 300W |    414MiB / 81920MiB |      0%      Default |
     |                               |                      |             Disabled |
     +-------------------------------+----------------------+----------------------+
     |   2  NVIDIA A100 80G...  On   | 0000000C:01:00.0 Off |                    0 |
     | N/A   36C    P0    61W / 300W |    414MiB / 81920MiB |      0%      Default |
     |                               |                      |             Disabled |
     +-------------------------------+----------------------+----------------------+
     |   3  NVIDIA A100 80G...  On   | 0000000D:01:00.0 Off |                    0 |
     | N/A   40C    P0    65W / 300W |    414MiB / 81920MiB |      0%      Default |
     |                               |                      |             Disabled |
     +-------------------------------+----------------------+----------------------+

     +-----------------------------------------------------------------------------+
     | Processes:                                                                  |
     |  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
     |        ID   ID                                                   Usage      |
     |=============================================================================|
     +-----------------------------------------------------------------------------+

     ***CPU***
     Architecture:                    aarch64
     CPU op-mode(s):                  32-bit, 64-bit
     Byte Order:                      Little Endian
     CPU(s):                          80
     On-line CPU(s) list:             0-79
     Vendor ID:                       ARM
     Model name:                      Neoverse-N1
     Model:                           1
     Thread(s) per core:              1
     Core(s) per cluster:             80
     Socket(s):                       -
     Cluster(s):                      1
     Stepping:                        r3p1
     Frequency boost:                 disabled
     CPU max MHz:                     3000.0000
     CPU min MHz:                     1000.0000
     BogoMIPS:                        50.00
     Flags:                           fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm lrcpc dcpop asimddp ssbs
     L1d cache:                       5 MiB (80 instances)
     L1i cache:                       5 MiB (80 instances)
     L2 cache:                        80 MiB (80 instances)
     NUMA node(s):                    1
     NUMA node0 CPU(s):               0-79
     Vulnerability Itlb multihit:     Not affected
     Vulnerability L1tf:              Not affected
     Vulnerability Mds:               Not affected
     Vulnerability Meltdown:          Not affected
     Vulnerability Mmio stale data:   Not affected
     Vulnerability Retbleed:          Not affected
     Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
     Vulnerability Spectre v1:        Mitigation; __user pointer sanitization
     Vulnerability Spectre v2:        Mitigation; CSV2, BHB
     Vulnerability Srbds:             Not affected
     Vulnerability Tsx async abort:   Not affected

     ***CMake***

     ***g++***

     ***nvcc***

     ***Python***
     /opt/conda/envs/rapids/bin/python
     Python 3.8.15

     ***Environment Variables***
     PATH                            : /opt/conda/envs/rapids/bin:/opt/conda/condabin:/opt/conda/condabin:/opt/conda/envs/rapids/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
     LD_LIBRARY_PATH                 : /usr/local/nvidia/lib:/usr/local/nvidia/lib64
     NUMBAPRO_NVVM                   :
     NUMBAPRO_LIBDEVICE              :
     CONDA_PREFIX                    : /opt/conda/envs/rapids
     PYTHON_PATH                     :

     ***conda packages***
     /opt/conda/condabin/conda
     # packages in environment at /opt/conda/envs/rapids:
     #
     # Name                    Version                   Build  Channel
     _openmp_mutex             4.5                       2_gnu    conda-forge
     aiobotocore               2.4.2              pyhd8ed1ab_0    conda-forge
     aiohttp                   3.8.3            py38hdacc76a_1    conda-forge
     aioitertools              0.11.0             pyhd8ed1ab_0    conda-forge
     aiosignal                 1.3.1              pyhd8ed1ab_0    conda-forge
     anyio                     3.6.2              pyhd8ed1ab_0    conda-forge
     aom                       3.5.0                headf329_0    conda-forge
     appdirs                   1.4.4              pyh9f0ad1d_0    conda-forge
     argon2-cffi               21.3.0             pyhd8ed1ab_0    conda-forge
     argon2-cffi-bindings      21.2.0           py38hdacc76a_3    conda-forge
     arrow-cpp                 10.0.1           hd7597ea_4_cpu    conda-forge
     async-timeout             4.0.2              pyhd8ed1ab_0    conda-forge
     attrs                     22.2.0             pyh71513ae_0    conda-forge
     aws-c-auth                0.6.21               ha31d3f7_1    conda-forge
     aws-c-cal                 0.5.20               hf7c3097_3    conda-forge
     aws-c-common              0.8.5                h4e544f5_0    conda-forge
     aws-c-compression         0.2.16               h831b76e_0    conda-forge
     aws-c-event-stream        0.2.16               h57a1bea_0    conda-forge
     aws-c-http                0.6.29               h096e8e9_0    conda-forge
     aws-c-io                  0.13.11              h95eb387_2    conda-forge
     aws-c-mqtt                0.7.13              h4615787_10    conda-forge
     aws-c-s3                  0.2.1                h436748e_2    conda-forge
     aws-c-sdkutils            0.1.7                h831b76e_0    conda-forge
     aws-checksums             0.1.14               h780bfcc_0    conda-forge
     aws-crt-cpp               0.18.16              ha184985_5    conda-forge
     aws-sdk-cpp               1.9.379              hd7da748_6    conda-forge
     babel                     2.11.0             pyhd8ed1ab_0    conda-forge
     backcall                  0.2.0              pyh9f0ad1d_0    conda-forge
     backports                 1.0                pyhd8ed1ab_3    conda-forge
     backports.functools_lru_cache 1.6.4              pyhd8ed1ab_0    conda-forge
     beautifulsoup4            4.11.1             pyha770c72_0    conda-forge
     blas                      2.104                    netlib    conda-forge
     bleach                    6.0.0              pyhd8ed1ab_0    conda-forge
     blosc                     1.21.3               h9515e8b_0    conda-forge
     bokeh                     2.4.3            py38he3eb160_0    conda-forge
     boost-cpp                 1.78.0               ha1c1135_1    conda-forge
     botocore                  1.27.59            pyhd8ed1ab_0    conda-forge
     branca                    0.6.0              pyhd8ed1ab_0    conda-forge
     brotli                    1.0.9                h4e544f5_8    conda-forge
     brotli-bin                1.0.9                h4e544f5_8    conda-forge
     brotlipy                  0.7.0           py38hdacc76a_1005    conda-forge
     brunsli                   0.1                  h01db608_0    conda-forge
     bzip2                     1.0.8                hf897c2e_4    conda-forge
     c-ares                    1.18.1               hf897c2e_0    conda-forge
     c-blosc2                  2.6.1                h9a29ef2_0    conda-forge
     ca-certificates           2022.12.7            h4fd8a4c_0    conda-forge
     cachetools                5.3.0              pyhd8ed1ab_0    conda-forge
     cairo                     1.16.0            hd19fb6e_1014    conda-forge
     certifi                   2022.12.7          pyhd8ed1ab_0    conda-forge
     cffi                      1.15.1           py38h6989fa8_3    conda-forge
     cfitsio                   4.2.0                h8b262d6_0    conda-forge
     charls                    2.3.4                h01db608_0    conda-forge
     charset-normalizer        2.1.1              pyhd8ed1ab_0    conda-forge
     click                     8.1.3            py38he3eb160_1    conda-forge
     click-plugins             1.1.1                      py_0    conda-forge
     cligj                     0.7.2              pyhd8ed1ab_1    conda-forge
     cloudpickle               2.2.1              pyhd8ed1ab_0    conda-forge
     colorama                  0.4.6              pyhd8ed1ab_0    conda-forge
     colorcet                  3.0.1              pyhd8ed1ab_0    conda-forge
     comm                      0.1.2              pyhd8ed1ab_0    conda-forge
     contourpy                 1.0.7            py38h721baa8_0    conda-forge
     cryptography              39.0.0           py38hd0a2be6_0    conda-forge
     cubinlinker               0.2.2            py38h1053575_0    rapidsai
     cucim                     23.02.00a230202 cuda_11_py38_g5b59e73_48    rapidsai-nightly
     cuda-profiler-api         11.8.86                       0    nvidia
     cuda-python               11.8.1           py38h5509dca_2    conda-forge
     cudatoolkit               11.8.0              h46baab3_11    conda-forge
     cudf                      23.02.00a230201 cuda_11_py38_g3c39be5a9d_309    rapidsai-nightly
     cudf_kafka                23.02.00a230201 py38_g3c39be5a9d_309    rapidsai-nightly
     cugraph                   23.02.00a230201 cuda11_py38_g64dabc36_149    rapidsai-nightly
     cuml                      23.02.00a230202 cuda11_py38_g85b33dfd9_106    rapidsai-nightly
     cupy                      11.5.0           py38ha9de562_0    conda-forge
     curl                      7.86.0               h22f3f83_1    conda-forge
     cusignal                  23.02.00a230201 py39_gcbdfc69_23    rapidsai-nightly
     cuspatial                 23.02.00a230201 py38_ge4e79a5_58    rapidsai-nightly
     custreamz                 23.02.00a230201 py38_g3c39be5a9d_309    rapidsai-nightly
     cuxfilter                 23.02.00a230201 py38_g0ffe93c_23    rapidsai-nightly
     cycler                    0.11.0             pyhd8ed1ab_0    conda-forge
     cyrus-sasl                2.1.27               h559b310_5    conda-forge
     cython                    0.29.33          py38h31bd7b8_0    conda-forge
     cytoolz                   0.12.0           py38h81aae68_1    conda-forge
     dask                      2023.1.1a230127 py_g1c6fb849_13    dask/label/dev
     dask-core                 2023.1.1a230127 py_g185eed2c9_23    dask/label/dev
     dask-cuda                 23.02.00a230201 py38_g84f4aa2_47    rapidsai-nightly
     dask-cudf                 23.02.00a230201 cuda_11_py38_g3c39be5a9d_309    rapidsai-nightly
     dask-glm                  0.2.0                      py_1    conda-forge
     dask-labextension         6.0.0              pyhd8ed1ab_0    conda-forge
     dask-ml                   2022.5.27          pyhd8ed1ab_0    conda-forge
     datashader                0.13.1a                    py_0    rapidsai
     datashape                 0.5.4                      py_1    conda-forge
     dav1d                     1.0.0                h4e544f5_1    conda-forge
     debugpy                   1.6.6            py38h31bd7b8_0    conda-forge
     decorator                 5.1.1              pyhd8ed1ab_0    conda-forge
     defusedxml                0.7.1              pyhd8ed1ab_0    conda-forge
     distributed               2023.1.1a230127 py_g1c6fb849_13    dask/label/dev
     dlpack                    0.5                  h01db608_0    conda-forge
     entrypoints               0.4                pyhd8ed1ab_0    conda-forge
     exceptiongroup            1.1.0              pyhd8ed1ab_0    conda-forge
     expat                     2.5.0                ha18d298_0    conda-forge
     faiss-proc                1.0.0                      cuda    rapidsai
     fastavro                  1.7.1            py38hd5eba46_0    conda-forge
     fastrlock                 0.8              py38h7cb29f2_3    conda-forge
     filterpy                  1.4.5                      py_1    conda-forge
     fiona                     1.8.22           py38h3d11595_2    conda-forge
     flit-core                 3.8.0              pyhd8ed1ab_0    conda-forge
     folium                    0.14.0             pyhd8ed1ab_0    conda-forge
     font-ttf-dejavu-sans-mono 2.37                 hab24e00_0    conda-forge
     font-ttf-inconsolata      3.000                h77eed37_0    conda-forge
     font-ttf-source-code-pro  2.038                h77eed37_0    conda-forge
     font-ttf-ubuntu           0.83                 hab24e00_0    conda-forge
     fontconfig                2.14.2               ha9a116f_0    conda-forge
     fonts-conda-ecosystem     1                             0    conda-forge
     fonts-conda-forge         1                             0    conda-forge
     fonttools                 4.38.0           py38hdacc76a_1    conda-forge
     freetype                  2.12.1               hbbbf32d_1    conda-forge
     freexl                    1.0.6                h4e544f5_1    conda-forge
     frozenlist                1.3.3            py38hdacc76a_0    conda-forge
     fsspec                    2023.1.0           pyhd8ed1ab_0    conda-forge
     gdal                      3.5.3           py38he03ebc6_10    conda-forge
     gdk-pixbuf                2.42.8               hb98dc3c_0    conda-forge
     geopandas                 0.12.2             pyhd8ed1ab_0    conda-forge
     geopandas-base            0.12.2             pyha770c72_0    conda-forge
     geos                      3.11.1               h4de3ea5_0    conda-forge
     geotiff                   1.7.1                hb764940_4    conda-forge
     gettext                   0.21.1               ha18d298_0    conda-forge
     gflags                    2.2.2             h54f1f3f_1004    conda-forge
     giflib                    5.2.1                hb9de7d4_2    conda-forge
     git                       2.39.1          pl5321hf85ec50_0    conda-forge
     glib                      2.74.1               h7866ba4_1    conda-forge
     glib-tools                2.74.1               h7866ba4_1    conda-forge
     glog                      0.6.0                h8ab10f1_0    conda-forge
     hdf4                      4.2.15               hb6731fe_5    conda-forge
     hdf5                      1.12.2          nompi_h3900512_100    conda-forge
     heapdict                  1.0.1                      py_0    conda-forge
     holoviews                 1.15.3             pyhd8ed1ab_0    conda-forge
     icu                       70.1                 ha18d298_0    conda-forge
     idna                      3.4                pyhd8ed1ab_0    conda-forge
     imagecodecs               2022.9.26        py38h85632d8_4    conda-forge
     imageio                   2.25.0             pyh24c5eb1_0    conda-forge
     importlib-metadata        6.0.0              pyha770c72_0    conda-forge
     importlib_metadata        6.0.0                hd8ed1ab_0    conda-forge
     importlib_resources       5.10.2             pyhd8ed1ab_0    conda-forge
     iniconfig                 2.0.0              pyhd8ed1ab_0    conda-forge
     ipykernel                 6.21.0             pyh210e3f2_0    conda-forge
     ipython                   7.31.1           py38h2063c64_0    conda-forge
     ipython_genutils          0.2.0                      py_1    conda-forge
     ipywidgets                8.0.4              pyhd8ed1ab_0    conda-forge
     jbig                      2.1               hf897c2e_2003    conda-forge
     jedi                      0.18.2             pyhd8ed1ab_0    conda-forge
     jinja2                    3.1.2              pyhd8ed1ab_1    conda-forge
     jmespath                  1.0.1              pyhd8ed1ab_0    conda-forge
     joblib                    1.2.0              pyhd8ed1ab_0    conda-forge
     jpeg                      9e                   h9cdd2b7_2    conda-forge
     json-c                    0.16                 hcc5b78b_0    conda-forge
     json5                     0.9.5              pyh9f0ad1d_0    conda-forge
     jsonschema                4.17.3             pyhd8ed1ab_0    conda-forge
     jupyter-packaging         0.7.12             pyhd8ed1ab_0    conda-forge
     jupyter-server-proxy      3.2.2              pyhd8ed1ab_0    conda-forge
     jupyter_client            8.0.2              pyhd8ed1ab_0    conda-forge
     jupyter_core              5.2.0            py38he3eb160_0    conda-forge
     jupyter_events            0.6.3              pyhd8ed1ab_0    conda-forge
     jupyter_server            2.1.0              pyhd8ed1ab_0    conda-forge
     jupyter_server_terminals  0.4.4              pyhd8ed1ab_1    conda-forge
     jupyterlab                3.5.3              pyhd8ed1ab_0    conda-forge
     jupyterlab-favorites      3.1.1              pyhd8ed1ab_0    conda-forge
     jupyterlab-nvdashboard    0.7.0                      py_0    rapidsai
     jupyterlab_pygments       0.2.2              pyhd8ed1ab_0    conda-forge
     jupyterlab_server         2.19.0             pyhd8ed1ab_0    conda-forge
     jupyterlab_widgets        3.0.5              pyhd8ed1ab_0    conda-forge
     jxrlib                    1.1                  hf897c2e_2    conda-forge
     kealib                    1.5.0                hf95f5c2_0    conda-forge
     keyutils                  1.6.1                h4e544f5_0    conda-forge
     kiwisolver                1.4.4            py38hbcd4e1f_1    conda-forge
     krb5                      1.19.3               h750e270_0    conda-forge
     lcms2                     2.14                 h5246980_0    conda-forge
     ld_impl_linux-aarch64     2.40                 h2d8c526_0    conda-forge
     lerc                      4.0.0                h4de3ea5_0    conda-forge
     libabseil                 20220623.0      cxx17_hf39f241_6    conda-forge
     libaec                    1.0.6                hd600fc2_1    conda-forge
     libarrow                  10.0.1           hbf62c9e_4_cpu    conda-forge
     libavif                   0.11.1               h2429a20_0    conda-forge
     libblas                   3.9.0           4_h6783c1e_netlib    conda-forge
     libbrotlicommon           1.0.9                h4e544f5_8    conda-forge
     libbrotlidec              1.0.9                h4e544f5_8    conda-forge
     libbrotlienc              1.0.9                h4e544f5_8    conda-forge
     libcblas                  3.9.0           4_h6783c1e_netlib    conda-forge
     libcrc32c                 1.1.2                h01db608_0    conda-forge
     libcublas                 11.11.3.6                     0    nvidia
     libcublas-dev             11.11.3.6                     0    nvidia
     libcucim                  23.02.00a230202 cuda11_g4b19225_49    rapidsai-nightly
     libcudf                   23.02.00a230201 cuda11_g3c39be5a9d_309    rapidsai-nightly
     libcudf_kafka             23.02.00a230201 g3c39be5a9d_309    rapidsai-nightly
     libcufft                  10.9.0.58                     0    nvidia
     libcugraph                23.02.00a230201 cuda11_g64dabc36_149    rapidsai-nightly
     libcugraph_etl            23.02.00a230201 cuda11_g64dabc36_149    rapidsai-nightly
     libcugraphops             23.02.00a230201 cuda11_gdc0b7f79_65    rapidsai-nightly
     libcuml                   23.02.00a230202 cuda11_g85b33dfd9_106    rapidsai-nightly
     libcumlprims              23.02.00a230126 cuda11_ge28945f_11    rapidsai-nightly
     libcurand                 10.3.0.86                     0    nvidia
     libcurand-dev             10.3.0.86                     0    nvidia
     libcurl                   7.86.0               h22f3f83_1    conda-forge
     libcusolver               11.4.1.48                     0    nvidia
     libcusolver-dev           11.4.1.48                     0    nvidia
     libcusparse               11.7.5.86                     0    nvidia
     libcusparse-dev           11.7.5.86                     0    nvidia
     libcuspatial              23.02.00a230201 cuda11_ge4e79a5_58    rapidsai-nightly
     libdap4                   3.20.6               hf7f982d_2    conda-forge
     libdeflate                1.14                 h4e544f5_0    conda-forge
     libedit                   3.1.20191231         he28a2e2_2    conda-forge
     libev                     4.33                 h516909a_1    conda-forge
     libevent                  2.1.10               h4f30969_4    conda-forge
     libfaiss                  1.7.2           cuda118h1234567_4_cuda    rapidsai
     libffi                    3.4.2                h3557bc0_5    conda-forge
     libgcc-ng                 12.2.0              h607ecd0_19    conda-forge
     libgcrypt                 1.10.1               h4e544f5_0    conda-forge
     libgdal                   3.5.3               hbd8f71b_10    conda-forge
     libgfortran-ng            12.2.0              he9431aa_19    conda-forge
     libgfortran5              12.2.0              hf695500_19    conda-forge
     libglib                   2.74.1               h01e6fbd_1    conda-forge
     libgomp                   12.2.0              h607ecd0_19    conda-forge
     libgoogle-cloud           2.5.0                h101168d_1    conda-forge
     libgpg-error              1.46                 haae8ae4_0    conda-forge
     libgrpc                   1.51.1               h2efb554_0    conda-forge
     libgsasl                  1.10.0               h06a33e9_0    conda-forge
     libhwloc                  2.8.0                hc0facba_1    conda-forge
     libiconv                  1.17                 h9cdd2b7_0    conda-forge
     libjpeg-turbo             2.1.4                h4e544f5_0    conda-forge
     libkml                    1.3.0             h068af62_1015    conda-forge
     liblapack                 3.9.0           4_h6783c1e_netlib    conda-forge
     liblapacke                3.9.0           4_h6783c1e_netlib    conda-forge
     libllvm11                 11.1.0               hb2805f8_5    conda-forge
     libnetcdf                 4.8.1           nompi_h88e613c_106    conda-forge
     libnghttp2                1.51.0               h674c3cc_0    conda-forge
     libnsl                    2.0.0                hf897c2e_0    conda-forge
     libntlm                   1.4               hf897c2e_1002    conda-forge
     libpng                    1.6.39               hf9034f9_0    conda-forge
     libpq                     15.1                 h68e116c_1    conda-forge
     libprotobuf               3.21.12              h7fe2111_0    conda-forge
     libraft-distance          23.02.00a230201 cuda11_g6a7e125b_123    rapidsai-nightly
     libraft-headers           23.02.00a230201 cuda11_g6a7e125b_123    rapidsai-nightly
     libraft-nn                23.02.00a230201 cuda11_g6a7e125b_123    rapidsai-nightly
     librdkafka                1.7.0                hf9019d4_1    conda-forge
     librmm                    23.02.00a230201 cuda11_g4e030a05_30    rapidsai-nightly
     librttopo                 1.1.0               h80ae33b_12    conda-forge
     libsodium                 1.0.18               hb9de7d4_1    conda-forge
     libspatialindex           1.9.3                h01db608_4    conda-forge
     libspatialite             5.0.1               h873af1b_22    conda-forge
     libsqlite                 3.40.0               hf9034f9_0    conda-forge
     libssh2                   1.10.0               he5a64b1_3    conda-forge
     libstdcxx-ng              12.2.0              hc13a102_19    conda-forge
     libthrift                 0.16.0               h6e53a4e_2    conda-forge
     libtiff                   4.4.0                hfcd36d1_5    conda-forge
     libtmglib                 3.9.0           4_h6783c1e_netlib    conda-forge
     libutf8proc               2.8.0                h4e544f5_0    conda-forge
     libuuid                   2.32.1            hf897c2e_1000    conda-forge
     libuv                     1.44.2               h4e544f5_0    conda-forge
     libwebp                   1.2.4                hef792ef_0    conda-forge
     libwebp-base              1.2.4                h4e544f5_0    conda-forge
     libxcb                    1.13              h3557bc0_1004    conda-forge
     libxgboost                1.7.1dev.rapidsai23.02       cuda_11_0    rapidsai
     libxml2                   2.10.3               h249b6dd_0    conda-forge
     libzip                    1.9.2                heb2ba21_1    conda-forge
     libzlib                   1.2.13               h4e544f5_4    conda-forge
     libzopfli                 1.0.3                h01db608_0    conda-forge
     llvmlite                  0.39.1           py38hd14ccfc_1    conda-forge
     locket                    1.0.0              pyhd8ed1ab_0    conda-forge
     lz4                       4.2.0            py38h4ab07a3_0    conda-forge
     lz4-c                     1.9.3                h01db608_1    conda-forge
     mapclassify               2.5.0              pyhd8ed1ab_1    conda-forge
     markdown                  3.4.1              pyhd8ed1ab_0    conda-forge
     markupsafe                2.1.2            py38hd5eba46_0    conda-forge
     matplotlib-base           3.6.3            py38h52fa05b_0    conda-forge
     matplotlib-inline         0.1.6              pyhd8ed1ab_0    conda-forge
     mistune                   2.0.4              pyhd8ed1ab_0    conda-forge
     msgpack-python            1.0.4            py38hbcd4e1f_1    conda-forge
     multidict                 6.0.4            py38hda48048_0    conda-forge
     multipledispatch          0.6.0                      py_0    conda-forge
     munch                     2.5.0                      py_0    conda-forge
     munkres                   1.1.4              pyh9f0ad1d_0    conda-forge
     nbclassic                 0.4.8              pyhd8ed1ab_0    conda-forge
     nbclient                  0.7.2              pyhd8ed1ab_0    conda-forge
     nbconvert                 7.2.9              pyhd8ed1ab_0    conda-forge
     nbconvert-core            7.2.9              pyhd8ed1ab_0    conda-forge
     nbconvert-pandoc          7.2.9              pyhd8ed1ab_0    conda-forge
     nbformat                  5.7.3              pyhd8ed1ab_0    conda-forge
     nccl                      2.14.3.1             h436c9ae_0    conda-forge
     ncurses                   6.3                  headf329_1    conda-forge
     nest-asyncio              1.5.6              pyhd8ed1ab_0    conda-forge
     networkx                  3.0                pyhd8ed1ab_0    conda-forge
     nodejs                    18.12.1              h928fb59_0    conda-forge
     notebook                  6.5.2              pyha770c72_1    conda-forge
     notebook-shim             0.2.2              pyhd8ed1ab_0    conda-forge
     nspr                      4.35                 h4de3ea5_0    conda-forge
     nss                       3.82                 h4b7554b_0    conda-forge
     numba                     0.56.4           py38h4c0ec4d_0    conda-forge
     numpy                     1.23.5           py38h0fc68d9_0    conda-forge
     nvtx                      0.2.3            py38h81aae68_2    conda-forge
     openjpeg                  2.5.0                h9b6de37_1    conda-forge
     openslide                 3.4.1                h4d8a784_5    conda-forge
     openssl                   3.0.7                hb4cce97_2    conda-forge
     orc                       1.8.2                h3aebf57_0    conda-forge
     packaging                 23.0               pyhd8ed1ab_0    conda-forge
     pandas                    1.5.3            py38h19f5adf_0    conda-forge
     pandoc                    2.19.2               h8af1aa0_1    conda-forge
     pandocfilters             1.5.0              pyhd8ed1ab_0    conda-forge
     panel                     0.14.1             pyhd8ed1ab_0    conda-forge
     param                     1.12.3             pyh1a96a4e_0    conda-forge
     parquet-cpp               1.5.1                         2    conda-forge
     parso                     0.8.3              pyhd8ed1ab_0    conda-forge
     partd                     1.3.0              pyhd8ed1ab_0    conda-forge
     patsy                     0.5.3              pyhd8ed1ab_0    conda-forge
     pcre2                     10.40                he7b27c6_0    conda-forge
     perl                      5.32.1          2_hf897c2e_perl5    conda-forge
     pexpect                   4.8.0            py38h32f6830_1    conda-forge
     pickleshare               0.7.5           py38h32f6830_1002    conda-forge
     pillow                    9.2.0            py38h29cb35a_3    conda-forge
     pip                       23.0               pyhd8ed1ab_0    conda-forge
     pixman                    0.40.0               hb9de7d4_0    conda-forge
     pkgutil-resolve-name      1.3.10             pyhd8ed1ab_0    conda-forge
     platformdirs              2.6.2              pyhd8ed1ab_0    conda-forge
     pluggy                    1.0.0            py38h2063c64_4    conda-forge
     pooch                     1.6.0              pyhd8ed1ab_0    conda-forge
     poppler                   22.12.0              h054b39a_0    conda-forge
     poppler-data              0.4.11               hd8ed1ab_0    conda-forge
     postgresql                15.1                 h8376ac0_1    conda-forge
     proj                      9.1.0                h92d5b9d_0    conda-forge
     prometheus_client         0.16.0             pyhd8ed1ab_0    conda-forge
     prompt-toolkit            3.0.36             pyha770c72_0    conda-forge
     protobuf                  4.21.12          py38h31bd7b8_0    conda-forge
     psutil                    5.9.4            py38hdacc76a_0    conda-forge
     pthread-stubs             0.4               hb9de7d4_1001    conda-forge
     ptxcompiler               0.7.0            py38h5509dca_3    conda-forge
     ptyprocess                0.7.0              pyhd3deb0d_0    conda-forge
     py-xgboost                1.7.1dev.rapidsai23.02  cuda_11_py38_0    rapidsai
     pyarrow                   10.0.1          py38hca9a70a_4_cpu    conda-forge
     pycparser                 2.21               pyhd8ed1ab_0    conda-forge
     pyct                      0.4.6                      py_0    conda-forge
     pyct-core                 0.4.6                      py_0    conda-forge
     pydeck                    0.5.0              pyh9f0ad1d_0    conda-forge
     pyee                      8.1.0              pyhd8ed1ab_0    conda-forge
     pygments                  2.14.0             pyhd8ed1ab_0    conda-forge
     pylibcugraph              23.02.00a230201 cuda11_py38_g64dabc36_149    rapidsai-nightly
     pylibraft                 23.02.00a230201 cuda11_py38_g6a7e125b_123    rapidsai-nightly
     pynndescent               0.5.8              pyh1a96a4e_0    conda-forge
     pynvml                    11.4.1             pyhd8ed1ab_0    conda-forge
     pyopenssl                 23.0.0             pyhd8ed1ab_0    conda-forge
     pyparsing                 3.0.9              pyhd8ed1ab_0    conda-forge
     pyppeteer                 1.0.2              pyhd8ed1ab_0    conda-forge
     pyproj                    3.4.0            py38h8a7638b_2    conda-forge
     pyrsistent                0.19.3           py38hd5eba46_0    conda-forge
     pysocks                   1.7.1            py38h2063c64_5    conda-forge
     pytest                    7.2.1              pyhd8ed1ab_0    conda-forge
     python                    3.8.15          ha43d526_1_cpython    conda-forge
     python-confluent-kafka    1.7.0            py38he362374_2    conda-forge
     python-dateutil           2.8.2              pyhd8ed1ab_0    conda-forge
     python-fastjsonschema     2.16.2             pyhd8ed1ab_0    conda-forge
     python-json-logger        2.0.4              pyhd8ed1ab_0    conda-forge
     python_abi                3.8                      3_cp38    conda-forge
     pytz                      2022.7.1           pyhd8ed1ab_0    conda-forge
     pyviz_comms               2.2.1              pyhd8ed1ab_1    conda-forge
     pywavelets                1.4.1            py38h50faf1d_0    conda-forge
     pyyaml                    6.0              py38hdacc76a_5    conda-forge
     pyzmq                     25.0.0           py38ha599cda_0    conda-forge
     raft-dask                 23.02.00a230201 cuda11_py38_g6a7e125b_123    rapidsai-nightly
     rapids                    23.02.00a230202 cuda11_py38_g2aee638_88    rapidsai-nightly
     rapids-xgboost            23.02.00a230202 cuda11_py38_g2aee638_88    rapidsai-nightly
     re2                       2022.06.01           h4de3ea5_1    conda-forge
     readline                  8.1.2                h38e3740_0    conda-forge
     requests                  2.28.2             pyhd8ed1ab_0    conda-forge
     rfc3339-validator         0.1.4              pyhd8ed1ab_0    conda-forge
     rfc3986-validator         0.1.1              pyh9f0ad1d_0    conda-forge
     rmm                       23.02.00a230201 cuda11_py38_g4e030a05_30    rapidsai-nightly
     rtree                     1.0.1            py38hc089ca2_1    conda-forge
     s2n                       1.3.30               hf55b2d5_0    conda-forge
     s3fs                      2023.1.0           pyhd8ed1ab_0    conda-forge
     scikit-image              0.19.3           py38hfb864cf_2    conda-forge
     scikit-learn              1.2.1            py38h7ceec22_0    conda-forge
     scipy                     1.10.0           py38h39f6eef_0    conda-forge
     seaborn                   0.12.2               hd8ed1ab_0    conda-forge
     seaborn-base              0.12.2             pyhd8ed1ab_0    conda-forge
     send2trash                1.8.0              pyhd8ed1ab_0    conda-forge
     setuptools                66.1.1             pyhd8ed1ab_0    conda-forge
     shapely                   2.0.1            py38ha1dc50b_0    conda-forge
     simpervisor               0.4                pyhd8ed1ab_0    conda-forge
     six                       1.16.0             pyh6c4a22f_0    conda-forge
     snappy                    1.1.9                hc7e91e1_2    conda-forge
     sniffio                   1.3.0              pyhd8ed1ab_0    conda-forge
     sortedcontainers          2.4.0              pyhd8ed1ab_0    conda-forge
     soupsieve                 2.3.2.post1        pyhd8ed1ab_0    conda-forge
     spdlog                    1.8.5                hd62202e_1    conda-forge
     sqlite                    3.40.0               h69ca7e5_0    conda-forge
     statsmodels               0.13.5           py38h9dfdfaa_2    conda-forge
     streamz                   0.6.4              pyh6c4a22f_0    conda-forge
     tbb                       2021.7.0             hdd96247_1    conda-forge
     tblib                     1.7.0              pyhd8ed1ab_0    conda-forge
     terminado                 0.17.1             pyh41d4057_0    conda-forge
     threadpoolctl             3.1.0              pyh8a188c0_0    conda-forge
     tifffile                  2022.10.10         pyhd8ed1ab_0    conda-forge
     tiledb                    2.13.2               he69c9f4_0    conda-forge
     tinycss2                  1.2.1              pyhd8ed1ab_0    conda-forge
     tk                        8.6.12               hd8af866_0    conda-forge
     tomli                     2.0.1              pyhd8ed1ab_0    conda-forge
     toolz                     0.12.0             pyhd8ed1ab_0    conda-forge
     tornado                   6.2              py38h81aae68_1    conda-forge
     tqdm                      4.64.1             pyhd8ed1ab_0    conda-forge
     traitlets                 5.9.0              pyhd8ed1ab_0    conda-forge
     treelite                  3.1.0            py38h8832b3c_0    conda-forge
     treelite-runtime          3.1.0                    pypi_0    pypi
     typing-extensions         4.4.0                hd8ed1ab_0    conda-forge
     typing_extensions         4.4.0              pyha770c72_0    conda-forge
     tzcode                    2022g                h4e544f5_0    conda-forge
     tzdata                    2022g                h191b570_0    conda-forge
     ucx                       1.13.1               h8ea8e7f_1    conda-forge
     ucx-proc                  1.0.0                       gpu    rapidsai
     ucx-py                    0.30.00a230201  py38_ga21f62a_17    rapidsai-nightly
     umap-learn                0.5.3            py38he3eb160_0    conda-forge
     unicodedata2              15.0.0           py38hdacc76a_0    conda-forge
     urllib3                   1.26.14            pyhd8ed1ab_0    conda-forge
     wcwidth                   0.2.6              pyhd8ed1ab_0    conda-forge
     webencodings              0.5.1                      py_1    conda-forge
     websocket-client          1.5.0              pyhd8ed1ab_0    conda-forge
     websockets                10.4             py38hdacc76a_1    conda-forge
     wheel                     0.38.4             pyhd8ed1ab_0    conda-forge
     widgetsnbextension        4.0.5              pyhd8ed1ab_0    conda-forge
     wrapt                     1.14.1           py38hdacc76a_1    conda-forge
     xarray                    2023.1.0           pyhd8ed1ab_0    conda-forge
     xerces-c                  3.2.4                hb872fcc_1    conda-forge
     xgboost                   1.7.1dev.rapidsai23.02  cuda_11_py38_0    rapidsai
     xorg-kbproto              1.0.7             h3557bc0_1002    conda-forge
     xorg-libice               1.0.10               h3557bc0_0    conda-forge
     xorg-libsm                1.2.3             h965e137_1000    conda-forge
     xorg-libx11               1.7.2                h3557bc0_0    conda-forge
     xorg-libxau               1.0.9                h3557bc0_0    conda-forge
     xorg-libxdmcp             1.1.3                h3557bc0_0    conda-forge
     xorg-libxext              1.3.4                h3557bc0_1    conda-forge
     xorg-libxrender           0.9.10            h3557bc0_1003    conda-forge
     xorg-renderproto          0.11.1            h3557bc0_1002    conda-forge
     xorg-xextproto            7.3.0             h3557bc0_1002    conda-forge
     xorg-xproto               7.0.31            h3557bc0_1007    conda-forge
     xyzservices               2022.9.0           pyhd8ed1ab_0    conda-forge
     xz                        5.2.6                h9cdd2b7_0    conda-forge
     yaml                      0.2.5                hf897c2e_2    conda-forge
     yarl                      1.8.2            py38hdacc76a_0    conda-forge
     zeromq                    4.3.4                h01db608_1    conda-forge
     zfp                       1.0.0                h4de3ea5_3    conda-forge
     zict                      2.2.0              pyhd8ed1ab_0    conda-forge
     zipp                      3.12.0             pyhd8ed1ab_0    conda-forge
     zlib                      1.2.13               h4e544f5_4    conda-forge
     zlib-ng                   2.0.6                h4e544f5_0    conda-forge
     zstd                      1.5.2                h44f6412_6    conda-forge

</pre></details>

Additional context Freezes notebooks testing

bdice commented 1 year ago

Closing in favor of dask-cuda issue: https://github.com/rapidsai/dask-cuda/issues/1104