rapidsai / cudf

cuDF - GPU DataFrame Library
https://docs.rapids.ai/api/cudf/stable/
Apache License 2.0
8.05k stars 873 forks source link

[BUG] Unexpected OOMs encountered with `read_csv` on WSL2 #10554

Closed charlesbluca closed 3 weeks ago

charlesbluca commented 2 years ago

Describe the bug While testing the cuGraph's UVM notebook, I encountered an OOM error when trying to read in a large (~26 GB) CSV dataset on an RTX 8000 48GB.

Steps/Code to reproduce bug Sorry for the the lengthy reproducer - happy to switch over a more readily available large dataset if possible:

import os
import urllib.request

import cudf
import rmm

rmm.mr.set_current_device_resource(rmm.mr.ManagedMemoryResource())

data_dir = './data/'
if not os.path.exists(data_dir):
    print('creating data directory')
    os.system('mkdir ./data')

# download the Twitter dataset
base_url = 'https://s3.us-east-2.amazonaws.com/rapidsai-data/cugraph/benchmark/'
fn = 'twitter-2010.csv'
comp = '.gz'
if not os.path.isfile(data_dir+fn):
    if not os.path.isfile(data_dir+fn+comp):
        print(f'Downloading {base_url+fn+comp} to {data_dir+fn+comp}')
        urllib.request.urlretrieve(base_url+fn+comp, data_dir+fn+comp)
    print(f'Decompressing {data_dir+fn+comp}...')
    os.system('gunzip '+data_dir+fn+comp)
    print(f'{data_dir+fn+comp} decompressed!')
else:
    print(f'Your data file, {data_dir+fn}, already exists')

# File path, assuming Notebook directory
input_data_path = data_dir+fn

e_list = cudf.read_csv(input_data_path, delimiter=' ', names=['src', 'dst'], dtype=['int32', 'int32'])

The above fails unless nrows is set to something under ~100,000,000:

MemoryError                               Traceback (most recent call last)
/tmp/ipykernel_880/4119338851.py in <module>
      3 
      4 # CSV reader
----> 5 e_list = cudf.read_csv(input_data_path, delimiter=' ', names=['src', 'dst'], dtype=['int32', 'int32'])
      6 
      7 # Print time

/opt/conda/envs/rapids/lib/python3.9/contextlib.py in inner(*args, **kwds)
     77         def inner(*args, **kwds):
     78             with self._recreate_cm():
---> 79                 return func(*args, **kwds)
     80         return inner
     81 

/opt/conda/envs/rapids/lib/python3.9/site-packages/cudf/io/csv.py in read_csv(filepath_or_buffer, lineterminator, quotechar, quoting, doublequote, header, mangle_dupe_cols, usecols, sep, delimiter, delim_whitespace, skipinitialspace, names, dtype, skipfooter, skiprows, dayfirst, compression, thousands, decimal, true_values, false_values, nrows, byte_range, skip_blank_lines, parse_dates, comment, na_values, keep_default_na, na_filter, prefix, index_col, use_python_file_object, **kwargs)
     71         na_values = [na_values]
     72 
---> 73     return libcudf.csv.read_csv(
     74         filepath_or_buffer,
     75         lineterminator=lineterminator,

cudf/_lib/csv.pyx in cudf._lib.csv.read_csv()

MemoryError: std::bad_alloc: out_of_memory: CUDA error at: /rapids/rmm/include/rmm/mr/device/managed_memory_resource.hpp:74: cudaErrorMemoryAllocation out of memory

Expected behavior I would expect the CSV dataset to be read in its entirety - the notebook and this code succeed on a standard Linux machine with a V100 32GB (DGX1).

Environment overview (please complete the following information)

docker run --gpus all -p 8888:8888 -p 8787:8787 -p 8786:8786 rapidsai/rapidsai-core-dev-nightly:22.04-cuda11.5-devel-ubuntu20.04-py3.9

Environment details

Click here to see environment details

     **git***
print_env.sh: 10: [: true: unexpected operator
     Not inside a git repository

     ***OS Information***
     DISTRIB_ID=Ubuntu
     DISTRIB_RELEASE=20.04
     DISTRIB_CODENAME=focal
     DISTRIB_DESCRIPTION="Ubuntu 20.04.4 LTS"
     NAME="Ubuntu"
     VERSION="20.04.4 LTS (Focal Fossa)"
     ID=ubuntu
     ID_LIKE=debian
     PRETTY_NAME="Ubuntu 20.04.4 LTS"
     VERSION_ID="20.04"
     HOME_URL="https://www.ubuntu.com/"
     SUPPORT_URL="https://help.ubuntu.com/"
     BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
     PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
     VERSION_CODENAME=focal
     UBUNTU_CODENAME=focal
     Linux 4ab1a0161978 5.10.102.1-microsoft-standard-WSL2 #1 SMP Wed Mar 2 00:30:59 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

     ***GPU Information***
     Thu Mar 31 14:45:02 2022
     +-----------------------------------------------------------------------------+
     | NVIDIA-SMI 510.47.03    Driver Version: 511.65       CUDA Version: 11.6     |
     |-------------------------------+----------------------+----------------------+
     | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
     | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
     |                               |                      |               MIG M. |
     |===============================+======================+======================|
     |   0  Quadro RTX 8000     On   | 00000000:15:00.0 Off |                  Off |
     | 34%   32C    P8    17W / 260W |    823MiB / 49152MiB |      0%      Default |
     |                               |                      |                  N/A |
     +-------------------------------+----------------------+----------------------+
     |   1  Quadro RTX 8000     On   | 00000000:2D:00.0  On |                  Off |
     | 35%   61C    P0    72W / 260W |   1906MiB / 49152MiB |     15%      Default |
     |                               |                      |                  N/A |
     +-------------------------------+----------------------+----------------------+

     +-----------------------------------------------------------------------------+
     | Processes:                                                                  |
     |  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
     |        ID   ID                                                   Usage      |
     |=============================================================================|
     |    0   N/A  N/A       804      C   /python3.9                      N/A      |
     |    1   N/A  N/A       804      C   /python3.9                      N/A      |
     +-----------------------------------------------------------------------------+

     ***CPU***
     Architecture:                    x86_64
     CPU op-mode(s):                  32-bit, 64-bit
     Byte Order:                      Little Endian
     Address sizes:                   46 bits physical, 48 bits virtual
     CPU(s):                          12
     On-line CPU(s) list:             0-11
     Thread(s) per core:              2
     Core(s) per socket:              6
     Socket(s):                       1
     Vendor ID:                       GenuineIntel
     CPU family:                      6
     Model:                           85
     Model name:                      Intel(R) Xeon(R) Gold 6128 CPU @ 3.40GHz
     Stepping:                        4
     CPU MHz:                         3391.499
     BogoMIPS:                        6782.99
     Virtualization:                  VT-x
     Hypervisor vendor:               Microsoft
     Virtualization type:             full
     L1d cache:                       192 KiB
     L1i cache:                       192 KiB
     L2 cache:                        6 MiB
     L3 cache:                        19.3 MiB
     Vulnerability Itlb multihit:     KVM: Mitigation: VMX disabled
     Vulnerability L1tf:              Mitigation; PTE Inversion; VMX conditional cache flushes, SMT vulnerable
     Vulnerability Mds:               Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown
     Vulnerability Meltdown:          Mitigation; PTI
     Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
     Vulnerability Spectre v1:        Mitigation; usercopy/swapgs barriers and __user pointer sanitization
     Vulnerability Spectre v2:        Mitigation; Full generic retpoline, IBPB conditional, IBRS_FW, STIBP conditional, RSB filling
     Vulnerability Srbds:             Not affected
     Vulnerability Tsx async abort:   Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown
     Flags:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology tsc_reliable nonstop_tsc cpuid pni pclmulqdq vmx ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single pti ssbd ibrs ibpb stibp tpr_shadow vnmi ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves flush_l1d arch_capabilities

     ***CMake***
     /opt/conda/envs/rapids/bin/cmake
     cmake version 3.20.5

     CMake suite maintained and supported by Kitware (kitware.com/cmake).

     ***g++***
     /usr/bin/g++
     g++ (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
     Copyright (C) 2019 Free Software Foundation, Inc.
     This is free software; see the source for copying conditions.  There is NO
     warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

     ***nvcc***
     /usr/local/cuda/bin/nvcc
     nvcc: NVIDIA (R) Cuda compiler driver
     Copyright (c) 2005-2021 NVIDIA Corporation
     Built on Thu_Nov_18_09:45:30_PST_2021
     Cuda compilation tools, release 11.5, V11.5.119
     Build cuda_11.5.r11.5/compiler.30672275_0

     ***Python***
     /opt/conda/envs/rapids/bin/python
     Python 3.9.12

     ***Environment Variables***
     PATH                            : /opt/conda/envs/rapids/bin:/opt/conda/condabin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
     LD_LIBRARY_PATH                 : /usr/local/nvidia/lib:/usr/local/nvidia/lib64:/usr/local/cuda/lib64:/usr/local/lib:/opt/conda/envs/rapids/lib
     NUMBAPRO_NVVM                   :
     NUMBAPRO_LIBDEVICE              :
     CONDA_PREFIX                    : /opt/conda/envs/rapids
     PYTHON_PATH                     :

     ***conda packages***
     conda is /opt/conda/envs/rapids/bin/conda
     /opt/conda/envs/rapids/bin/conda
     # packages in environment at /opt/conda/envs/rapids:
     #
     # Name                    Version                   Build  Channel
     _libgcc_mutex             0.1                 conda_forge    conda-forge
     _openmp_mutex             4.5                       1_gnu    conda-forge
     abseil-cpp                20211102.0           h27087fc_1    conda-forge
     aiobotocore               2.1.0              pyhd8ed1ab_0    conda-forge
     aiohttp                   3.8.1            py39h3811e60_0    conda-forge
     aioitertools              0.10.0             pyhd8ed1ab_0    conda-forge
     aiosignal                 1.2.0              pyhd8ed1ab_0    conda-forge
     alabaster                 0.7.12                     py_0    conda-forge
     anyio                     3.5.0            py39hf3d152e_0    conda-forge
     appdirs                   1.4.4              pyh9f0ad1d_0    conda-forge
     argon2-cffi               21.3.0             pyhd8ed1ab_0    conda-forge
     argon2-cffi-bindings      21.2.0           py39h3811e60_1    conda-forge
     arrow-cpp                 6.0.1           py39hbfed05e_11_cuda    conda-forge
     arrow-cpp-proc            3.0.0                      cuda    conda-forge
     asn1crypto                1.5.1              pyhd8ed1ab_0    conda-forge
     asvdb                     0.4.2               g90e8f2c_40    rapidsai-nightly
     async-timeout             4.0.2              pyhd8ed1ab_0    conda-forge
     atk-1.0                   2.36.0               h3371d22_4    conda-forge
     attrs                     21.4.0             pyhd8ed1ab_0    conda-forge
     autoconf                  2.69            pl5321hd708f79_11    conda-forge
     automake                  1.16.5          pl5321ha770c72_0    conda-forge
     aws-c-cal                 0.5.11               h95a6274_0    conda-forge
     aws-c-common              0.6.2                h7f98852_0    conda-forge
     aws-c-event-stream        0.2.7               h3541f99_13    conda-forge
     aws-c-io                  0.10.5               hfb6a706_0    conda-forge
     aws-checksums             0.1.11               ha31a3da_7    conda-forge
     aws-sam-translator        1.43.0             pyhd8ed1ab_0    conda-forge
     aws-sdk-cpp               1.8.186              hb4091e7_3    conda-forge
     aws-xray-sdk              2.9.0              pyhd8ed1ab_0    conda-forge
     babel                     2.9.1              pyh44b312d_0    conda-forge
     backcall                  0.2.0              pyh9f0ad1d_0    conda-forge
     backports                 1.0                        py_2    conda-forge
     backports.functools_lru_cache 1.6.4              pyhd8ed1ab_0    conda-forge
     beautifulsoup4            4.10.0             pyha770c72_0    conda-forge
     benchmark                 1.5.1                he1b5a44_2    conda-forge
     binutils_impl_linux-64    2.36.1               h193b22a_2    conda-forge
     black                     19.10b0                    py_4    conda-forge
     blas                      2.105                    netlib    conda-forge
     blas-devel                3.9.0                  5_netlib    conda-forge
     bleach                    4.1.0              pyhd8ed1ab_0    conda-forge
     blinker                   1.4                        py_1    conda-forge
     blosc                     1.21.0               h9c3ff4c_0    conda-forge
     bokeh                     2.4.2            py39hf3d152e_0    conda-forge
     boost                     1.72.0           py39ha90915f_1    conda-forge
     boost-cpp                 1.72.0               h312852a_5    conda-forge
     boto3                     1.20.24            pyhd8ed1ab_0    conda-forge
     botocore                  1.23.24            pyhd8ed1ab_0    conda-forge
     breathe                   4.33.0             pyhd8ed1ab_0    conda-forge
     brotli                    1.0.9                h7f98852_6    conda-forge
     brotli-bin                1.0.9                h7f98852_6    conda-forge
     brotlipy                  0.7.0           py39h3811e60_1003    conda-forge
     brunsli                   0.1                  h9c3ff4c_0    conda-forge
     bzip2                     1.0.8                h7f98852_4    conda-forge
     c-ares                    1.18.1               h7f98852_0    conda-forge
     c-blosc2                  2.0.4                h5f21a17_1    conda-forge
     ca-certificates           2021.10.8            ha878542_0    conda-forge
     cached-property           1.5.2                hd8ed1ab_1    conda-forge
     cached_property           1.5.2              pyha770c72_1    conda-forge
     cachetools                5.0.0              pyhd8ed1ab_0    conda-forge
     cairo                     1.16.0            h6cf1ce9_1008    conda-forge
     certifi                   2021.10.8        py39hf3d152e_1    conda-forge
     cffi                      1.15.0           py39h4bc2ebd_0    conda-forge
     cfgv                      3.3.1              pyhd8ed1ab_0    conda-forge
     cfitsio                   3.470                hb418390_7    conda-forge
     cfn-lint                  0.54.2           py39hf3d152e_0    conda-forge
     chardet                   4.0.0            py39hf3d152e_2    conda-forge
     charls                    2.2.0                h9c3ff4c_0    conda-forge
     charset-normalizer        2.0.12             pyhd8ed1ab_0    conda-forge
     clang                     11.1.0               ha770c72_1    conda-forge
     clang-11                  11.1.0          default_ha53f305_1    conda-forge
     clang-tools               11.1.0          default_ha53f305_1    conda-forge
     clangxx                   11.1.0          default_ha53f305_1    conda-forge
     click                     7.1.2              pyh9f0ad1d_0    conda-forge
     click-plugins             1.1.1                      py_0    conda-forge
     cligj                     0.7.2              pyhd8ed1ab_1    conda-forge
     cloudpickle               2.0.0              pyhd8ed1ab_0    conda-forge
     cmake                     3.20.5               h8897547_0    conda-forge
     cmake-format              0.6.11             pyh9f0ad1d_0    conda-forge
     cmake_setuptools          0.1.3                      py_0    rapidsai-nightly
     cmarkgfm                  0.8.0            py39hb9d737c_0    conda-forge
     colorama                  0.4.4              pyh9f0ad1d_0    conda-forge
     colorcet                  3.0.0              pyhd8ed1ab_0    conda-forge
     commonmark                0.9.1                      py_0    conda-forge
     conda                     4.10.3           py39hf3d152e_4    conda-forge
     conda-build               3.21.7           py39hf3d152e_0    conda-forge
     conda-package-handling    1.8.0            py39hb9d737c_0    conda-forge
     conda-verify              3.1.1           py39hf3d152e_1004    conda-forge
     coverage                  6.3.2            py39hb9d737c_1    conda-forge
     cryptography              36.0.2           py39hd97740a_0    conda-forge
     cuda-python               11.6.1           py39h3fd9d12_0    nvidia
     cudatoolkit               11.5.1               hcf5317a_9    nvidia
     cudf                      0+untagged.1.gc71fe1b          pypi_0    pypi
     cudf-kafka                0+untagged.1.gc71fe1b          pypi_0    pypi
     cugraph                   0+untagged.1.g87be0b3          pypi_0    pypi
     cuml                      0+untagged.1.g3798925          pypi_0    pypi
     cupy                      10.2.0           py39hc3c280e_0    conda-forge
     curl                      7.82.0               h7bff187_0    conda-forge
     cusignal                  0+untagged.1.g8878bf7          pypi_0    pypi
     cuspatial                 0+untagged.1.g3637da5          pypi_0    pypi
     cuxfilter                 0+untagged.1.g53c9564          pypi_0    pypi
     cycler                    0.11.0             pyhd8ed1ab_0    conda-forge
     cyrus-sasl                2.1.27               h230043b_5    conda-forge
     cython                    0.29.28          py39he80948d_0    conda-forge
     cytoolz                   0.11.2           py39h3811e60_1    conda-forge
     dask                      2022.3.0           pyhd8ed1ab_0    conda-forge
     dask-core                 2022.3.0           pyhd8ed1ab_0    conda-forge
     dask-cudf                 0+untagged.1.gc71fe1b          pypi_0    pypi
     dask-glm                  0.2.0                      py_1    conda-forge
     dask-labextension         5.2.0              pyhd8ed1ab_0    conda-forge
     dask-ml                   1.9.0              pyhd8ed1ab_0    conda-forge
     dataclasses               0.8                pyhc8e2a94_3    conda-forge
     datashader                0.13.0             pyh6c4a22f_0    conda-forge
     datashape                 0.5.4                      py_1    conda-forge
     dbus                      1.13.6               h5008d03_3    conda-forge
     debugpy                   1.5.1            py39he80948d_0    conda-forge
     decorator                 5.1.1              pyhd8ed1ab_0    conda-forge
     defusedxml                0.7.1              pyhd8ed1ab_0    conda-forge
     distlib                   0.3.4              pyhd8ed1ab_0    conda-forge
     distributed               2022.3.0           pyhd8ed1ab_0    conda-forge
     distro                    1.6.0              pyhd8ed1ab_0    conda-forge
     dlpack                    0.5                  h9c3ff4c_0    conda-forge
     docker-py                 5.0.3            py39hf3d152e_2    conda-forge
     docker-pycreds            0.4.0                      py_0    conda-forge
     docutils                  0.17.1           py39hf3d152e_1    conda-forge
     double-conversion         3.1.5                h9c3ff4c_2    conda-forge
     doxygen                   1.8.20               had0d8f1_0    conda-forge
     ecdsa                     0.17.0             pyhd8ed1ab_0    conda-forge
     entrypoints               0.4                pyhd8ed1ab_0    conda-forge
     execnet                   1.9.0              pyhd8ed1ab_0    conda-forge
     expat                     2.4.7                h27087fc_0    conda-forge
     faiss-proc                1.0.0                      cuda    rapidsai
     fastavro                  1.4.10           py39hb9d737c_0    conda-forge
     fastrlock                 0.8              py39he80948d_1    conda-forge
     feather-format            0.4.1              pyh9f0ad1d_0    conda-forge
     filelock                  3.6.0              pyhd8ed1ab_0    conda-forge
     filterpy                  1.4.5                      py_1    conda-forge
     fiona                     1.8.20           py39h427c1bf_1    conda-forge
     flake8                    3.8.4                      py_0    conda-forge
     flask                     2.0.3              pyhd8ed1ab_0    conda-forge
     flask_cors                3.0.10             pyhd3deb0d_0    conda-forge
     flit-core                 3.7.1              pyhd8ed1ab_0    conda-forge
     font-ttf-dejavu-sans-mono 2.37                 hab24e00_0    conda-forge
     font-ttf-inconsolata      3.000                h77eed37_0    conda-forge
     font-ttf-source-code-pro  2.038                h77eed37_0    conda-forge
     font-ttf-ubuntu           0.83                 hab24e00_0    conda-forge
     fontconfig                2.13.96              h8e229c2_2    conda-forge
     fonts-conda-ecosystem     1                             0    conda-forge
     fonts-conda-forge         1                             0    conda-forge
     fonttools                 4.31.2           py39hb9d737c_0    conda-forge
     freetype                  2.10.4               h0708190_1    conda-forge
     freexl                    1.0.6                h7f98852_0    conda-forge
     fribidi                   1.0.10               h36c2ea0_0    conda-forge
     frozenlist                1.3.0            py39h3811e60_0    conda-forge
     fsspec                    2022.2.0           pyhd8ed1ab_0    conda-forge
     future                    0.18.2           py39hf3d152e_4    conda-forge
     gcc_impl_linux-64         11.2.0              h82a94d6_14    conda-forge
     gcsfs                     2022.2.0           pyhd8ed1ab_0    conda-forge
     gdal                      3.3.1            py39h218ed2d_3    conda-forge
     gdk-pixbuf                2.42.8               hff1cb4f_0    conda-forge
     geopandas                 0.9.0              pyhd8ed1ab_1    conda-forge
     geopandas-base            0.9.0              pyhd8ed1ab_1    conda-forge
     geos                      3.9.1                h9c3ff4c_2    conda-forge
     geotiff                   1.6.0                h4f31c25_6    conda-forge
     gettext                   0.19.8.1          h73d1719_1008    conda-forge
     gflags                    2.2.2             he1b5a44_1004    conda-forge
     giflib                    5.2.1                h36c2ea0_2    conda-forge
     git                       2.35.1          pl5321h36853c3_0    conda-forge
     git-lfs                   3.1.2                ha770c72_0    conda-forge
     glib                      2.70.2               h780b84a_4    conda-forge
     glib-tools                2.70.2               h780b84a_4    conda-forge
     glob2                     0.7                        py_0    conda-forge
     glog                      0.5.0                h48cff8f_0    conda-forge
     gmock                     1.10.0               h4bd325d_7    conda-forge
     gmp                       6.2.1                h58526e2_0    conda-forge
     google-api-core           2.5.0              pyhd8ed1ab_0    conda-forge
     google-auth               2.6.2              pyh6c4a22f_0    conda-forge
     google-auth-oauthlib      0.5.1              pyhd8ed1ab_0    conda-forge
     google-cloud-core         2.2.2              pyh6c4a22f_0    conda-forge
     google-cloud-storage      2.1.0              pyh6c4a22f_0    conda-forge
     google-crc32c             1.1.2            py39hb81f231_2    conda-forge
     google-resumable-media    2.1.0              pyh6c4a22f_0    conda-forge
     googleapis-common-protos  1.56.0           py39hf3d152e_0    conda-forge
     gpuci-tools               0.3.1                        12    gpuci
     graphite2                 1.3.13            h58526e2_1001    conda-forge
     graphql-core              3.2.0              pyhd8ed1ab_0    conda-forge
     graphviz                  2.50.0               h85b4f2f_1    conda-forge
     grpc-cpp                  1.44.0               h3d78c48_1    conda-forge
     grpcio                    1.45.0           py39h0f497a6_0    conda-forge
     gtest                     1.10.0               h4bd325d_7    conda-forge
     gtk2                      2.24.33              h539f30e_1    conda-forge
     gts                       0.7.6                h64030ff_2    conda-forge
     h5py                      3.6.0           nompi_py39h7e08c79_100    conda-forge
     harfbuzz                  3.1.1                h83ec7ef_0    conda-forge
     hdbscan                   0.8.28           py39hce5d2b2_1    conda-forge
     hdf4                      4.2.15               h10796ff_3    conda-forge
     hdf5                      1.12.1          nompi_h2386368_104    conda-forge
     heapdict                  1.0.1                      py_0    conda-forge
     holoviews                 1.14.6             pyhd8ed1ab_0    conda-forge
     html5lib                  1.1                pyh9f0ad1d_0    conda-forge
     httpretty                 1.1.4              pyhd8ed1ab_0    conda-forge
     huggingface_hub           0.4.0              pyhd8ed1ab_0    conda-forge
     hypothesis                6.39.5             pyhd8ed1ab_0    conda-forge
     icu                       68.2                 h9c3ff4c_0    conda-forge
     identify                  2.4.12             pyhd8ed1ab_0    conda-forge
     idna                      3.3                pyhd8ed1ab_0    conda-forge
     imagecodecs               2021.8.26        py39h44211f0_1    conda-forge
     imageio                   2.16.1             pyhcf75d05_0    conda-forge
     imagesize                 1.3.0              pyhd8ed1ab_0    conda-forge
     importlib-metadata        4.11.3           py39hf3d152e_0    conda-forge
     importlib_metadata        4.11.3               hd8ed1ab_0    conda-forge
     iniconfig                 1.1.1              pyh9f0ad1d_0    conda-forge
     ipykernel                 6.9.2            py39hef51801_0    conda-forge
     ipython                   7.31.1           py39hf3d152e_0    conda-forge
     ipython_genutils          0.2.0                      py_1    conda-forge
     ipywidgets                7.7.0              pyhd8ed1ab_0    conda-forge
     isort                     5.6.4                      py_0    conda-forge
     itsdangerous              2.1.2              pyhd8ed1ab_0    conda-forge
     jbig                      2.1               h7f98852_2003    conda-forge
     jedi                      0.18.1           py39hf3d152e_0    conda-forge
     jeepney                   0.7.1              pyhd8ed1ab_0    conda-forge
     jinja2                    3.1.1              pyhd8ed1ab_0    conda-forge
     jmespath                  0.10.0             pyh9f0ad1d_0    conda-forge
     joblib                    1.1.0              pyhd8ed1ab_0    conda-forge
     jpeg                      9e                   h7f98852_0    conda-forge
     json-c                    0.15                 h98cffda_0    conda-forge
     json5                     0.9.5              pyh9f0ad1d_0    conda-forge
     jsondiff                  1.3.1              pyhd8ed1ab_0    conda-forge
     jsonpatch                 1.32               pyhd8ed1ab_0    conda-forge
     jsonpointer               2.0                        py_0    conda-forge
     jsonschema                3.2.0              pyhd8ed1ab_3    conda-forge
     junit-xml                 1.9                pyh9f0ad1d_0    conda-forge
     jupyter-packaging         0.7.12             pyhd8ed1ab_0    conda-forge
     jupyter-server-proxy      3.2.1              pyhd8ed1ab_0    conda-forge
     jupyter_client            7.1.2              pyhd8ed1ab_0    conda-forge
     jupyter_core              4.9.2            py39hf3d152e_0    conda-forge
     jupyter_server            1.15.6             pyhd8ed1ab_1    conda-forge
     jupyter_sphinx            0.3.2            py39hf3d152e_0    conda-forge
     jupyterlab                3.3.2              pyhd8ed1ab_0    conda-forge
     jupyterlab-favorites      3.0.0              pyhd8ed1ab_0    conda-forge
     jupyterlab-nvdashboard    0.7.0a220307              py_21    rapidsai-nightly
     jupyterlab_pygments       0.1.2              pyh9f0ad1d_0    conda-forge
     jupyterlab_server         2.11.0             pyhd8ed1ab_0    conda-forge
     jupyterlab_widgets        1.1.0              pyhd8ed1ab_0    conda-forge
     jxrlib                    1.1                  h7f98852_2    conda-forge
     kealib                    1.4.14               h87e4c3c_3    conda-forge
     kernel-headers_linux-64   2.6.32              he073ed8_15    conda-forge
     keyring                   23.4.0           py39hf3d152e_2    conda-forge
     keyutils                  1.6.1                h166bdaf_0    conda-forge
     kiwisolver                1.4.0            py39hf939315_0    conda-forge
     krb5                      1.19.3               h3790be6_0    conda-forge
     lapack                    3.9.0                    netlib    conda-forge
     lcms2                     2.12                 hddcbb42_0    conda-forge
     ld_impl_linux-64          2.36.1               hea4e1c9_2    conda-forge
     lerc                      3.0                  h9c3ff4c_0    conda-forge
     libaec                    1.0.6                h9c3ff4c_0    conda-forge
     libarchive                3.5.2                hccf745f_1    conda-forge
     libblas                   3.9.0           5_h92ddd45_netlib    conda-forge
     libbrotlicommon           1.0.9                h7f98852_6    conda-forge
     libbrotlidec              1.0.9                h7f98852_6    conda-forge
     libbrotlienc              1.0.9                h7f98852_6    conda-forge
     libcblas                  3.9.0           5_h92ddd45_netlib    conda-forge
     libclang-cpp11.1          11.1.0          default_ha53f305_1    conda-forge
     libcrc32c                 1.1.2                h9c3ff4c_0    conda-forge
     libcugraphops             22.04.00a220329 cuda11_g91ab46e_30    rapidsai-nightly
     libcumlprims              22.04.00a220324 cuda11_g99e8d8f_15    rapidsai-nightly
     libcurl                   7.82.0               h7bff187_0    conda-forge
     libcusolver               11.3.4.124           h33c3c4e_0    nvidia
     libcypher-parser          0.6.2                         1    rapidsai-nightly
     libdap4                   3.20.6               hd7c4107_2    conda-forge
     libdeflate                1.8                  h7f98852_0    conda-forge
     libedit                   3.1.20191231         he28a2e2_2    conda-forge
     libev                     4.33                 h516909a_1    conda-forge
     libevent                  2.1.10               h9b69904_4    conda-forge
     libfaiss                  1.7.0           cuda112h5bea7ad_8_cuda    conda-forge
     libffi                    3.4.2                h7f98852_5    conda-forge
     libgcc-devel_linux-64     11.2.0              h0952999_14    conda-forge
     libgcc-ng                 11.2.0              h1d223b6_14    conda-forge
     libgcrypt                 1.10.0               h7f98852_0    conda-forge
     libgd                     2.3.3                h6ad9fb6_0    conda-forge
     libgdal                   3.3.1                h6214c1d_3    conda-forge
     libgfortran-ng            11.2.0              h69a702a_14    conda-forge
     libgfortran5              11.2.0              h5c6108e_14    conda-forge
     libglib                   2.70.2               h174f98d_4    conda-forge
     libgomp                   11.2.0              h1d223b6_14    conda-forge
     libgpg-error              1.44                 h9eb791d_0    conda-forge
     libgsasl                  1.10.0               h5b4c23d_0    conda-forge
     libhwloc                  2.3.0                h5e5b7d1_1    conda-forge
     libiconv                  1.16                 h516909a_0    conda-forge
     libkml                    1.3.0             hd79254b_1012    conda-forge
     liblapack                 3.9.0           5_h92ddd45_netlib    conda-forge
     liblapacke                3.9.0           5_h92ddd45_netlib    conda-forge
     liblief                   0.11.5               h9c3ff4c_1    conda-forge
     libllvm11                 11.1.0               hf817b99_3    conda-forge
     libnetcdf                 4.8.1           nompi_hb3fd0d9_101    conda-forge
     libnghttp2                1.47.0               h727a467_0    conda-forge
     libnsl                    2.0.0                h7f98852_0    conda-forge
     libntlm                   1.4               h7f98852_1002    conda-forge
     libpng                    1.6.37               h21135ba_2    conda-forge
     libpq                     13.5                 hd57d9b9_1    conda-forge
     libprotobuf               3.19.4               h780b84a_0    conda-forge
     librdkafka                1.7.0                hc49e61c_1    conda-forge
     librmm                    22.04.00a220329 cuda11_g220ba88_44    rapidsai-nightly
     librsvg                   2.52.5               hc3c00ef_1    conda-forge
     librttopo                 1.1.0                h1185371_6    conda-forge
     libsanitizer              11.2.0              he4da1e4_14    conda-forge
     libsodium                 1.0.18               h36c2ea0_1    conda-forge
     libspatialindex           1.9.3                h9c3ff4c_4    conda-forge
     libspatialite             5.0.1                h8694cbe_6    conda-forge
     libssh2                   1.10.0               ha56f1ee_2    conda-forge
     libstdcxx-ng              11.2.0              he4da1e4_14    conda-forge
     libthrift                 0.15.0               he6d91bd_1    conda-forge
     libtiff                   4.3.0                h6f004c6_2    conda-forge
     libtmglib                 3.9.0           5_h92ddd45_netlib    conda-forge
     libtool                   2.4.6             h9c3ff4c_1008    conda-forge
     libutf8proc               2.7.0                h7f98852_0    conda-forge
     libuuid                   2.32.1            h7f98852_1000    conda-forge
     libuv                     1.43.0               h7f98852_0    conda-forge
     libwebp                   1.2.2                h3452ae3_0    conda-forge
     libwebp-base              1.2.2                h7f98852_1    conda-forge
     libxcb                    1.13              h7f98852_1004    conda-forge
     libxml2                   2.9.12               h72842e0_0    conda-forge
     libzip                    1.8.0                h4de3113_1    conda-forge
     libzlib                   1.2.11            h166bdaf_1014    conda-forge
     libzopfli                 1.0.3                h9c3ff4c_0    conda-forge
     lightgbm                  3.3.2            py39he80948d_0    conda-forge
     llvmlite                  0.38.0           py39h1bbdace_0    conda-forge
     locket                    0.2.1                    pypi_0    pypi
     lz4-c                     1.9.3                h9c3ff4c_1    conda-forge
     lzo                       2.10              h516909a_1000    conda-forge
     m4                        1.4.18            h516909a_1001    conda-forge
     make                      4.3                  hd18ef5c_1    conda-forge
     mapclassify               2.4.3              pyhd8ed1ab_0    conda-forge
     markdown                  3.3.6              pyhd8ed1ab_0    conda-forge
     markupsafe                2.1.1            py39hb9d737c_0    conda-forge
     matplotlib-base           3.5.1            py39h2fa2bec_0    conda-forge
     matplotlib-inline         0.1.3              pyhd8ed1ab_0    conda-forge
     mccabe                    0.6.1                      py_1    conda-forge
     mimesis                   4.0.0              pyh9f0ad1d_0    conda-forge
     mistune                   0.8.4           py39h3811e60_1005    conda-forge
     mock                      4.0.3            py39hf3d152e_2    conda-forge
     moto                      3.1.2              pyhd8ed1ab_0    conda-forge
     msgpack-python            1.0.3            py39h1a9c180_0    conda-forge
     multidict                 6.0.2            py39h3811e60_0    conda-forge
     multipledispatch          0.6.0                      py_0    conda-forge
     munch                     2.5.0                      py_0    conda-forge
     munkres                   1.1.4              pyh9f0ad1d_0    conda-forge
     mypy                      0.782                      py_0    conda-forge
     mypy_extensions           0.4.3            py39hf3d152e_4    conda-forge
     nbclassic                 0.3.7              pyhd8ed1ab_0    conda-forge
     nbclient                  0.5.13             pyhd8ed1ab_0    conda-forge
     nbconvert                 6.4.4            py39hf3d152e_0    conda-forge
     nbformat                  5.2.0              pyhd8ed1ab_0    conda-forge
     nbsphinx                  0.8.8              pyhd8ed1ab_0    conda-forge
     nccl                      2.12.7.1             h0800d71_0    conda-forge
     ncurses                   6.3                  h9c3ff4c_0    conda-forge
     nest-asyncio              1.5.4              pyhd8ed1ab_0    conda-forge
     networkx                  2.6.3              pyhd8ed1ab_1    conda-forge
     ninja                     1.10.2               h4bd325d_1    conda-forge
     nltk                      3.6.7              pyhd8ed1ab_0    conda-forge
     nodeenv                   1.6.0              pyhd8ed1ab_0    conda-forge
     nodejs                    14.18.3              h92b4a50_1    conda-forge
     notebook                  6.4.10             pyha770c72_0    conda-forge
     notebook-shim             0.1.0              pyhd8ed1ab_0    conda-forge
     numba                     0.55.1           py39h56b8d98_0    conda-forge
     numpy                     1.21.5           py39haac66dc_0    conda-forge
     numpydoc                  1.2                pyhd8ed1ab_0    conda-forge
     nvtx                      0.2.3            py39h3811e60_1    conda-forge
     oauthlib                  3.2.0              pyhd8ed1ab_0    conda-forge
     openjpeg                  2.4.0                hb52868f_1    conda-forge
     openslide                 3.4.1                h978ee9a_4    conda-forge
     openssl                   1.1.1n               h166bdaf_0    conda-forge
     orc                       1.7.3                h1be678f_0    conda-forge
     packaging                 21.3               pyhd8ed1ab_0    conda-forge
     pandas                    1.3.5            py39hde0f152_0    conda-forge
     pandoc                    1.19.2                        0    conda-forge
     pandocfilters             1.5.0              pyhd8ed1ab_0    conda-forge
     panel                     0.12.6             pyhd8ed1ab_0    conda-forge
     pango                     1.48.10              h54213e6_2    conda-forge
     param                     1.12.0             pyh6c4a22f_0    conda-forge
     parquet-cpp               1.5.1                         2    conda-forge
     parso                     0.8.3              pyhd8ed1ab_0    conda-forge
     partd                     1.2.0              pyhd8ed1ab_0    conda-forge
     patchelf                  0.14.5               h58526e2_0    conda-forge
     pathspec                  0.9.0              pyhd8ed1ab_0    conda-forge
     patsy                     0.5.2              pyhd8ed1ab_0    conda-forge
     pcre                      8.45                 h9c3ff4c_0    conda-forge
     pcre2                     10.37                h032f7d1_0    conda-forge
     perl                      5.32.1          2_h7f98852_perl5    conda-forge
     pexpect                   4.8.0              pyh9f0ad1d_2    conda-forge
     pickleshare               0.7.5                   py_1003    conda-forge
     pillow                    9.0.1            py39hae2aec6_2    conda-forge
     pip                       22.0.4             pyhd8ed1ab_0    conda-forge
     pixman                    0.40.0               h36c2ea0_0    conda-forge
     pkg-config                0.29.2            h36c2ea0_1008    conda-forge
     pkginfo                   1.8.2              pyhd8ed1ab_0    conda-forge
     platformdirs              2.5.1              pyhd8ed1ab_0    conda-forge
     pluggy                    1.0.0            py39hf3d152e_2    conda-forge
     poppler                   21.03.0              h93df280_0    conda-forge
     poppler-data              0.4.11               hd8ed1ab_0    conda-forge
     postgresql                13.5                 h2510834_1    conda-forge
     pre-commit                2.17.0           py39hf3d152e_0    conda-forge
     proj                      8.0.1                h277dcde_0    conda-forge
     prometheus_client         0.13.1             pyhd8ed1ab_0    conda-forge
     prompt-toolkit            3.0.27             pyha770c72_0    conda-forge
     protobuf                  3.19.4           py39he80948d_0    conda-forge
     psutil                    5.9.0            py39h3811e60_0    conda-forge
     pthread-stubs             0.4               h36c2ea0_1001    conda-forge
     ptxcompiler               0.2.0            py39h107f55c_0    rapidsai-nightly
     ptyprocess                0.7.0              pyhd3deb0d_0    conda-forge
     py                        1.11.0             pyh6c4a22f_0    conda-forge
     py-cpuinfo                8.0.0              pyhd8ed1ab_0    conda-forge
     py-lief                   0.11.5           py39he80948d_1    conda-forge
     pyarrow                   6.0.1           py39h1ed2e5d_11_cuda    conda-forge
     pyasn1                    0.4.8                      py_0    conda-forge
     pyasn1-modules            0.2.7                      py_0    conda-forge
     pycodestyle               2.6.0              pyh9f0ad1d_0    conda-forge
     pycosat                   0.6.3           py39h3811e60_1009    conda-forge
     pycparser                 2.21               pyhd8ed1ab_0    conda-forge
     pyct                      0.4.6                      py_0    conda-forge
     pyct-core                 0.4.6                      py_0    conda-forge
     pydata-sphinx-theme       0.8.1              pyhd8ed1ab_0    conda-forge
     pydeck                    0.5.0              pyh9f0ad1d_0    conda-forge
     pydocstyle                6.1.1              pyhd8ed1ab_0    conda-forge
     pyee                      8.1.0              pyhd8ed1ab_0    conda-forge
     pyflakes                  2.2.0              pyh9f0ad1d_0    conda-forge
     pygal                     3.0.0                    pypi_0    pypi
     pygments                  2.11.2             pyhd8ed1ab_0    conda-forge
     pyjwt                     2.3.0              pyhd8ed1ab_1    conda-forge
     pylibcugraph              0+untagged.1.g87be0b3          pypi_0    pypi
     pylibraft                 0+untagged.1.g6aa9ac8          pypi_0    pypi
     pynndescent               0.5.6              pyh6c4a22f_0    conda-forge
     pynvml                    11.4.1             pyhd8ed1ab_0    conda-forge
     pyopenssl                 22.0.0             pyhd8ed1ab_0    conda-forge
     pyorc                     0.6.0            py39h40ac9ea_0    conda-forge
     pyparsing                 3.0.7              pyhd8ed1ab_0    conda-forge
     pyppeteer                 1.0.2              pyhd8ed1ab_0    conda-forge
     pyproj                    3.1.0            py39ha9a7ae0_4    conda-forge
     pyrsistent                0.18.1           py39h3811e60_0    conda-forge
     pysocks                   1.7.1            py39hf3d152e_4    conda-forge
     pytest                    7.1.1            py39hf3d152e_0    conda-forge
     pytest-asyncio            0.12.0           py39hde42818_2    conda-forge
     pytest-benchmark          3.4.1              pyhd8ed1ab_0    conda-forge
     pytest-cov                3.0.0              pyhd8ed1ab_0    conda-forge
     pytest-forked             1.4.0              pyhd8ed1ab_0    conda-forge
     pytest-timeout            2.1.0              pyhd8ed1ab_0    conda-forge
     pytest-xdist              2.5.0              pyhd8ed1ab_0    conda-forge
     python                    3.9.12          h9a8a25e_1_cpython    conda-forge
     python-confluent-kafka    1.7.0            py39h3811e60_2    conda-forge
     python-dateutil           2.8.2              pyhd8ed1ab_0    conda-forge
     python-jose               3.3.0              pyh6c4a22f_1    conda-forge
     python-libarchive-c       4.0              py39hf3d152e_0    conda-forge
     python-louvain            0.15               pyhd8ed1ab_1    conda-forge
     python-snappy             0.6.0            py39h300dd49_1    conda-forge
     python_abi                3.9                      2_cp39    conda-forge
     pytz                      2022.1             pyhd8ed1ab_0    conda-forge
     pyu2f                     0.1.5              pyhd8ed1ab_0    conda-forge
     pyviz_comms               2.1.0              pyhd8ed1ab_0    conda-forge
     pywavelets                1.3.0            py39hd257fcd_0    conda-forge
     pyyaml                    6.0              py39h3811e60_3    conda-forge
     pyzmq                     22.3.0           py39h37b5a0c_1    conda-forge
     raft                      0+untagged.1.g6aa9ac8          pypi_0    pypi
     rapidjson                 1.1.0             he1b5a44_1002    conda-forge
     re2                       2022.02.01           h9c3ff4c_0    conda-forge
     readline                  8.1                  h46c0cb4_0    conda-forge
     readme_renderer           27.0               pyh9f0ad1d_0    conda-forge
     recommonmark              0.7.1              pyhd8ed1ab_0    conda-forge
     regex                     2022.3.15        py39hb9d737c_0    conda-forge
     requests                  2.27.1             pyhd8ed1ab_0    conda-forge
     requests-oauthlib         1.3.1              pyhd8ed1ab_0    conda-forge
     requests-toolbelt         0.9.1                      py_0    conda-forge
     responses                 0.20.0             pyhd8ed1ab_0    conda-forge
     rfc3986                   2.0.0              pyhd8ed1ab_0    conda-forge
     rhash                     1.4.1                h7f98852_0    conda-forge
     ripgrep                   13.0.0               h2f28480_2    conda-forge
     rmm                       0+untagged.1.g220ba88          pypi_0    pypi
     rsa                       4.8                pyhd8ed1ab_0    conda-forge
     rtree                     0.9.7            py39hb102c33_3    conda-forge
     ruamel_yaml               0.15.80         py39h3811e60_1006    conda-forge
     s2n                       1.0.10               h9b69904_0    conda-forge
     s3fs                      2022.2.0           pyhd8ed1ab_0    conda-forge
     s3transfer                0.5.2              pyhd8ed1ab_0    conda-forge
     sacremoses                0.0.49             pyhd8ed1ab_0    conda-forge
     sccache                   0.2.15               h9b69904_1    conda-forge
     scikit-build              0.13.1             pyhca92ed8_0    conda-forge
     scikit-image              0.19.2           py39hde0f152_0    conda-forge
     scikit-learn              0.24.2           py39h7c5d8c9_1    conda-forge
     scipy                     1.6.0            py39hee8e79c_0    conda-forge
     seaborn                   0.11.2               hd8ed1ab_0    conda-forge
     seaborn-base              0.11.2             pyhd8ed1ab_0    conda-forge
     secretstorage             3.3.1            py39hf3d152e_1    conda-forge
     send2trash                1.8.0              pyhd8ed1ab_0    conda-forge
     setuptools                59.8.0           py39hf3d152e_0    conda-forge
     shapely                   1.8.0            py39ha61afbd_0    conda-forge
     shellcheck                0.8.0                ha770c72_0    conda-forge
     simpervisor               0.4                pyhd8ed1ab_0    conda-forge
     six                       1.16.0             pyh6c4a22f_0    conda-forge
     snappy                    1.1.8                he1b5a44_3    conda-forge
     sniffio                   1.2.0            py39hf3d152e_2    conda-forge
     snowballstemmer           2.2.0              pyhd8ed1ab_0    conda-forge
     sortedcontainers          2.4.0              pyhd8ed1ab_0    conda-forge
     soupsieve                 2.3.1              pyhd8ed1ab_0    conda-forge
     spdlog                    1.8.5                h4bd325d_1    conda-forge
     sphinx                    4.4.0              pyh6c4a22f_1    conda-forge
     sphinx-click              3.1.0              pyhd8ed1ab_0    conda-forge
     sphinx-copybutton         0.5.0              pyhd8ed1ab_0    conda-forge
     sphinx-markdown-tables    0.0.15             pyhd3deb0d_0    conda-forge
     sphinx_rtd_theme          1.0.0              pyhd8ed1ab_0    conda-forge
     sphinxcontrib-applehelp   1.0.2                      py_0    conda-forge
     sphinxcontrib-devhelp     1.0.2                      py_0    conda-forge
     sphinxcontrib-htmlhelp    2.0.0              pyhd8ed1ab_0    conda-forge
     sphinxcontrib-jsmath      1.0.1                      py_0    conda-forge
     sphinxcontrib-qthelp      1.0.3                      py_0    conda-forge
     sphinxcontrib-serializinghtml 1.1.5              pyhd8ed1ab_1    conda-forge
     sphinxcontrib-websupport  1.2.4              pyhd8ed1ab_1    conda-forge
     sqlite                    3.37.1               h4ff8645_0    conda-forge
     sshpubkeys                3.1.0                      py_0    conda-forge
     statsmodels               0.13.2           py39hce5d2b2_0    conda-forge
     streamz                   0.6.3              pyh6c4a22f_0    conda-forge
     sysroot_linux-64          2.12                he073ed8_15    conda-forge
     tbb                       2021.5.0             h4bd325d_0    conda-forge
     tblib                     1.7.0              pyhd8ed1ab_0    conda-forge
     terminado                 0.13.3           py39hf3d152e_0    conda-forge
     testpath                  0.6.0              pyhd8ed1ab_0    conda-forge
     threadpoolctl             3.1.0              pyh8a188c0_0    conda-forge
     tifffile                  2021.11.2          pyhd8ed1ab_0    conda-forge
     tiledb                    2.3.4                he87e0bf_0    conda-forge
     tk                        8.6.12               h27826a3_0    conda-forge
     tokenizers                0.10.3           py39hd6d55de_1    conda-forge
     toml                      0.10.2             pyhd8ed1ab_0    conda-forge
     tomli                     2.0.1              pyhd8ed1ab_0    conda-forge
     toolz                     0.11.2             pyhd8ed1ab_0    conda-forge
     tornado                   6.1              py39h3811e60_2    conda-forge
     tqdm                      4.63.1             pyhd8ed1ab_0    conda-forge
     traitlets                 5.1.1              pyhd8ed1ab_0    conda-forge
     transformers              4.6.1              pyhd8ed1ab_0    conda-forge
     treelite                  2.3.0            py39heaea588_0    conda-forge
     treelite-runtime          2.3.0                    pypi_0    pypi
     twine                     3.8.0              pyhd8ed1ab_0    conda-forge
     typed-ast                 1.4.3            py39h3811e60_1    conda-forge
     typing-extensions         4.1.1                hd8ed1ab_0    conda-forge
     typing_extensions         4.1.1              pyha770c72_0    conda-forge
     tzcode                    2022a                h166bdaf_0    conda-forge
     tzdata                    2022a                h191b570_0    conda-forge
     ucx                       1.12.0+gd367332      cuda11.2_0    rapidsai-nightly
     ucx-proc                  1.0.0                       gpu    rapidsai-nightly
     ucx-py                    0.25.00a220328  py39_gd367332_13    rapidsai-nightly
     ukkonen                   1.0.1            py39h1a9c180_1    conda-forge
     umap-learn                0.5.2            py39hf3d152e_1    conda-forge
     unicodedata2              14.0.0           py39h3811e60_0    conda-forge
     urllib3                   1.26.9             pyhd8ed1ab_0    conda-forge
     virtualenv                20.14.0          py39hf3d152e_0    conda-forge
     wcwidth                   0.2.5              pyh9f0ad1d_2    conda-forge
     webencodings              0.5.1                      py_1    conda-forge
     websocket-client          1.3.1              pyhd8ed1ab_0    conda-forge
     websockets                10.2             py39hb9d737c_0    conda-forge
     werkzeug                  2.0.3              pyhd8ed1ab_1    conda-forge
     wheel                     0.37.1             pyhd8ed1ab_0    conda-forge
     widgetsnbextension        3.6.0            py39hf3d152e_0    conda-forge
     wrapt                     1.14.0           py39hb9d737c_0    conda-forge
     xarray                    2022.3.0           pyhd8ed1ab_0    conda-forge
     xerces-c                  3.2.3                h9d8b166_3    conda-forge
     xgboost                   1.5.2                    pypi_0    pypi
     xmltodict                 0.12.0                     py_0    conda-forge
     xorg-kbproto              1.0.7             h7f98852_1002    conda-forge
     xorg-libice               1.0.10               h7f98852_0    conda-forge
     xorg-libsm                1.2.3             hd9c2040_1000    conda-forge
     xorg-libx11               1.7.2                h7f98852_0    conda-forge
     xorg-libxau               1.0.9                h7f98852_0    conda-forge
     xorg-libxdmcp             1.1.3                h7f98852_0    conda-forge
     xorg-libxext              1.3.4                h7f98852_1    conda-forge
     xorg-libxrender           0.9.10            h7f98852_1003    conda-forge
     xorg-renderproto          0.11.1            h7f98852_1002    conda-forge
     xorg-xextproto            7.3.0             h7f98852_1002    conda-forge
     xorg-xproto               7.0.31            h7f98852_1007    conda-forge
     xz                        5.2.5                h516909a_1    conda-forge
     yaml                      0.2.5                h7f98852_2    conda-forge
     yarl                      1.7.2            py39h3811e60_1    conda-forge
     zeromq                    4.3.4                h9c3ff4c_1    conda-forge
     zfp                       0.5.5                h9c3ff4c_8    conda-forge
     zict                      2.1.0              pyhd8ed1ab_0    conda-forge
     zipp                      3.7.0              pyhd8ed1ab_1    conda-forge
     zlib                      1.2.11            h166bdaf_1014    conda-forge
     zstd                      1.5.2                ha95c52a_0    conda-forge

github-actions[bot] commented 2 years ago

This issue has been labeled inactive-30d due to no recent activity in the past 30 days. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed. This issue will be labeled inactive-90d if there is no activity in the next 60 days.

github-actions[bot] commented 1 year ago

This issue has been labeled inactive-90d due to no recent activity in the past 90 days. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed.

GregoryKimball commented 11 months ago

I landed back on this issue while looking for UVM discussions in cuDF. How could we get an OOM error if UVM (Unified Virtual Memory) is enabled and working properly?

I've never seen an OOM with UVM, just full system hangs when host memory is exhausted.

Perhaps the original issue came out of UVM not working properly rather than a cuDF issues. @charlesbluca would you please share any updates since your original filing?

charlesbluca commented 11 months ago

Sorry for the delay, went ahead and retried the script with updated nightlies and this still seems to be an issue:

Conda environment:

```
○ → conda list
List of packages in environment: "/home/charlesb/micromamba/envs/cudf-23.10"

  Name                           Version       Build                               Channel         
─────────────────────────────────────────────────────────────────────────────────────────────────────
  _libgcc_mutex                  0.1           conda_forge                         conda-forge     
  _openmp_mutex                  4.5           2_gnu                               conda-forge     
  asttokens                      2.2.1         pyhd8ed1ab_0                        conda-forge     
  aws-c-auth                     0.7.0         hf8751d9_2                          conda-forge     
  aws-c-cal                      0.6.0         h93469e0_0                          conda-forge     
  aws-c-common                   0.8.23        hd590300_0                          conda-forge     
  aws-c-compression              0.2.17        h862ab75_1                          conda-forge     
  aws-c-event-stream             0.3.1         h9599702_1                          conda-forge     
  aws-c-http                     0.7.11        hbe98c3e_0                          conda-forge     
  aws-c-io                       0.13.28       h3870b5a_0                          conda-forge     
  aws-c-mqtt                     0.8.14        h2e270ba_2                          conda-forge     
  aws-c-s3                       0.3.13        heb0bb06_2                          conda-forge     
  aws-c-sdkutils                 0.1.11        h862ab75_1                          conda-forge     
  aws-checksums                  0.1.16        h862ab75_1                          conda-forge     
  aws-crt-cpp                    0.20.3        he9c0e7f_4                          conda-forge     
  aws-sdk-cpp                    1.10.57       hbc2ea52_17                         conda-forge     
  backcall                       0.2.0         pyh9f0ad1d_0                        conda-forge     
  backports                      1.0           pyhd8ed1ab_3                        conda-forge     
  backports.functools_lru_cache  1.6.5         pyhd8ed1ab_0                        conda-forge     
  bzip2                          1.0.8         h7f98852_4                          conda-forge     
  c-ares                         1.19.1        hd590300_0                          conda-forge     
  ca-certificates                2023.7.22     hbcca054_0                          conda-forge     
  cachetools                     5.3.1         pyhd8ed1ab_0                        conda-forge     
  cubinlinker                    0.3.0         py310hfdf336d_0                     rapidsai-nightly
  cuda-python                    11.8.2        py310h01a121a_0                     conda-forge     
  cuda-version                   11.8          h70ddcb2_2                          conda-forge     
  cudatoolkit                    11.8.0        h4ba93d1_12                         conda-forge     
  cudf                           23.10.00a     cuda11_py310_230807_ge92de8113d_60  rapidsai-nightly
  cupy                           12.1.0        py310h53f8385_1                     conda-forge     
  decorator                      5.1.1         pyhd8ed1ab_0                        conda-forge     
  dlpack                         0.5           h9c3ff4c_0                          conda-forge     
  executing                      1.2.0         pyhd8ed1ab_0                        conda-forge     
  fastrlock                      0.8           py310hd8f1fbe_3                     conda-forge     
  fmt                            9.1.0         h924138e_0                          conda-forge     
  fsspec                         2023.6.0      pyh1a96a4e_0                        conda-forge     
  gflags                         2.2.2         he1b5a44_1004                       conda-forge     
  glog                           0.6.0         h6f12383_0                          conda-forge     
  gmock                          1.14.0        ha770c72_0                          conda-forge     
  gtest                          1.14.0        h00ab1b0_0                          conda-forge     
  ipython                        8.14.0        pyh41d4057_0                        conda-forge     
  jedi                           0.19.0        pyhd8ed1ab_0                        conda-forge     
  keyutils                       1.6.1         h166bdaf_0                          conda-forge     
  krb5                           1.21.1        h659d440_0                          conda-forge     
  ld_impl_linux-64               2.40          h41732ed_0                          conda-forge     
  libabseil                      20230125.3    cxx17_h59595ed_0                    conda-forge     
  libarrow                       12.0.1        h657c46f_7_cpu                      conda-forge     
  libblas                        3.9.0         17_linux64_openblas                 conda-forge     
  libbrotlicommon                1.0.9         h166bdaf_9                          conda-forge     
  libbrotlidec                   1.0.9         h166bdaf_9                          conda-forge     
  libbrotlienc                   1.0.9         h166bdaf_9                          conda-forge     
  libcblas                       3.9.0         17_linux64_openblas                 conda-forge     
  libcrc32c                      1.1.2         h9c3ff4c_0                          conda-forge     
  libcudf                        23.10.00a     cuda11_230807_ge92de8113d_60        rapidsai-nightly
  libcufile                      1.4.0.31      0                                   nvidia          
  libcufile-dev                  1.4.0.31      0                                   nvidia          
  libcurl                        8.2.1         hca28451_0                          conda-forge     
  libedit                        3.1.20191231  he28a2e2_2                          conda-forge     
  libev                          4.33          h516909a_1                          conda-forge     
  libevent                       2.1.12        hf998b51_1                          conda-forge     
  libffi                         3.4.2         h7f98852_5                          conda-forge     
  libgcc-ng                      13.1.0        he5830b7_0                          conda-forge     
  libgfortran-ng                 13.1.0        h69a702a_0                          conda-forge     
  libgfortran5                   13.1.0        h15d22d2_0                          conda-forge     
  libgomp                        13.1.0        he5830b7_0                          conda-forge     
  libgoogle-cloud                2.12.0        h840a212_1                          conda-forge     
  libgrpc                        1.56.2        h3905398_0                          conda-forge     
  libkvikio                      23.10.00a     cuda11_230807_g0247ca6_6            rapidsai-nightly
  liblapack                      3.9.0         17_linux64_openblas                 conda-forge     
  libllvm14                      14.0.6        hcd5def8_4                          conda-forge     
  libnghttp2                     1.52.0        h61bc06f_0                          conda-forge     
  libnsl                         2.0.0         h7f98852_0                          conda-forge     
  libnuma                        2.0.16        h0b41bf4_1                          conda-forge     
  libopenblas                    0.3.23        pthreads_h80387f5_0                 conda-forge     
  libprotobuf                    4.23.3        hd1fb520_0                          conda-forge     
  librmm                         23.10.00a     cuda11_230807_gcd37245e_9           rapidsai-nightly
  libsqlite                      3.42.0        h2797004_0                          conda-forge     
  libssh2                        1.11.0        h0841786_0                          conda-forge     
  libstdcxx-ng                   13.1.0        hfd8a6a1_0                          conda-forge     
  libthrift                      0.18.1        h8fd135c_2                          conda-forge     
  libutf8proc                    2.8.0         h166bdaf_0                          conda-forge     
  libuuid                        2.38.1        h0b41bf4_0                          conda-forge     
  libzlib                        1.2.13        hd590300_5                          conda-forge     
  llvmlite                       0.40.1        py310h1b8f574_0                     conda-forge     
  lz4-c                          1.9.4         hcb278e6_0                          conda-forge     
  matplotlib-inline              0.1.6         pyhd8ed1ab_0                        conda-forge     
  ncurses                        6.4           hcb278e6_0                          conda-forge     
  numba                          0.57.1        py310h0f6aa51_0                     conda-forge     
  numpy                          1.24.4        py310ha4c1d20_0                     conda-forge     
  nvcomp                         2.6.1         h0800d71_2                          conda-forge     
  nvtx                           0.2.5         py310h1fa729e_0                     conda-forge     
  openssl                        3.1.2         hd590300_0                          conda-forge     
  orc                            1.9.0         h385abfd_1                          conda-forge     
  packaging                      23.1          pyhd8ed1ab_0                        conda-forge     
  pandas                         1.5.3         py310h9b08913_1                     conda-forge     
  parso                          0.8.3         pyhd8ed1ab_0                        conda-forge     
  pexpect                        4.8.0         pyh1a96a4e_2                        conda-forge     
  pickleshare                    0.7.5         py_1003                             conda-forge     
  pip                            23.2.1        pyhd8ed1ab_0                        conda-forge     
  prompt-toolkit                 3.0.39        pyha770c72_0                        conda-forge     
  prompt_toolkit                 3.0.39        hd8ed1ab_0                          conda-forge     
  protobuf                       4.23.3        py310hb875b13_0                     conda-forge     
  ptxcompiler                    0.8.1         py310h01a121a_0                     conda-forge     
  ptyprocess                     0.7.0         pyhd3deb0d_0                        conda-forge     
  pure_eval                      0.2.2         pyhd8ed1ab_0                        conda-forge     
  pyarrow                        12.0.1        py310h0576679_7_cpu                 conda-forge     
  pygments                       2.16.1        pyhd8ed1ab_0                        conda-forge     
  python                         3.10.12       hd12c33a_0_cpython                  conda-forge     
  python-dateutil                2.8.2         pyhd8ed1ab_0                        conda-forge     
  python_abi                     3.10          3_cp310                             conda-forge     
  pytz                           2023.3        pyhd8ed1ab_0                        conda-forge     
  rdma-core                      28.9          h59595ed_1                          conda-forge     
  re2                            2023.03.02    h8c504da_0                          conda-forge     
  readline                       8.2           h8228510_1                          conda-forge     
  rmm                            23.10.00a     cuda11_py310_230807_gcd37245e_9     rapidsai-nightly
  s2n                            1.3.46        h06160fa_0                          conda-forge     
  setuptools                     68.0.0        pyhd8ed1ab_0                        conda-forge     
  six                            1.16.0        pyh6c4a22f_0                        conda-forge     
  snappy                         1.1.10        h9fff704_0                          conda-forge     
  spdlog                         1.11.0        h9b3ece8_1                          conda-forge     
  stack_data                     0.6.2         pyhd8ed1ab_0                        conda-forge     
  tk                             8.6.12        h27826a3_0                          conda-forge     
  traitlets                      5.9.0         pyhd8ed1ab_0                        conda-forge     
  typing_extensions              4.7.1         pyha770c72_0                        conda-forge     
  tzdata                         2023c         h71feb2d_0                          conda-forge     
  ucx                            1.14.1        h4a2ce2d_2                          conda-forge     
  wcwidth                        0.2.6         pyhd8ed1ab_0                        conda-forge     
  wheel                          0.41.1        pyhd8ed1ab_0                        conda-forge     
  xz                             5.2.6         h166bdaf_0                          conda-forge     
  zstd                           1.5.2         hfc55251_7                          conda-forge  
```

Perhaps the original issue came out of UVM not working properly rather than a cuDF issues

Any additional information I could provide here to help narrow down the cause? 🙂

vyasr commented 2 months ago

Could this be some WSL + UVM bug? CC @harrism

harrism commented 1 month ago

@charlesbluca I asked and it turns out that this is expected on WSL. The UVM support on windows display driver model (WDDM) is a limited form of UVM that doesn't support oversubscription (or simultaneous CPU / GPU access). Pages are not migrated on WDDM.

That said, there is a limited form of oversubscription that is supported for regular cudaMalloc calls. Could you do me a favor and try your script with this line commented out so that it uses the default memory resource?

rmm.mr.set_current_device_resource(rmm.mr.ManagedMemoryResource())

I kind of expect this to change or move the failure, rather than solve it, but it will be interesting to see.

jarmak-personal commented 1 month ago

Tested the script below on my WSL2 (Win 11/Ubuntu22.04) machine that has an RTX4090:

import cupy as cp
import cudf
import os

def generate_random_data(num_rows, num_columns):
    """Generate random numbers using CuPy and return a cuDF DataFrame."""
    # Generate a random CuPy array
    data = cp.random.rand(num_rows, num_columns)
    # Convert to cuDF DataFrame
    df = cudf.DataFrame(data, columns=[f'col_{i}' for i in range(num_columns)])
    return df

def generate_csv_gpu(target_size, filename='cudf_uvm_data_20gb.csv'):
    num_columns = 5
    row_estimate = 1000  # Initial guess for number of rows

    # Generate initial data
    df = generate_random_data(row_estimate, num_columns)
    df.to_csv(filename, index=False)

    # Check file size and adjust
    current_size = os.path.getsize(filename)
    row_size = current_size / row_estimate
    total_rows_needed = int(target_size / row_size)

    # Generate the correct amount of data
    df = generate_random_data(total_rows_needed, num_columns)
    df.to_csv(filename, index=False, chunksize=1000000)

    # Report final file size
    final_size = os.path.getsize(filename)
    print(f"Targeted file size was {target_size} bytes.")
    print(f"Final file size is {final_size} bytes.")

# Usage example
generate_csv_gpu(20000000000)

input_data_path ='cudf_uvm_data_20gb.csv'

df = cudf.read_csv(input_data_path)
len(df)
>>> 336309673

So no issues with a CSV that's ~18gb with an available 24gb GPU memory. In the Windows task manager GPU perf tab we can see it needs to jump into the shared memory pool, but does so successfully.

image

So - I think this has been resolved at some point in WSL.

harrism commented 1 month ago

I wonder what "shared GPU Memory" is.

harrism commented 1 month ago

Confirmed internally that this means it is over subscribing. OK so the behavior with managed memory is expected, but this option is available.

I think we can close this now.