[BUG] `cudf::strings::from_integers` does not appear to be `compute-sanitizer --tool initcheck` clean

wence- commented 1 year ago

Conversion from (at least) integer to string columns (done a lot for printing on the cudf-python side of things) appears to have uninitialized device memory accesses. Related #8873.

Consider the following:

#include <algorithm>
#include <cstdint>
#include <vector>

#include <cudf/column/column.hpp>
#include <cudf/column/column_view.hpp>
#include <cudf/strings/convert/convert_integers.hpp>
#include <cudf/types.hpp>

int main(int argc, char **argv) {
  using T = int64_t;
  using size_type = cudf::size_type;
  size_type size;
  if (argc > 1) {
    size = std::stoi(argv[1]);
  } else {
    size = 1;
  }
  std::vector<T> data{size};
  std::generate(data.begin(), data.end(), []() { return 1; });
  auto column = cudf::column{cudf::data_type{cudf::type_to_id<T>()}, size,
                             rmm::device_buffer{data.data(), size * sizeof(T),
                                                cudf::get_default_stream()}};
  cudf::get_default_stream().synchronize();
  auto string_col = cudf::strings::from_integers(
      column.view(), rmm::mr::get_current_device_resource());
  cudf::get_default_stream().synchronize();
  return 0;
}

When run as:

$ compute-sanitizer --tool initcheck ./test
========= COMPUTE-SANITIZER
========= Uninitialized __global__ memory read of size 4 bytes
=========     at 0x1c0 in void cub::CUB_101702_860_NS::DeviceScanKernel<cub::CUB_101702_860_NS::DeviceScanPolicy<long>::Policy600, int *, cudf::detail::sizes_to_offsets_iterator<int *, long>, cub::CUB_101702_860_NS::ScanTileState<long, (bool)1>, thrust::plus<void>, cub::CUB_101702_860_NS::detail::InputValue<long, long *>, int>(T2, T3, T4, int, T5, T6, T7)
=========     by thread (1,0,0) in block (0,0,0)
=========     Address 0x7f26c3e00204
=========     Saved host backtrace up to driver entry point at kernel launch time
=========     Host Frame: [0x304e32]
=========                in /usr/lib/x86_64-linux-gnu/libcuda.so.1
=========     Host Frame: [0x1488c]
=========                in /home/wence/Documents/src/rapids/compose/etc/conda/cuda_11.8/envs/rapids/lib/libcudart.so.11.0
=========     Host Frame:cudaLaunchKernel [0x6c318]
=========                in /home/wence/Documents/src/rapids/compose/etc/conda/cuda_11.8/envs/rapids/lib/libcudart.so.11.0
=========     Host Frame:cudf::detail::sizes_to_offsets_iterator<int*, long> thrust::cuda_cub::detail::exclusive_scan_n_impl<thrust::detail::execute_with_allocator<rmm::mr::thrust_allocator<char>, thrust::cuda_cub::execute_on_stream_base>, int*, long, cudf::detail::sizes_to_offsets_iterator<int*, long>, long, thrust::plus<void> >(thrust::cuda_cub::execution_policy<thrust::detail::execute_with_allocator<rmm::mr::thrust_allocator<char>, thrust::cuda_cub::execute_on_stream_base> >&, int*, long, cudf::detail::sizes_to_offsets_iterator<int*, long>, long, thrust::plus<void>) [0x161bf95]
=========                in /home/wence/Documents/src/rapids/cudf/cpp/build/libcudf.so
=========     Host Frame:auto cudf::detail::sizes_to_offsets<int*, int*>(int*, int*, int*, rmm::cuda_stream_view) [0x161c612]
=========                in /home/wence/Documents/src/rapids/cudf/cpp/build/libcudf.so
=========     Host Frame:auto cudf::strings::detail::make_strings_children<cudf::strings::detail::(anonymous namespace)::from_integers_fn<long> >(cudf::strings::detail::(anonymous namespace)::from_integers_fn<long>, int, int, rmm::cuda_stream_view, rmm::mr::device_memory_resource*) [0x264fb1f]
=========                in /home/wence/Documents/src/rapids/cudf/cpp/build/libcudf.so
=========     Host Frame:std::unique_ptr<cudf::column, std::default_delete<cudf::column> > cudf::strings::detail::(anonymous namespace)::dispatch_from_integers_fn::operator()<long, (void*)0>(cudf::column_view const&, rmm::cuda_stream_view, rmm::mr::device_memory_resource*) const [0x264feb1]
=========                in /home/wence/Documents/src/rapids/cudf/cpp/build/libcudf.so
=========     Host Frame:cudf::strings::detail::from_integers(cudf::column_view const&, rmm::cuda_stream_view, rmm::mr::device_memory_resource*) [0x264a1e9]
=========                in /home/wence/Documents/src/rapids/cudf/cpp/build/libcudf.so
=========     Host Frame:cudf::strings::from_integers(cudf::column_view const&, rmm::mr::device_memory_resource*) [0x264a2c7]
=========                in /home/wence/Documents/src/rapids/cudf/cpp/build/libcudf.so
=========     Host Frame:/home/wence/Documents/src/rapids/doodles/c++/test_from_integers.cpp:26:main [0x1ae5e]
=========                in /home/wence/Documents/src/rapids/doodles/c++/./test
=========     Host Frame:__libc_start_main [0x24083]
=========                in /usr/lib/x86_64-linux-gnu/libc.so.6
=========     Host Frame: [0x13079]
=========                in /home/wence/Documents/src/rapids/doodles/c++/./test
=========
========= ERROR SUMMARY: 1 error

This looks like an off-by-one, but my tracking through the libcudf side of things didn't spot anything, so perhaps it is a bug in thrust or CUB.

Environment overview (please complete the following information)

Environment location: [Bare-metal, Docker, Cloud(specify cloud provider)]
Method of cuDF install: [conda, Docker, or from source]
- If method of install is [Docker], provide docker pull & docker run commands used

Environment details

Click here to see environment details


     **git***
     commit 55ef6018ed92bfd6d3cbb67d9736c72aadaa2668 (HEAD -> branch-23.02)
     Author: Karthikeyan <6488848+karthikeyann@users.noreply.github.com>
     Date:   Sat Jan 28 07:09:29 2023 +0530

     Add JSON Writer (#12474)

     Adds JSON writer with nested support.
     It supports numeric, datetime, duration, strings,  nested types such as struct and list types.
     `orient='records'` is only supported now, with `lines=True/False`.
     Usage: `df.to_json(engine='cudf')`

     closes https://github.com/rapidsai/cudf/issues/11165

     Authors:
     - Karthikeyan (https://github.com/karthikeyann)

     Approvers:
     - Vukasin Milovanovic (https://github.com/vuule)
     - GALI PREM SAGAR (https://github.com/galipremsagar)
     - David Wendt (https://github.com/davidwendt)
     - Michael Wang (https://github.com/isVoid)
     - Robert Maynard (https://github.com/robertmaynard)

     URL: https://github.com/rapidsai/cudf/pull/12474
     **git submodules***

     ***OS Information***
     DISTRIB_ID=Ubuntu
     DISTRIB_RELEASE=20.04
     DISTRIB_CODENAME=focal
     DISTRIB_DESCRIPTION="Ubuntu 20.04.5 LTS"
     NAME="Ubuntu"
     VERSION="20.04.5 LTS (Focal Fossa)"
     ID=ubuntu
     ID_LIKE=debian
     PRETTY_NAME="Ubuntu 20.04.5 LTS"
     VERSION_ID="20.04"
     HOME_URL="https://www.ubuntu.com/"
     SUPPORT_URL="https://help.ubuntu.com/"
     BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
     PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
     VERSION_CODENAME=focal
     UBUNTU_CODENAME=focal
     Linux shallot 5.15.0-58-generic NVIDIA/cub#64-Ubuntu SMP Thu Jan 5 11:43:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

     ***GPU Information***
     Wed Feb  1 16:01:12 2023
     +-----------------------------------------------------------------------------+
     | NVIDIA-SMI 525.85.12    Driver Version: 525.85.12    CUDA Version: 12.0     |
     |-------------------------------+----------------------+----------------------+
     | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
     | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
     |                               |                      |               MIG M. |
     |===============================+======================+======================|
     |   0  NVIDIA RTX A6000    On   | 00000000:17:00.0 Off |                  Off |
     | 30%   31C    P8    21W / 300W |      6MiB / 49140MiB |      0%      Default |
     |                               |                      |                  N/A |
     +-------------------------------+----------------------+----------------------+
     |   1  NVIDIA RTX A6000    On   | 00000000:B3:00.0  On |                  Off |
     | 30%   43C    P3    49W / 300W |   1343MiB / 49140MiB |     13%      Default |
     |                               |                      |                  N/A |
     +-------------------------------+----------------------+----------------------+

     +-----------------------------------------------------------------------------+
     | Processes:                                                                  |
     |  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
     |        ID   ID                                                   Usage      |
     |=============================================================================|
     |    0   N/A  N/A      3089      G   /usr/lib/xorg/Xorg                  4MiB |
     |    1   N/A  N/A      3089      G   /usr/lib/xorg/Xorg                738MiB |
     |    1   N/A  N/A      3267      G   /usr/bin/gnome-shell              185MiB |
     |    1   N/A  N/A     44652      G   ...veSuggestionsOnlyOnDemand      154MiB |
     |    1   N/A  N/A     45338      G   /usr/bin/wezterm-gui               11MiB |
     +-----------------------------------------------------------------------------+

     ***CPU***
     Architecture:                    x86_64
     CPU op-mode(s):                  32-bit, 64-bit
     Byte Order:                      Little Endian
     Address sizes:                   46 bits physical, 48 bits virtual
     CPU(s):                          32
     On-line CPU(s) list:             0-31
     Thread(s) per core:              2
     Core(s) per socket:              16
     Socket(s):                       1
     NUMA node(s):                    1
     Vendor ID:                       GenuineIntel
     CPU family:                      6
     Model:                           85
     Model name:                      Intel(R) Xeon(R) Gold 6226R CPU @ 2.90GHz
     Stepping:                        7
     CPU MHz:                         2900.000
     CPU max MHz:                     3900.0000
     CPU min MHz:                     1200.0000
     BogoMIPS:                        5800.00
     Virtualization:                  VT-x
     L1d cache:                       512 KiB
     L1i cache:                       512 KiB
     L2 cache:                        16 MiB
     L3 cache:                        22 MiB
     NUMA node0 CPU(s):               0-31
     Vulnerability Itlb multihit:     KVM: Mitigation: VMX disabled
     Vulnerability L1tf:              Not affected
     Vulnerability Mds:               Not affected
     Vulnerability Meltdown:          Not affected
     Vulnerability Mmio stale data:   Mitigation; Clear CPU buffers; SMT vulnerable
     Vulnerability Retbleed:          Mitigation; Enhanced IBRS
     Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
     Vulnerability Spectre v1:        Mitigation; usercopy/swapgs barriers and __user pointer sanitization
     Vulnerability Spectre v2:        Mitigation; Enhanced IBRS, IBPB conditional, RSB filling, PBRSB-eIBRS SW sequence
     Vulnerability Srbds:             Not affected
     Vulnerability Tsx async abort:   Mitigation; TSX disabled
     Flags:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single intel_ppin ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req pku ospke avx512_vnni md_clear flush_l1d arch_capabilities

     ***CMake***
     /home/wence/Documents/src/rapids/compose/etc/conda/cuda_11.8/envs/rapids/bin/cmake
     cmake version 3.25.2

     CMake suite maintained and supported by Kitware (kitware.com/cmake).

     ***g++***
     /usr/local/sbin/g++
     g++ (conda-forge gcc 9.5.0-19) 9.5.0
     Copyright (C) 2019 Free Software Foundation, Inc.
     This is free software; see the source for copying conditions.  There is NO
     warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

     ***nvcc***
     /usr/local/sbin/nvcc
     nvcc: NVIDIA (R) Cuda compiler driver
     Copyright (c) 2005-2022 NVIDIA Corporation
     Built on Wed_Sep_21_10:33:58_PDT_2022
     Cuda compilation tools, release 11.8, V11.8.89
     Build cuda_11.8.r11.8/compiler.31833905_0

     ***Python***
     /home/wence/Documents/src/rapids/compose/etc/conda/cuda_11.8/envs/rapids/bin/python
     Python 3.8.15

     ***Environment Variables***
     PATH                            : /usr/local/sbin:/usr/local/bin:/home/wence/Documents/src/rapids/compose/etc/conda/cuda_11.8/envs/rapids/bin:/home/wence/Documents/src/rapids/compose/etc/conda/cuda_11.8/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/cuda/bin
     LD_LIBRARY_PATH                 : /home/wence/Documents/src/rapids/compose/etc/conda/cuda_11.8/envs/rapids/lib:/home/wence/Documents/src/rapids/compose/etc/conda/cuda_11.8/lib:/usr/local/nvidia/lib:/usr/local/nvidia/lib64:/usr/local/cuda/lib64:/usr/local/lib:/home/wence/Documents/src/rapids/rmm/build/release:/home/wence/Documents/src/rapids/cudf/cpp/build/release:/home/wence/Documents/src/rapids/raft/cpp/build/release:/home/wence/Documents/src/rapids/cuml/cpp/build/release:/home/wence/Documents/src/rapids/cugraph/cpp/build/release:/home/wence/Documents/src/rapids/cuspatial/cpp/build/release
     NUMBAPRO_NVVM                   :
     NUMBAPRO_LIBDEVICE              :
     CONDA_PREFIX                    : /home/wence/Documents/src/rapids/compose/etc/conda/cuda_11.8/envs/rapids
     PYTHON_PATH                     :

     ***conda packages***
     /home/wence/Documents/src/rapids/compose/etc/conda/cuda_11.8/bin/conda

     # packages in environment at /home/wence/Documents/src/rapids/compose/etc/conda/cuda_11.8/envs/rapids:
     #
     # Name                    Version                   Build  Channel
     _libgcc_mutex             0.1                 conda_forge    conda-forge
     _openmp_mutex             4.5                  2_kmp_llvm    conda-forge
     _sysroot_linux-64_curr_repodata_hack 3                   h5bd9786_13    conda-forge
     aiobotocore               2.4.2              pyhd8ed1ab_0    conda-forge
     aiohttp                   3.8.3            py38h0a891b7_1    conda-forge
     aioitertools              0.11.0             pyhd8ed1ab_0    conda-forge
     aiosignal                 1.3.1              pyhd8ed1ab_0    conda-forge
     alabaster                 0.7.13             pyhd8ed1ab_0    conda-forge
     anyio                     3.6.2              pyhd8ed1ab_0    conda-forge
     appdirs                   1.4.4              pyh9f0ad1d_0    conda-forge
     argon2-cffi               21.3.0             pyhd8ed1ab_0    conda-forge
     argon2-cffi-bindings      21.2.0           py38h0a891b7_3    conda-forge
     arrow-cpp                 10.0.1           ha770c72_6_cpu    conda-forge
     asttokens                 2.2.1              pyhd8ed1ab_0    conda-forge
     async-timeout             4.0.2              pyhd8ed1ab_0    conda-forge
     attrs                     22.2.0             pyh71513ae_0    conda-forge
     aws-c-auth                0.6.21               hd93a3ba_3    conda-forge
     aws-c-cal                 0.5.20               hff2c3d7_3    conda-forge
     aws-c-common              0.8.5                h166bdaf_0    conda-forge
     aws-c-compression         0.2.16               hf5f93bc_0    conda-forge
     aws-c-event-stream        0.2.18               h57874a7_0    conda-forge
     aws-c-http                0.7.0                h96ef541_0    conda-forge
     aws-c-io                  0.13.12              h57ca295_1    conda-forge
     aws-c-mqtt                0.7.13              h0b5698f_12    conda-forge
     aws-c-s3                  0.2.3                h82cbbf9_0    conda-forge
     aws-c-sdkutils            0.1.7                hf5f93bc_0    conda-forge
     aws-checksums             0.1.14               h6027aba_0    conda-forge
     aws-crt-cpp               0.18.16             hf80f573_10    conda-forge
     aws-sam-translator        1.58.1             pyhd8ed1ab_0    conda-forge
     aws-sdk-cpp               1.10.57              ha834a50_1    conda-forge
     aws-xray-sdk              2.11.0             pyhd8ed1ab_0    conda-forge
     babel                     2.11.0             pyhd8ed1ab_0    conda-forge
     backcall                  0.2.0              pyh9f0ad1d_0    conda-forge
     backports                 1.0                pyhd8ed1ab_3    conda-forge
     backports.functools_lru_cache 1.6.4              pyhd8ed1ab_0    conda-forge
     backports.zoneinfo        0.2.1            py38h0a891b7_7    conda-forge
     bcrypt                    3.2.2            py38h0a891b7_1    conda-forge
     beautifulsoup4            4.11.1             pyha770c72_0    conda-forge
     binutils                  2.39                 hdd6e379_1    conda-forge
     binutils_impl_linux-64    2.39                 he00db2b_1    conda-forge
     binutils_linux-64         2.39                h5fc0e48_11    conda-forge
     blas                      1.0                         mkl    conda-forge
     bleach                    6.0.0              pyhd8ed1ab_0    conda-forge
     blosc                     1.21.3               hafa529b_0    conda-forge
     bokeh                     2.4.3              pyhd8ed1ab_3    conda-forge
     boost-cpp                 1.78.0               h75c5d50_1    conda-forge
     boto3                     1.24.59            pyhd8ed1ab_0    conda-forge
     botocore                  1.27.59            pyhd8ed1ab_0    conda-forge
     branca                    0.6.0              pyhd8ed1ab_0    conda-forge
     breathe                   4.34.0             pyhd8ed1ab_0    conda-forge
     brotli                    1.0.9                h166bdaf_8    conda-forge
     brotli-bin                1.0.9                h166bdaf_8    conda-forge
     brotlipy                  0.7.0           py38h0a891b7_1005    conda-forge
     bzip2                     1.0.8                h7f98852_4    conda-forge
     c-ares                    1.18.1               h7f98852_0    conda-forge
     c-compiler                1.3.0                h7f98852_0    conda-forge
     ca-certificates           2022.12.7            ha878542_0    conda-forge
     cachetools                5.3.0              pyhd8ed1ab_0    conda-forge
     cairo                     1.16.0            ha61ee94_1014    conda-forge
     certifi                   2022.12.7          pyhd8ed1ab_0    conda-forge
     cffi                      1.15.1           py38h4a40e3a_3    conda-forge
     cfgv                      3.3.1              pyhd8ed1ab_0    conda-forge
     cfitsio                   4.2.0                hd9d235c_0    conda-forge
     cfn-lint                  0.24.8                   py38_0    conda-forge
     charset-normalizer        2.1.1              pyhd8ed1ab_0    conda-forge
     clang                     11.1.0               ha770c72_1    conda-forge
     clang-11                  11.1.0          default_ha53f305_1    conda-forge
     clang-tools               11.1.0          default_ha53f305_1    conda-forge
     clangxx                   11.1.0          default_ha53f305_1    conda-forge
     click                     8.1.3           unix_pyhd8ed1ab_2    conda-forge
     click-plugins             1.1.1                      py_0    conda-forge
     cligj                     0.7.2              pyhd8ed1ab_1    conda-forge
     cloudpickle               2.2.1              pyhd8ed1ab_0    conda-forge
     cmake                     3.25.2               h077f3f9_0    conda-forge
     cmake_setuptools          0.1.3                      py_0    rapidsai
     colorama                  0.4.6              pyhd8ed1ab_0    conda-forge
     comm                      0.1.2              pyhd8ed1ab_0    conda-forge
     commonmark                0.9.1                      py_0    conda-forge
     contourpy                 1.0.7            py38hfbd4bf9_0    conda-forge
     coverage                  7.1.0            py38h1de0b5d_0    conda-forge
     cryptography              39.0.0           py38h3d167d9_0    conda-forge
     cubinlinker               0.2.2            py38h7144610_0    rapidsai
     cuda-profiler-api         11.8.86                       0    nvidia
     cuda-python               11.8.1           py38h241159d_2    conda-forge
     cudatoolkit               11.8.0              h37601d7_11    conda-forge
     cupy                      11.5.0           py38h405e1b6_0    conda-forge
     curl                      7.87.0               hdc1c0ab_0    conda-forge
     cxx-compiler              1.3.0                h4bd325d_0    conda-forge
     cycler                    0.11.0             pyhd8ed1ab_0    conda-forge
     cyrus-sasl                2.1.27               h9033bb2_6    conda-forge
     cython                    0.29.33          py38h8dc9893_0    conda-forge
     cytoolz                   0.12.0           py38h0a891b7_1    conda-forge
     dask                      2023.1.0+17.g2a2b9d3c           dev_0    
     dask-core                 2023.1.1a230127 py_g185eed2c9_23    dask/label/dev
     dask-cuda                 23.2.0a0+41.g66a6a46.dirty          pypi_0    pypi
     dask-glm                  0.2.1.dev52+g1daf4c5          pypi_0    pypi
     dask-ml                   2022.5.27          pyhd8ed1ab_0    conda-forge
     dataclasses               0.8                pyhc8e2a94_3    conda-forge
     datasets                  1.18.3             pyhd8ed1ab_0    conda-forge
     debugpy                   1.6.6            py38h8dc9893_0    conda-forge
     decopatch                 1.4.10             pyhd8ed1ab_0    conda-forge
     decorator                 5.1.1              pyhd8ed1ab_0    conda-forge
     defusedxml                0.7.1              pyhd8ed1ab_0    conda-forge
     dill                      0.3.6              pyhd8ed1ab_1    conda-forge
     distlib                   0.3.6              pyhd8ed1ab_0    conda-forge
     distributed               2023.1.0+9.g0161991f.dirty           dev_0    
     distro                    1.8.0              pyhd8ed1ab_0    conda-forge
     dlpack                    0.5                  h9c3ff4c_0    conda-forge
     docker-py                 6.0.0              pyhd8ed1ab_0    conda-forge
     docutils                  0.19             py38h578d9bd_1    conda-forge
     doxygen                   1.8.20               had0d8f1_0    conda-forge
     ecdsa                     0.18.0             pyhd8ed1ab_1    conda-forge
     entrypoints               0.4                pyhd8ed1ab_0    conda-forge
     exceptiongroup            1.1.0              pyhd8ed1ab_0    conda-forge
     execnet                   1.9.0              pyhd8ed1ab_0    conda-forge
     executing                 1.2.0              pyhd8ed1ab_0    conda-forge
     expat                     2.5.0                h27087fc_0    conda-forge
     faiss-proc                1.0.0                      cuda    rapidsai
     fastavro                  1.7.1            py38h1de0b5d_0    conda-forge
     fastrlock                 0.8              py38hfa26641_3    conda-forge
     filelock                  3.9.0              pyhd8ed1ab_0    conda-forge
     fiona                     1.8.22           py38hc72d8cd_2    conda-forge
     flask                     2.1.3              pyhd8ed1ab_0    conda-forge
     flask_cors                3.0.10             pyhd3deb0d_0    conda-forge
     flit-core                 3.8.0              pyhd8ed1ab_0    conda-forge
     folium                    0.14.0             pyhd8ed1ab_0    conda-forge
     font-ttf-dejavu-sans-mono 2.37                 hab24e00_0    conda-forge
     font-ttf-inconsolata      3.000                h77eed37_0    conda-forge
     font-ttf-source-code-pro  2.038                h77eed37_0    conda-forge
     font-ttf-ubuntu           0.83                 hab24e00_0    conda-forge
     fontconfig                2.14.2               h14ed4e7_0    conda-forge
     fonts-conda-ecosystem     1                             0    conda-forge
     fonts-conda-forge         1                             0    conda-forge
     fonttools                 4.38.0           py38h0a891b7_1    conda-forge
     freetype                  2.12.1               hca18f0e_1    conda-forge
     freexl                    1.0.6                h166bdaf_1    conda-forge
     frozenlist                1.3.3            py38h0a891b7_0    conda-forge
     fsspec                    2023.1.0           pyhd8ed1ab_0    conda-forge
     future                    0.18.3             pyhd8ed1ab_0    conda-forge
     gcc                       9.5.0               h1fea6ba_11    conda-forge
     gcc_impl_linux-64         9.5.0               h99780fb_19    conda-forge
     gcc_linux-64              9.5.0               h4258300_11    conda-forge
     gcovr                     5.1                pyhd8ed1ab_0    conda-forge
     gdal                      3.5.3           py38h58634bd_15    conda-forge
     geopandas                 0.12.2             pyhd8ed1ab_0    conda-forge
     geopandas-base            0.12.2             pyha770c72_0    conda-forge
     geos                      3.11.1               h27087fc_0    conda-forge
     geotiff                   1.7.1                h7a142b4_6    conda-forge
     gettext                   0.21.1               h27087fc_0    conda-forge
     gflags                    2.2.2             he1b5a44_1004    conda-forge
     giflib                    5.2.1                h36c2ea0_2    conda-forge
     glog                      0.6.0                h6f12383_0    conda-forge
     gmock                     1.10.0               h4bd325d_7    conda-forge
     gmp                       6.2.1                h58526e2_0    conda-forge
     gmpy2                     2.1.2            py38h793c122_1    conda-forge
     graphql-core              3.2.3              pyhd8ed1ab_0    conda-forge
     greenlet                  2.0.2            py38h8dc9893_0    conda-forge
     gtest                     1.10.0               h4bd325d_7    conda-forge
     gxx                       9.5.0               h1fea6ba_11    conda-forge
     gxx_impl_linux-64         9.5.0               h99780fb_19    conda-forge
     gxx_linux-64              9.5.0               h43f449f_11    conda-forge
     hdbscan                   0.8.29           py38h26c90d9_1    conda-forge
     hdf4                      4.2.15               h9772cbc_5    conda-forge
     hdf5                      1.12.2          nompi_h4df4325_101    conda-forge
     heapdict                  1.0.1                      py_0    conda-forge
     huggingface_hub           0.12.0             pyhd8ed1ab_0    conda-forge
     hypothesis                6.65.2             pyha770c72_0    conda-forge
     icu                       70.1                 h27087fc_0    conda-forge
     identify                  2.5.17             pyhd8ed1ab_0    conda-forge
     idna                      3.4                pyhd8ed1ab_0    conda-forge
     imagesize                 1.4.1              pyhd8ed1ab_0    conda-forge
     importlib-metadata        6.0.0              pyha770c72_0    conda-forge
     importlib_metadata        6.0.0                hd8ed1ab_0    conda-forge
     iniconfig                 2.0.0              pyhd8ed1ab_0    conda-forge
     ipykernel                 6.21.0             pyh210e3f2_0    conda-forge
     ipython                   8.9.0              pyh41d4057_0    conda-forge
     ipython_genutils          0.2.0                      py_1    conda-forge
     itsdangerous              2.1.2              pyhd8ed1ab_0    conda-forge
     jedi                      0.18.2             pyhd8ed1ab_0    conda-forge
     jinja2                    3.1.2              pyhd8ed1ab_1    conda-forge
     jmespath                  1.0.1              pyhd8ed1ab_0    conda-forge
     joblib                    1.2.0              pyhd8ed1ab_0    conda-forge
     jpeg                      9e                   h166bdaf_2    conda-forge
     json-c                    0.16                 hc379101_0    conda-forge
     jsondiff                  2.0.0              pyhd8ed1ab_0    conda-forge
     jsonpatch                 1.32               pyhd8ed1ab_0    conda-forge
     jsonpointer               2.0                        py_0    conda-forge
     jsonschema                3.2.0              pyhd8ed1ab_3    conda-forge
     jupyter-cache             0.5.0              pyhd8ed1ab_0    conda-forge
     jupyter_client            8.0.1              pyhd8ed1ab_0    conda-forge
     jupyter_core              5.2.0            py38h578d9bd_0    conda-forge
     jupyter_events            0.6.3              pyhd8ed1ab_0    conda-forge
     jupyter_server            2.1.0              pyhd8ed1ab_0    conda-forge
     jupyter_server_terminals  0.4.4              pyhd8ed1ab_1    conda-forge
     jupyterlab_pygments       0.2.2              pyhd8ed1ab_0    conda-forge
     kealib                    1.5.0                ha7026e8_0    conda-forge
     kernel-headers_linux-64   3.10.0              h4a8ded7_13    conda-forge
     keyutils                  1.6.1                h166bdaf_0    conda-forge
     kiwisolver                1.4.4            py38h43d8883_1    conda-forge
     krb5                      1.20.1               h81ceb04_0    conda-forge
     lcms2                     2.14                 hfd0df8a_1    conda-forge
     ld_impl_linux-64          2.39                 hcc3a1bd_1    conda-forge
     lerc                      4.0.0                h27087fc_0    conda-forge
     libabseil                 20220623.0      cxx17_h05df665_6    conda-forge
     libaec                    1.0.6                hcb278e6_1    conda-forge
     libarrow                  10.0.1           hf9c26a6_6_cpu    conda-forge
     libblas                   3.9.0            16_linux64_mkl    conda-forge
     libbrotlicommon           1.0.9                h166bdaf_8    conda-forge
     libbrotlidec              1.0.9                h166bdaf_8    conda-forge
     libbrotlienc              1.0.9                h166bdaf_8    conda-forge
     libcblas                  3.9.0            16_linux64_mkl    conda-forge
     libclang-cpp11.1          11.1.0          default_ha53f305_1    conda-forge
     libcrc32c                 1.1.2                h9c3ff4c_0    conda-forge
     libcublas                 11.11.3.6                     0    nvidia
     libcublas-dev             11.11.3.6                     0    nvidia
     libcufft                  10.9.0.58                     0    nvidia
     libcufft-dev              10.9.0.58                     0    nvidia
     libcumlprims              23.02.00a230126 cuda11_ge28945f_11    rapidsai-nightly
     libcurand                 10.3.0.86                     0    nvidia
     libcurand-dev             10.3.0.86                     0    nvidia
     libcurl                   7.87.0               hdc1c0ab_0    conda-forge
     libcusolver               11.4.1.48                     0    nvidia
     libcusolver-dev           11.4.1.48                     0    nvidia
     libcusparse               11.7.5.86                     0    nvidia
     libcusparse-dev           11.7.5.86                     0    nvidia
     libdap4                   3.20.6               hd7c4107_2    conda-forge
     libdeflate                1.17                 h0b41bf4_0    conda-forge
     libedit                   3.1.20191231         he28a2e2_2    conda-forge
     libev                     4.33                 h516909a_1    conda-forge
     libevent                  2.1.10               h28343ad_4    conda-forge
     libfaiss                  1.7.2           cuda118h2d43ea4_4_cuda    rapidsai
     libffi                    3.4.2                h7f98852_5    conda-forge
     libgcc-devel_linux-64     9.5.0               h0a57e50_19    conda-forge
     libgcc-ng                 12.2.0              h65d4601_19    conda-forge
     libgcrypt                 1.10.1               h166bdaf_0    conda-forge
     libgdal                   3.5.3               h084b287_15    conda-forge
     libgfortran-ng            12.2.0              h69a702a_19    conda-forge
     libgfortran5              12.2.0              h337968e_19    conda-forge
     libglib                   2.74.1               h606061b_1    conda-forge
     libgomp                   12.2.0              h65d4601_19    conda-forge
     libgoogle-cloud           2.5.0                h21dfe5b_1    conda-forge
     libgpg-error              1.46                 h620e276_0    conda-forge
     libgrpc                   1.51.1               h30feacc_0    conda-forge
     libgsasl                  1.8.0                         2    conda-forge
     libhwloc                  2.8.0                h32351e8_1    conda-forge
     libiconv                  1.17                 h166bdaf_0    conda-forge
     libjpeg-turbo             2.1.4                h166bdaf_0    conda-forge
     libkml                    1.3.0             h37653c0_1015    conda-forge
     liblapack                 3.9.0            16_linux64_mkl    conda-forge
     libllvm11                 11.1.0               he0ac6c6_5    conda-forge
     libnetcdf                 4.8.1           nompi_h261ec11_106    conda-forge
     libnghttp2                1.51.0               hff17c54_0    conda-forge
     libnsl                    2.0.0                h7f98852_0    conda-forge
     libntlm                   1.4               h7f98852_1002    conda-forge
     libpng                    1.6.39               h753d276_0    conda-forge
     libpq                     15.1                 hb675445_3    conda-forge
     libprotobuf               3.21.12              h3eb15da_0    conda-forge
     librdkafka                1.7.0                hb1989a6_1    conda-forge
     librttopo                 1.1.0               ha49c73b_12    conda-forge
     libsanitizer              9.5.0               h2f262e1_19    conda-forge
     libsodium                 1.0.18               h36c2ea0_1    conda-forge
     libspatialindex           1.9.3                h9c3ff4c_4    conda-forge
     libspatialite             5.0.1               h221c8f1_23    conda-forge
     libsqlite                 3.40.0               h753d276_0    conda-forge
     libssh2                   1.10.0               hf14f497_3    conda-forge
     libstdcxx-devel_linux-64  9.5.0               h0a57e50_19    conda-forge
     libstdcxx-ng              12.2.0              h46fd767_19    conda-forge
     libthrift                 0.16.0               he500d00_2    conda-forge
     libtiff                   4.5.0                h6adf6a1_2    conda-forge
     libutf8proc               2.8.0                h166bdaf_0    conda-forge
     libuuid                   2.32.1            h7f98852_1000    conda-forge
     libuv                     1.44.2               h166bdaf_0    conda-forge
     libwebp-base              1.2.4                h166bdaf_0    conda-forge
     libxcb                    1.13              h7f98852_1004    conda-forge
     libxml2                   2.10.3               h7463322_0    conda-forge
     libxslt                   1.1.37               h873f0b0_0    conda-forge
     libzip                    1.9.2                hc929e4a_1    conda-forge
     libzlib                   1.2.13               h166bdaf_4    conda-forge
     littleutils               0.2.2                      py_0    conda-forge
     livereload                2.6.3              pyh9f0ad1d_0    conda-forge
     llvm-openmp               15.0.7               h0cdce71_0    conda-forge
     llvmlite                  0.39.1           py38h38d86a4_1    conda-forge
     locket                    1.0.0              pyhd8ed1ab_0    conda-forge
     lxml                      4.9.2            py38h215a2d7_0    conda-forge
     lz4                       4.2.0            py38hd012fdc_0    conda-forge
     lz4-c                     1.9.3                h9c3ff4c_1    conda-forge
     make                      4.3                  hd18ef5c_1    conda-forge
     makefun                   1.15.0             pyhd8ed1ab_0    conda-forge
     mapclassify               2.5.0              pyhd8ed1ab_1    conda-forge
     markdown                  3.4.1              pyhd8ed1ab_0    conda-forge
     markdown-it-py            2.1.0              pyhd8ed1ab_0    conda-forge
     markupsafe                2.1.2            py38h1de0b5d_0    conda-forge
     matplotlib-base           3.6.3            py38hd6c3c57_0    conda-forge
     matplotlib-inline         0.1.6              pyhd8ed1ab_0    conda-forge
     mdit-py-plugins           0.3.3              pyhd8ed1ab_0    conda-forge
     mdurl                     0.1.0              pyhd8ed1ab_0    conda-forge
     mimesis                   7.0.0              pyhd8ed1ab_0    conda-forge
     mistune                   2.0.4              pyhd8ed1ab_0    conda-forge
     mkl                       2022.2.1         h84fe81f_16997    conda-forge
     moto                      4.1.1              pyhd8ed1ab_0    conda-forge
     mpc                       1.3.1                hfe3b2da_0    conda-forge
     mpfr                      4.1.0                h9202a9a_1    conda-forge
     mpi                       1.0                     openmpi    conda-forge
     mpi4py                    3.1.4                    pypi_0    pypi
     msgpack-python            1.0.4            py38h43d8883_1    conda-forge
     multidict                 6.0.4            py38h1de0b5d_0    conda-forge
     multipledispatch          0.6.0                      py_0    conda-forge
     multiprocess              0.70.14          py38h0a891b7_3    conda-forge
     munch                     2.5.0                      py_0    conda-forge
     munkres                   1.1.4              pyh9f0ad1d_0    conda-forge
     myst-nb                   0.17.1             pyhd8ed1ab_0    conda-forge
     myst-parser               0.18.1             pyhd8ed1ab_0    conda-forge
     nbclassic                 0.4.8              pyhd8ed1ab_0    conda-forge
     nbclient                  0.5.13             pyhd8ed1ab_0    conda-forge
     nbconvert                 7.2.9              pyhd8ed1ab_0    conda-forge
     nbconvert-core            7.2.9              pyhd8ed1ab_0    conda-forge
     nbconvert-pandoc          7.2.9              pyhd8ed1ab_0    conda-forge
     nbformat                  5.7.3              pyhd8ed1ab_0    conda-forge
     nbsphinx                  0.8.12             pyhd8ed1ab_0    conda-forge
     nccl                      2.14.3.1             h0800d71_0    conda-forge
     ncurses                   6.3                  h27087fc_1    conda-forge
     nest-asyncio              1.5.6              pyhd8ed1ab_0    conda-forge
     networkx                  3.0                pyhd8ed1ab_0    conda-forge
     ninja                     1.11.0               h924138e_0    conda-forge
     nltk                      3.8.1              pyhd8ed1ab_0    conda-forge
     nodeenv                   1.7.0              pyhd8ed1ab_0    conda-forge
     notebook                  6.5.2              pyha770c72_1    conda-forge
     notebook-shim             0.2.2              pyhd8ed1ab_0    conda-forge
     nspr                      4.35                 h27087fc_0    conda-forge
     nss                       3.82                 he02c5a1_0    conda-forge
     numba                     0.56.4           py38h9a4aae9_0    conda-forge
     numpy                     1.23.5           py38h7042d01_0    conda-forge
     numpydoc                  1.5.0              pyhd8ed1ab_0    conda-forge
     nvcc_linux-64             11.8                ha67cc55_22    conda-forge
     nvtx                      0.2.3            py38h0a891b7_2    conda-forge
     ogb                       1.3.5              pyhd8ed1ab_0    conda-forge
     openapi-schema-validator  0.2.3              pyhd8ed1ab_0    conda-forge
     openapi-spec-validator    0.4.0              pyhd8ed1ab_1    conda-forge
     openjpeg                  2.5.0                hfec8fc6_2    conda-forge
     openmpi                   4.1.4              ha1ae619_102    conda-forge
     openssl                   3.0.7                h0b41bf4_2    conda-forge
     orc                       1.8.2                hfdbbad2_0    conda-forge
     outdated                  0.2.2              pyhd8ed1ab_0    conda-forge
     packaging                 23.0               pyhd8ed1ab_0    conda-forge
     pandas                    1.5.3            py38hdc8b05c_0    conda-forge
     pandoc                    1.19.2                        0    conda-forge
     pandocfilters             1.5.0              pyhd8ed1ab_0    conda-forge
     paramiko                  3.0.0              pyhd8ed1ab_0    conda-forge
     parquet-cpp               1.5.1                         2    conda-forge
     parso                     0.8.3              pyhd8ed1ab_0    conda-forge
     partd                     1.3.0              pyhd8ed1ab_0    conda-forge
     patsy                     0.5.3              pyhd8ed1ab_0    conda-forge
     pcre2                     10.40                hc3806b6_0    conda-forge
     pexpect                   4.8.0              pyh1a96a4e_2    conda-forge
     pickleshare               0.7.5                   py_1003    conda-forge
     pillow                    9.4.0            py38hb32c036_0    conda-forge
     pip                       23.0               pyhd8ed1ab_0    conda-forge
     pixman                    0.40.0               h36c2ea0_0    conda-forge
     platformdirs              2.6.2              pyhd8ed1ab_0    conda-forge
     pluggy                    1.0.0              pyhd8ed1ab_5    conda-forge
     pooch                     1.6.0              pyhd8ed1ab_0    conda-forge
     poppler                   23.01.0              h091648b_0    conda-forge
     poppler-data              0.4.11               hd8ed1ab_0    conda-forge
     postgresql                15.1                 h3248436_3    conda-forge
     pre-commit                3.0.2            py38h578d9bd_0    conda-forge
     proj                      9.1.1                h8ffa02c_2    conda-forge
     prometheus_client         0.16.0             pyhd8ed1ab_0    conda-forge
     prompt-toolkit            3.0.36             pyha770c72_0    conda-forge
     protobuf                  4.21.12          py38h8dc9893_0    conda-forge
     psutil                    5.9.4            py38h0a891b7_0    conda-forge
     pthread-stubs             0.4               h36c2ea0_1001    conda-forge
     ptxcompiler               0.7.0            py38h241159d_3    conda-forge
     ptyprocess                0.7.0              pyhd3deb0d_0    conda-forge
     pure_eval                 0.2.2              pyhd8ed1ab_0    conda-forge
     py                        1.11.0             pyh6c4a22f_0    conda-forge
     py-cpuinfo                9.0.0              pyhd8ed1ab_0    conda-forge
     pyarrow                   10.0.1          py38hf05218d_6_cpu    conda-forge
     pyasn1                    0.4.8                      py_0    conda-forge
     pycparser                 2.21               pyhd8ed1ab_0    conda-forge
     pydantic                  1.10.4           py38h1de0b5d_1    conda-forge
     pydata-sphinx-theme       0.12.0             pyhd8ed1ab_0    conda-forge
     pygments                  2.14.0             pyhd8ed1ab_0    conda-forge
     pynacl                    1.5.0            py38h0a891b7_2    conda-forge
     pynndescent               0.5.8              pyh1a96a4e_0    conda-forge
     pynvml                    11.4.1             pyhd8ed1ab_0    conda-forge
     pyopenssl                 23.0.0             pyhd8ed1ab_0    conda-forge
     pyorc                     0.8.0            py38h4492b77_2    conda-forge
     pyparsing                 3.0.9              pyhd8ed1ab_0    conda-forge
     pyproj                    3.4.1            py38h58d5fe2_1    conda-forge
     pyrsistent                0.19.3           py38h1de0b5d_0    conda-forge
     pysocks                   1.7.1              pyha2e5f31_6    conda-forge
     pytest                    7.2.1              pyhd8ed1ab_0    conda-forge
     pytest-benchmark          4.0.0              pyhd8ed1ab_0    conda-forge
     pytest-cases              3.6.13             pyhd8ed1ab_0    conda-forge
     pytest-cov                4.0.0              pyhd8ed1ab_0    conda-forge
     pytest-xdist              3.1.0              pyhd8ed1ab_0    conda-forge
     python                    3.8.15          he550d4f_1_cpython    conda-forge
     python-confluent-kafka    1.7.0            py38h497a2fe_2    conda-forge
     python-dateutil           2.8.2              pyhd8ed1ab_0    conda-forge
     python-fastjsonschema     2.16.2             pyhd8ed1ab_0    conda-forge
     python-jose               3.3.0              pyh6c4a22f_1    conda-forge
     python-json-logger        2.0.4              pyhd8ed1ab_0    conda-forge
     python-louvain            0.16               pyhd8ed1ab_0    conda-forge
     python-snappy             0.6.1            py38h1ddbb56_0    conda-forge
     python-xxhash             3.2.0            py38h1de0b5d_0    conda-forge
     python_abi                3.8                      3_cp38    conda-forge
     pytorch                   1.11.0              py3.8_cpu_0    pytorch
     pytorch-mutex             1.0                         cpu    pytorch
     pytz                      2022.7.1           pyhd8ed1ab_0    conda-forge
     pywin32-on-windows        0.1.0              pyh1179c8e_3    conda-forge
     pyyaml                    6.0              py38h0a891b7_5    conda-forge
     pyzmq                     25.0.0           py38he24dcef_0    conda-forge
     re2                       2022.06.01           h27087fc_1    conda-forge
     readline                  8.1.2                h0f457ee_0    conda-forge
     recommonmark              0.7.1              pyhd8ed1ab_0    conda-forge
     regex                     2022.10.31       py38h0a891b7_0    conda-forge
     requests                  2.28.2             pyhd8ed1ab_0    conda-forge
     responses                 0.21.0             pyhd8ed1ab_0    conda-forge
     rfc3339-validator         0.1.4              pyhd8ed1ab_0    conda-forge
     rfc3986-validator         0.1.1              pyh9f0ad1d_0    conda-forge
     rhash                     1.4.3                h166bdaf_0    conda-forge
     rsa                       4.9                pyhd8ed1ab_0    conda-forge
     rtree                     1.0.1            py38h02d302b_1    conda-forge
     s2n                       1.3.31               h3358134_0    conda-forge
     s3fs                      2023.1.0           pyhd8ed1ab_0    conda-forge
     s3transfer                0.6.0              pyhd8ed1ab_0    conda-forge
     sacremoses                0.0.53             pyhd8ed1ab_0    conda-forge
     scikit-build              0.16.6             pyh56297ac_0    conda-forge
     scikit-learn              1.2.1            py38h1e1a916_0    conda-forge
     scipy                     1.10.0           py38h10c12cc_0    conda-forge
     seaborn                   0.12.2               hd8ed1ab_0    conda-forge
     seaborn-base              0.12.2             pyhd8ed1ab_0    conda-forge
     sed                       4.8                  he412f7d_0    conda-forge
     send2trash                1.8.0              pyhd8ed1ab_0    conda-forge
     setuptools                67.0.0                   pypi_0    pypi
     shapely                   2.0.1            py38hd07e089_0    conda-forge
     six                       1.16.0             pyh6c4a22f_0    conda-forge
     snappy                    1.1.9                hbd366e4_2    conda-forge
     sniffio                   1.3.0              pyhd8ed1ab_0    conda-forge
     snowballstemmer           2.2.0              pyhd8ed1ab_0    conda-forge
     sortedcontainers          2.4.0              pyhd8ed1ab_0    conda-forge
     soupsieve                 2.3.2.post1        pyhd8ed1ab_0    conda-forge
     sparse                    0.13.0             pyhd8ed1ab_0    conda-forge
     spdlog                    1.8.5                h4bd325d_1    conda-forge
     sphinx                    5.3.0              pyhd8ed1ab_0    conda-forge
     sphinx-autobuild          2021.3.14          pyhd8ed1ab_0    conda-forge
     sphinx-copybutton         0.5.0              pyhd8ed1ab_0    conda-forge
     sphinx-markdown-tables    0.0.17             pyh6c4a22f_0    conda-forge
     sphinxcontrib-applehelp   1.0.4              pyhd8ed1ab_0    conda-forge
     sphinxcontrib-devhelp     1.0.2                      py_0    conda-forge
     sphinxcontrib-htmlhelp    2.0.0              pyhd8ed1ab_0    conda-forge
     sphinxcontrib-jsmath      1.0.1                      py_0    conda-forge
     sphinxcontrib-qthelp      1.0.3                      py_0    conda-forge
     sphinxcontrib-serializinghtml 1.1.5              pyhd8ed1ab_2    conda-forge
     sphinxcontrib-websupport  1.2.4              pyhd8ed1ab_1    conda-forge
     sqlalchemy                1.4.46           py38h1de0b5d_0    conda-forge
     sqlite                    3.40.0               h4ff8645_0    conda-forge
     sshpubkeys                3.3.1              pyhd8ed1ab_0    conda-forge
     stack_data                0.6.2              pyhd8ed1ab_0    conda-forge
     statsmodels               0.13.5           py38h26c90d9_2    conda-forge
     streamz                   0.6.4              pyh6c4a22f_0    conda-forge
     sysroot_linux-64          2.17                h4a8ded7_13    conda-forge
     tabulate                  0.9.0              pyhd8ed1ab_1    conda-forge
     tbb                       2021.7.0             h924138e_1    conda-forge
     tblib                     1.7.0              pyhd8ed1ab_0    conda-forge
     terminado                 0.17.1             pyh41d4057_0    conda-forge
     threadpoolctl             3.1.0              pyh8a188c0_0    conda-forge
     tiledb                    2.13.2               hd532e3d_0    conda-forge
     tinycss2                  1.2.1              pyhd8ed1ab_0    conda-forge
     tk                        8.6.12               h27826a3_0    conda-forge
     tokenizers                0.13.1           py38h8bed557_2    conda-forge
     toml                      0.10.2             pyhd8ed1ab_0    conda-forge
     tomli                     2.0.1              pyhd8ed1ab_0    conda-forge
     toolz                     0.12.0             pyhd8ed1ab_0    conda-forge
     tornado                   6.2              py38h0a891b7_1    conda-forge
     tqdm                      4.64.1             pyhd8ed1ab_0    conda-forge
     traitlets                 5.9.0              pyhd8ed1ab_0    conda-forge
     transformers              4.24.0             pyhd8ed1ab_0    conda-forge
     treelite                  3.1.0            py38h2820b77_0    conda-forge
     treelite-runtime          3.1.0                    pypi_0    pypi
     typing-extensions         4.4.0                hd8ed1ab_0    conda-forge
     typing_extensions         4.4.0              pyha770c72_0    conda-forge
     tzcode                    2022g                h166bdaf_0    conda-forge
     tzdata                    2022g                h191b570_0    conda-forge
     ucx                       1.13.1               h538f049_1    conda-forge
     ucx-proc                  1.0.0                       gpu    rapidsai
     ucx-py                    0.30.00a230131  py38_ga21f62a_17    rapidsai-nightly
     ukkonen                   1.0.1            py38h43d8883_3    conda-forge
     umap-learn                0.5.3            py38h578d9bd_0    conda-forge
     unicodedata2              15.0.0           py38h0a891b7_0    conda-forge
     urllib3                   1.26.14            pyhd8ed1ab_0    conda-forge
     virtualenv                20.17.1          py38h578d9bd_0    conda-forge
     wcwidth                   0.2.6              pyhd8ed1ab_0    conda-forge
     webencodings              0.5.1                      py_1    conda-forge
     websocket-client          1.5.0              pyhd8ed1ab_0    conda-forge
     werkzeug                  2.1.2              pyhd8ed1ab_1    conda-forge
     wheel                     0.38.4             pyhd8ed1ab_0    conda-forge
     wrapt                     1.14.1           py38h0a891b7_1    conda-forge
     xerces-c                  3.2.4                h55805fa_1    conda-forge
     xmltodict                 0.13.0             pyhd8ed1ab_0    conda-forge
     xorg-kbproto              1.0.7             h7f98852_1002    conda-forge
     xorg-libice               1.0.10               h7f98852_0    conda-forge
     xorg-libsm                1.2.3             hd9c2040_1000    conda-forge
     xorg-libx11               1.7.2                h7f98852_0    conda-forge
     xorg-libxau               1.0.9                h7f98852_0    conda-forge
     xorg-libxdmcp             1.1.3                h7f98852_0    conda-forge
     xorg-libxext              1.3.4                h7f98852_1    conda-forge
     xorg-libxrender           0.9.10            h7f98852_1003    conda-forge
     xorg-renderproto          0.11.1            h7f98852_1002    conda-forge
     xorg-xextproto            7.3.0             h7f98852_1002    conda-forge
     xorg-xproto               7.0.31            h7f98852_1007    conda-forge
     xxhash                    0.8.1                h0b41bf4_0    conda-forge
     xyzservices               2022.9.0           pyhd8ed1ab_0    conda-forge
     xz                        5.2.6                h166bdaf_0    conda-forge
     yaml                      0.2.5                h7f98852_2    conda-forge
     yarl                      1.8.2            py38h0a891b7_0    conda-forge
     zeromq                    4.3.4                h9c3ff4c_1    conda-forge
     zict                      2.2.0              pyhd8ed1ab_0    conda-forge
     zipp                      3.12.0             pyhd8ed1ab_0    conda-forge
     zlib                      1.2.13               h166bdaf_4    conda-forge
     zstd                      1.5.2                h3eb15da_6    conda-forge

davidwendt commented 1 year ago

Investigating this I found the problem is reported through thrust::exclusive_scan which is implemented using CUB's Device::ExclusiveScan. Exclusive-scan is a unique function where the last element is not processed. For example,

int data[6] = {1, 0, 2, 2, 1, 3};
thrust::exclusive_scan(data, data + 6, data); // in-place scan
// data is now {0, 1, 1, 3, 5, 6}

Note that the last element in data is not actually used. Any value can be there and it would not affect the result.

int data[6] = {1, 0, 2, 2, 1, X};
thrust::exclusive_scan(data, data + 6, data); // in-place scan
// data is now {0, 1, 1, 3, 5, 6}

The normal processing of strings APIs that return strings in libcudf is to do two passes. The 1st pass will compute the output sizes and is used to build the output chars column and offsets. The 2nd pass will fill in the chars column. The number of output offsets are usually known and so the output offsets column is created first. The 1st pass will place the sizes into this column temporarily. The sizes are usually only needed to build the offsets so there is normally no need to put the sizes into a separate temporary buffer/vector since the offsets can be computed directly in-place.

Although the number of output sizes is the same as the number of output rows, the offsets count is always +1 the number of output rows. The first element is 0 and the final element is the the total size in bytes. The exclusive-scan does this for us in a single step:

int output_rows = 5;
int offsets[] = {1, 2, 0, 2, 4, X};  // populated with sizes, the last entry is garbage
thrust::exclusive_scan(offsets, offsets + 6, offsets);
// offsets is now {0, 1, 3, 3, 5, 9}

Although the last element may be read by thrust/CUB, it is never actually used. This is a limitation of the initcheck in compute-sanitizer. It only knows the uninitialized memory has been read, it has no idea that is not actually used in producing output. So I don't think we can reliably use initcheck to verify invalid usage of uninitialized data.

I would not recommend fixing this by artificially setting the last entry since it would impact the performance unnecessarily in the many, many places that this is used.

wence- commented 1 year ago

Is this something we should take to the compute-sanitizer team?

I would have thought that a write to the final entry would not provoke an uninitialised read warning (it wouldn't in host code with valgrind I think)

davidwendt commented 1 year ago

Is this something we should take to the compute-sanitizer team?

I doubt they can trace the read down to where it eventually is used/not-used -- at what level in the stack-trace does it actually get discarded. I'm guessing the values are block loaded into an array and then to a local register and then finally discarded.

I would have thought that a write to the final entry would not provoke an uninitialised read warning (it wouldn't in host code with valgrind I think)

I believe thrust/CUB is doing a read of all available data in some low-level utility probably common to both types of scans or other CUB functions. The fact the last value eventually is not used is specific to exclusive-scan. It may be worth opening an issue with thrust but I fear the solution would be non-performant checks.

vyasr commented 1 year ago

In an ideal world (not saying this is in any way possible right now) this seems like a case where compute sanitizer would offer something like pragmas to allow code to say "this block is going to include unsafe accesses, ignore them because it's actually OK". I don't know that there's any other general fix, since IIUC effectively what is happening is that thrust is performing an unsafe access to potentially garbage data but it's OK because the value that is read will never be used.

wence- commented 1 year ago

I believe thrust/CUB is doing a read of all available data in some low-level utility probably common to both types of scans or other CUB functions. The fact the last value eventually is not used is specific to exclusive-scan. It may be worth opening an issue with thrust but I fear the solution would be non-performant checks.

FWIW, I think the cudf usage here is UB (if we're going by thrust modelling std::algorithm). Since the first two arguments to thrust::exclusive_scan must model InputIterator which says that the full range of the iterator must be dereferencable. Since it is UB to read an uninitialised value, dereferencing the last entry in the input iterator is bad (since it is uninitialised).

Arguably an exclusive_scan implementation doesn't need to dereference the last entry of the input iterator, but I can't see anything in the standard that requires that of an implementation.

wence- commented 1 year ago

Related request to guarantee that the final entry of the input iterator is never read by CUB https://github.com/NVIDIA/cccl/issues/876

In terms of general uninitialized memory. Perhaps there should be a compile time option (off by default) for libcudf to zero-initialize device allocations (this is probably much harder than it sounds), to be used as a debugging tool.

vyasr commented 1 year ago

I agree, it seems like another case of technically UB but we're taking advantage of knowing the implementation detail that in practice this is safe. I doubt that that std::exclusive_scan standard will be modified to account for the desire to not read the final value, but I suppose thrust could promise that anyway. If there's a cub request then a thrust request also seems appropriate, @jrhemstad WDYT?

wence- commented 1 year ago

technically UB but we're taking advantage of knowing the implementation detail that in practice this is safe.

Although this works right now, this is not how UB works. A sufficiently smart compiler™ will treat UB as an impossible situation, and is free to (for example) eliminate any code related to it. In this circumstance, that might be elimination of the call to exclusive_scan completely (admittedly the reasoning to get here is pretty convoluted, so I don't think it is very likely that this would occur, since the iteration bounds to the iterator that partially initialise the input to exclusive_scan are not compile-time known). So it would be a totally valid to erase any calls to sizes_to_offsets that occur in these contexts. Even if the implementation of exclusive_scan remains unchanged.

I agree with @davidwendt that running an extra kernel to memset the final entry in the allocated offsets array to zero is not the solution.

I would prefer to try and fix this in CUB, and ideally the C++ standard, though I don't know how amenable they'd be to changing the definition of exclusive_scan.

vyasr commented 1 year ago

I mean, that's the definition of an implementation detail right? UB says an implementation can do whatever it wants. One of the options is to do something that we think is sensible. We know empirically that the code thrust produces is not being elided in the way you describe by the CUDA compiler, even though it could be. I'm not saying that we should rely on the current behavior, of course. We need thrust to promise not to read the last value, which in turn means we need cub to promise not to read the last value. If we can get that upstreamed to the C++ standard for std::exclusive_scan, that's even better.

davidwendt commented 1 month ago

I believe the summary here results in no recommended changes to libcudf.

vyasr commented 4 weeks ago

Would https://github.com/NVIDIA/cccl/issues/876 address this upstream, or do we need more changes? I'm inclined to agree with @wence- that ultimately this can only really be fixed by the C++ standard specifying that exclusive_scan will not read the last value, but neither that nor any of the other proposed changes above (e.g. making thrust do this safely) seem to be forthcoming since they either are inherently slow processes or they have potentially deleterious performance consequences that we're unlikely to stomach.

wence- commented 3 weeks ago

Would https://github.com/NVIDIA/cccl/issues/876 address this upstream, or do we need more changes?

I believe it would, yes.

rapidsai / cudf

[BUG] `cudf::strings::from_integers` does not appear to be `compute-sanitizer --tool initcheck` clean #12667