rapidsai / cuml

cuML - RAPIDS Machine Learning Library
https://docs.rapids.ai/api/cuml/stable/
Apache License 2.0
4.25k stars 534 forks source link

[QST]Why is cuml GPU dozens of times slower than scikit-learn CPU in Lasso,Ridge and ElasticNet models? #6080

Open JiayuChen02 opened 1 month ago

JiayuChen02 commented 1 month ago

I tried to compare the speed difference of the same model on CPU and GPU. Here is my code:

import numpy as np
import pandas as pd
import time
import cudf
import cupy as cp
from cuml.linear_model import Lasso as cuml_model
from sklearn.linear_model import Lasso as sk_model

n_samples = 300 
n_features = 120000  

X = np.random.rand(n_samples, n_features)
y = np.random.rand(n_samples)

param = {
    "alpha": 0.001,
    "max_iter": 1000,
    "tol": 1e-3,
}

sklearn_model = sk_model()
sklearn_model.set_params(**param)

start_time = time.time()
sklearn_model.fit(X, y)
end_time = time.time()

print(sklearn_model.get_params())

print(f"scikit-learn training time: {end_time - start_time:.4f} seconds")

X_cudf = cp.array(X)
y_cudf = cp.array(y)

rapids_model = cuml_model()
rapids_model.set_params(**param)

start_time = time.time()

rapids_model.fit(X_cudf, y_cudf)

end_time = time.time()
print(rapids_model.get_params())
print(f"cuML GPU training time: {end_time - start_time:.4f} seconds")

And the result is

{'alpha': 0.001, 'copy_X': True, 'fit_intercept': True, 'max_iter': 1000, 'positive': False, 'precompute': False, 'random_state': None, 'selection': 'cyclic', 'tol': 0.001, 'warm_start': False}
scikit-learn training time: 14.0172 seconds
{'selection': 'cyclic', 'solver': 'cd', 'tol': 0.001, 'output_type': 'input', 'normalize': False, 'handle': <pylibraft.common.handle.Handle object at 0x7fa5b99e02d0>, 'alpha': 0.001, 'fit_intercept': True, 'verbose': 4, 'max_iter': 1000}
cuML GPU training time: 486.6329 seconds

Why?

dantegd commented 1 month ago

Thanks for the issue @JiayuChen02, I just tried your code on a machine with a T4 and this is the results I got:

{'alpha': 0.001, 'copy_X': True, 'fit_intercept': True, 'max_iter': 1000, 'positive': False, 'precompute': False, 'random_state': None, 'selection': 'cyclic', 'tol': 0.001, 'warm_start': False}
scikit-learn training time: 7.7504 seconds
{'output_type': 'input', 'verbose': 4, 'handle': <pylibraft.common.handle.Handle object at 0x7fd8addf73f0>, 'fit_intercept': True, 'normalize': False, 'tol': 0.001, 'solver': 'cd', 'max_iter': 1000, 'alpha': 0.001, 'selection': 'cyclic'}
cuML GPU training time: 0.8425 seconds

In general, 300 rows is not a dataset where I would expect massive GPU speedups, but things seem to be working fine. What could be happening is you could be running maybe into some JIT compilation issues? What GPU are you using? It would be very helpful if you could run this script and put the output here: https://github.com/rapidsai/cuml/blob/branch-24.10/print_env.sh

JiayuChen02 commented 1 month ago

Thanks for the issue @JiayuChen02, I just tried your code on a machine with a T4 and this is the results I got:

{'alpha': 0.001, 'copy_X': True, 'fit_intercept': True, 'max_iter': 1000, 'positive': False, 'precompute': False, 'random_state': None, 'selection': 'cyclic', 'tol': 0.001, 'warm_start': False}
scikit-learn training time: 7.7504 seconds
{'output_type': 'input', 'verbose': 4, 'handle': <pylibraft.common.handle.Handle object at 0x7fd8addf73f0>, 'fit_intercept': True, 'normalize': False, 'tol': 0.001, 'solver': 'cd', 'max_iter': 1000, 'alpha': 0.001, 'selection': 'cyclic'}
cuML GPU training time: 0.8425 seconds

In general, 300 rows is not a dataset where I would expect massive GPU speedups, but things seem to be working fine. What could be happening is you could be running maybe into some JIT compilation issues? What GPU are you using? It would be very helpful if you could run this script and put the output here: https://github.com/rapidsai/cuml/blob/branch-24.10/print_env.sh

@dantegd Thanks for your reply. This is the output of print_env.sh

Click here to see environment details

     **git***
     Not inside a git repository

     ***OS Information***
     DISTRIB_ID=Ubuntu
     DISTRIB_RELEASE=22.04
     DISTRIB_CODENAME=jammy
     DISTRIB_DESCRIPTION="Ubuntu 22.04.4 LTS"
     PRETTY_NAME="Ubuntu 22.04.4 LTS"
     NAME="Ubuntu"
     VERSION_ID="22.04"
     VERSION="22.04.4 LTS (Jammy Jellyfish)"
     VERSION_CODENAME=jammy
     ID=ubuntu
     ID_LIKE=debian
     HOME_URL="https://www.ubuntu.com/"
     SUPPORT_URL="https://help.ubuntu.com/"
     BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
     PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
     UBUNTU_CODENAME=jammy
     Linux 6eae716c1e8c 5.4.0-100-generic #113-Ubuntu SMP Thu Feb 3 18:43:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

***GPU Information***
     Wed Sep 25 01:13:48 2024
     +-----------------------------------------------------------------------------------------+
     | NVIDIA-SMI 555.42.02              Driver Version: 555.42.02      CUDA Version: 12.5     |
     |-----------------------------------------+------------------------+----------------------+
     | GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
     | Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
     |                                         |                        |               MIG M. |
     |=========================================+========================+======================|
     |   0  NVIDIA A100 80GB PCIe          Off |   00000000:17:00.0 Off |                    0 |
     | N/A   35C    P0             63W /  300W |     989MiB /  81920MiB |      0%      Default |
     |                                         |                        |             Disabled |
     +-----------------------------------------+------------------------+----------------------+
     |   1  NVIDIA A100 80GB PCIe          Off |   00000000:CA:00.0 Off |                    0 |
     | N/A   34C    P0             43W /  300W |       4MiB /  81920MiB |      0%      Default |
     |                                         |                        |             Disabled |
     +-----------------------------------------+------------------------+----------------------+

     +-----------------------------------------------------------------------------------------+
     | Processes:                                                                              |
     |  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
     |        ID   ID                                                               Usage      |
     |=========================================================================================|
     +-----------------------------------------------------------------------------------------+
***CPU***
     Architecture:                    x86_64
     CPU op-mode(s):                  32-bit, 64-bit
     Address sizes:                   46 bits physical, 57 bits virtual
     Byte Order:                      Little Endian
     CPU(s):                          104
     On-line CPU(s) list:             0-103
     Vendor ID:                       GenuineIntel
     Model name:                      Intel(R) Xeon(R) Gold 5320 CPU @ 2.20GHz
     CPU family:                      6
     Model:                           106
     Thread(s) per core:              2
     Core(s) per socket:              26
     Socket(s):                       2
     Stepping:                        6
     BogoMIPS:                        4400.00
     Flags:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 invpcid_single ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb intel_pt avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local wbnoinvd dtherm ida arat pln pts avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg tme avx512_vpopcntdq rdpid md_clear pconfig flush_l1d arch_capabilities

Virtualization:                  VT-x
     L1d cache:                       2.4 MiB (52 instances)
     L1i cache:                       1.6 MiB (52 instances)
     L2 cache:                        65 MiB (52 instances)
     L3 cache:                        78 MiB (2 instances)
     NUMA node(s):                    2
     NUMA node0 CPU(s):               0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46,48,50,52,54,56,58,60,62,64,66,68,70,72,74,76,78,80,82,84,86,88,90,92,94,96,98,100,102
     NUMA node1 CPU(s):               1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47,49,51,53,55,57,59,61,63,65,67,69,71,73,75,77,79,81,83,85,87,89,91,93,95,97,99,101,103
     Vulnerability Itlb multihit:     Not affected
     Vulnerability L1tf:              Not affected
     Vulnerability Mds:               Not affected
     Vulnerability Meltdown:          Not affected
     Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
     Vulnerability Spectre v1:        Mitigation; usercopy/swapgs barriers and __user pointer sanitization
     Vulnerability Spectre v2:        Mitigation; Enhanced IBRS, IBPB conditional, RSB filling
     Vulnerability Srbds:             Not affected
     Vulnerability Tsx async abort:   Not affected

     ***CMake***
***g++***

     ***nvcc***
     /opt/conda/bin/nvcc
     nvcc: NVIDIA (R) Cuda compiler driver
     Copyright (c) 2005-2024 NVIDIA Corporation
     Built on Thu_Jun__6_02:18:23_PDT_2024
     Cuda compilation tools, release 12.5, V12.5.82
     Build cuda_12.5.r12.5/compiler.34385749_0

     ***Python***
     /opt/conda/bin/python
     Python 3.9.19

     ***Environment Variables***
     PATH                            : /opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
     LD_LIBRARY_PATH                 : /usr/local/nvidia/lib:/usr/local/nvidia/lib64
     NUMBAPRO_NVVM                   :
     NUMBAPRO_LIBDEVICE              :
     CONDA_PREFIX                    :
     PYTHON_PATH                     :  

***conda packages***
     /opt/conda/bin/conda
     # packages in environment at /opt/conda:
     #
     # Name                    Version                   Build  Channel
     _libgcc_mutex             0.1                 conda_forge    conda-forge
     _openmp_mutex             4.5                       2_gnu    conda-forge
     aiohappyeyeballs          2.3.5              pyhd8ed1ab_0    conda-forge
     aiohttp                   3.10.3           py39hcd6043d_0    conda-forge
     aiosignal                 1.3.1              pyhd8ed1ab_0    conda-forge
     anyio                     4.4.0              pyhd8ed1ab_0    conda-forge
     appdirs                   1.4.4              pyh9f0ad1d_0    conda-forge
     archspec                  0.2.3              pyhd8ed1ab_0    conda-forge
     argon2-cffi               23.1.0             pyhd8ed1ab_0    conda-forge
     argon2-cffi-bindings      21.2.0           py39hd1e30aa_4    conda-forge
     arrow                     1.3.0              pyhd8ed1ab_0    conda-forge
     asttokens                 2.4.1              pyhd8ed1ab_0    conda-forge
     async-timeout             4.0.3              pyhd8ed1ab_0    conda-forge
     attrs                     24.2.0             pyh71513ae_0    conda-forge
     aws-c-auth                0.7.25               hff137af_5    conda-forge
     aws-c-cal                 0.7.3                h7970872_0    conda-forge
     aws-c-common              0.9.25               h4bc722e_0    conda-forge
     aws-c-compression         0.2.18               hc649ecc_8    conda-forge
     aws-c-event-stream        0.4.2               h04a40c0_20    conda-forge
     aws-c-http                0.8.7                he2d3600_3    conda-forge
     aws-c-io                  0.14.18              h7d46f39_3    conda-forge
     aws-c-mqtt                0.10.4              h674cf7e_16    conda-forge
     aws-c-s3                  0.6.4                h28a8003_7    conda-forge
     aws-c-sdkutils            0.1.19               hc649ecc_0    conda-forge
     aws-checksums             0.1.18               hc649ecc_8    conda-forge
     aws-crt-cpp               0.27.5               heec6497_6    conda-forge
     aws-sdk-cpp               1.11.379             he20dfa5_2    conda-forge
     azure-core-cpp            1.13.0               h935415a_0    conda-forge
     azure-identity-cpp        1.8.0                hd126650_2    conda-forge
     azure-storage-blobs-cpp   12.12.0              hd2e3451_0    conda-forge
     azure-storage-common-cpp  12.7.0               h10ac4d7_1    conda-forge
     azure-storage-files-datalake-cpp 12.11.0              h325d260_1    conda-forge
     beautifulsoup4            4.12.3             pyha770c72_0    conda-forge

bleach                    6.1.0              pyhd8ed1ab_0    conda-forge
     blosc                     1.21.6               hef167b5_0    conda-forge
     bokeh                     3.4.2              pyhd8ed1ab_0    conda-forge
     boltons                   24.0.0             pyhd8ed1ab_0    conda-forge
     branca                    0.7.2              pyhd8ed1ab_0    conda-forge
     brotli                    1.1.0                hd590300_1    conda-forge
     brotli-bin                1.1.0                hd590300_1    conda-forge
     brotli-python             1.1.0            py39h3d6467e_1    conda-forge
     bzip2                     1.0.8                h4bc722e_7    conda-forge
     c-ares                    1.33.0               ha66036c_0    conda-forge
     ca-certificates           2024.7.4             hbcca054_0    conda-forge
     cached-property           1.5.2                hd8ed1ab_1    conda-forge
     cached_property           1.5.2              pyha770c72_1    conda-forge
     cachetools                5.4.0              pyhd8ed1ab_0    conda-forge
     cairo                     1.18.0               h3faef2a_0    conda-forge
     certifi                   2024.7.4           pyhd8ed1ab_0    conda-forge
     cffi                      1.17.0           py39h49a4b6b_0    conda-forge
     cfitsio                   4.3.1                hbdc6101_0    conda-forge
     charset-normalizer        3.3.2              pyhd8ed1ab_0    conda-forge
     click                     8.1.7           unix_pyh707e725_0    conda-forge
     cloudpickle               3.0.0              pyhd8ed1ab_0    conda-forge
     colorama                  0.4.6              pyhd8ed1ab_0    conda-forge
     colorcet                  3.1.0              pyhd8ed1ab_0    conda-forge
     conda                     24.7.1           py39hf3d152e_0    conda-forge
     conda-libmamba-solver     24.7.0             pyhd8ed1ab_0    conda-forge
     conda-package-handling    2.3.0              pyh7900ff3_0    conda-forge
     conda-package-streaming   0.10.0             pyhd8ed1ab_0    conda-forge
     contourpy                 1.2.1            py39h7633fee_0    conda-forge
     cucim                     24.08.00        cuda12_py39_240807_gc22f905_0    rapidsai
     cuda-cccl_linux-64        12.5.39              ha770c72_0    conda-forge
     cuda-crt-dev_linux-64     12.5.82              ha770c72_0    conda-forge
     cuda-crt-tools            12.5.82              ha770c72_0    conda-forge
     cuda-cudart               12.5.82              he02047a_0    conda-forge
     cuda-cudart-dev           12.5.82              he02047a_0    conda-forge
     cuda-cudart-dev_linux-64  12.5.82              h85509e4_0    conda-forge
     cuda-cudart-static        12.5.82              he02047a_0    conda-forge
cuda-cudart-static_linux-64 12.5.82              h85509e4_0    conda-forge
     cuda-cudart_linux-64      12.5.82              h85509e4_0    conda-forge
     cuda-nvcc-dev_linux-64    12.5.82              ha770c72_0    conda-forge
     cuda-nvcc-impl            12.5.82              hd3aeb46_0    conda-forge
     cuda-nvcc-tools           12.5.82              hd3aeb46_0    conda-forge
     cuda-nvrtc                12.5.82              he02047a_0    conda-forge
     cuda-nvvm-dev_linux-64    12.5.82              ha770c72_0    conda-forge
     cuda-nvvm-impl            12.5.82              h59595ed_0    conda-forge
     cuda-nvvm-tools           12.5.82              h59595ed_0    conda-forge
     cuda-profiler-api         12.5.39              ha770c72_0    conda-forge
     cuda-python               12.6.0           py39h65f4db9_0    conda-forge
     cuda-version              12.5                 hd4f0392_3    conda-forge
     cudf                      24.08.02        cuda12_py39_240814_ge776742c28_0    rapidsai
     cudf_kafka                24.08.02        cuda12_py39_240814_ge776742c28_0    rapidsai
     cugraph                   24.08.00        cuda12_py39_240808_gfc880db0c_0    rapidsai
     cuml                      24.08.00        cuda12_py39_240808_gb17f2dbbe_0    rapidsai
     cuproj                    24.08.00        cuda12_py39_240808_g601ae02f_0    rapidsai
     cupy                      13.2.0           py39hd6399d1_1    conda-forge
     cupy-core                 13.2.0           py39h3c1559c_1    conda-forge
     cuspatial                 24.08.00        cuda12_py39_240808_g601ae02f_0    rapidsai
     custreamz                 24.08.02        cuda12_py39_240814_ge776742c28_0    rapidsai
     cuvs                      24.08.00        cuda12_py39_240808_g8f250ac_0    rapidsai
     cuxfilter                 24.08.00        cuda12_py39_240808_g8e12b58_0    rapidsai
     cycler                    0.12.1             pyhd8ed1ab_0    conda-forge
     cyrus-sasl                2.1.27               h54b06d7_7    conda-forge
     cytoolz                   0.12.3           py39hd1e30aa_0    conda-forge
     dask                      2024.7.1           pyhd8ed1ab_0    conda-forge
     dask-core                 2024.7.1           pyhd8ed1ab_0    conda-forge
     dask-cuda                 24.08.02        py39_240807_gdda3243_0    rapidsai
     dask-cudf                 24.08.02        cuda12_py39_240814_ge776742c28_0    rapidsai
     dask-expr                 1.1.9              pyhd8ed1ab_0    conda-forge
     datashader                0.16.3             pyhd8ed1ab_0    conda-forge
     decorator                 5.1.1              pyhd8ed1ab_0    conda-forge
     defusedxml                0.7.1              pyhd8ed1ab_0    conda-forge
     distributed               2024.7.1           pyhd8ed1ab_0    conda-forge
     distributed-ucxx          0.39.01         py3.9_240807_g19a66b4_0    rapidsai
     distro                    1.9.0              pyhd8ed1ab_0    conda-forge
     dlpack                    0.8                  h59595ed_3    conda-forge
     ecos                      2.0.14                   pypi_0    pypi
entrypoints               0.4                pyhd8ed1ab_0    conda-forge
     exceptiongroup            1.2.2              pyhd8ed1ab_0    conda-forge
     executing                 2.0.1              pyhd8ed1ab_0    conda-forge
     expat                     2.6.2                h59595ed_0    conda-forge
     fastrlock                 0.8.2            py39h3d6467e_2    conda-forge
     fmt                       10.2.1               h00ab1b0_0    conda-forge
     folium                    0.17.0             pyhd8ed1ab_0    conda-forge
     font-ttf-dejavu-sans-mono 2.37                 hab24e00_0    conda-forge
     font-ttf-inconsolata      3.000                h77eed37_0    conda-forge
     font-ttf-source-code-pro  2.038                h77eed37_0    conda-forge
     font-ttf-ubuntu           0.83                 h77eed37_2    conda-forge
     fontconfig                2.14.2               h14ed4e7_0    conda-forge
     fonts-conda-ecosystem     1                             0    conda-forge
     fonts-conda-forge         1                             0    conda-forge
     fonttools                 4.53.1           py39hcd6043d_0    conda-forge
     fqdn                      1.5.1              pyhd8ed1ab_0    conda-forge
     freetype                  2.12.1               h267a509_2    conda-forge
     freexl                    2.0.0                h743c826_0    conda-forge
     frozendict                2.4.4            py39hd3abc70_0    conda-forge
     frozenlist                1.4.1            py39hd1e30aa_0    conda-forge
     fsspec                    2024.6.1           pyhff2d567_0    conda-forge
     gdal                      3.8.1            py39h14df8fe_3    conda-forge
     geopandas                 1.0.1              pyhd8ed1ab_0    conda-forge
     geopandas-base            1.0.1              pyha770c72_0    conda-forge
     geos                      3.12.1               h59595ed_0    conda-forge
     geotiff                   1.7.1               hf074850_14    conda-forge
     gettext                   0.22.5               he02047a_3    conda-forge
     gettext-tools             0.22.5               he02047a_3    conda-forge
     gflags                    2.2.2             he1b5a44_1004    conda-forge
     giflib                    5.2.2                hd590300_0    conda-forge
     glog                      0.7.1                hbabe93e_0    conda-forge
     h2                        4.1.0              pyhd8ed1ab_0    conda-forge
     hdf4                      4.2.15               h2a13503_7    conda-forge
     hdf5                      1.14.3          nompi_hdf9ad27_105    conda-forge
     holoviews                 1.19.1             pyhd8ed1ab_0    conda-forge
     hpack                     4.0.0              pyh9f0ad1d_0    conda-forge
     hyperframe                6.0.1              pyhd8ed1ab_0    conda-forge
     icu                       73.2                 h59595ed_0    conda-forge
     idna                      3.7                pyhd8ed1ab_0    conda-forge
imagecodecs-lite          2019.12.3        py39hd92a3bb_8    conda-forge
     imageio                   2.34.2             pyh12aca89_0    conda-forge
     importlib-metadata        8.2.0              pyha770c72_0    conda-forge
     importlib-resources       6.4.2              pyhd8ed1ab_0    conda-forge
     importlib_metadata        8.2.0                hd8ed1ab_0    conda-forge
     importlib_resources       6.4.2              pyhd8ed1ab_0    conda-forge
     ipython                   8.18.1             pyh707e725_3    conda-forge
     isoduration               20.11.0            pyhd8ed1ab_0    conda-forge
     jbig                      2.1               h7f98852_2003    conda-forge
     jedi                      0.19.1             pyhd8ed1ab_0    conda-forge
     jinja2                    3.1.4              pyhd8ed1ab_0    conda-forge
     joblib                    1.4.2              pyhd8ed1ab_0    conda-forge
     json-c                    0.17                 h1220068_1    conda-forge
     jsonpatch                 1.33               pyhd8ed1ab_0    conda-forge
     jsonpointer               3.0.0            py39hf3d152e_0    conda-forge
     jsonschema                4.23.0             pyhd8ed1ab_0    conda-forge
     jsonschema-specifications 2023.12.1          pyhd8ed1ab_0    conda-forge
     jsonschema-with-format-nongpl 4.23.0               hd8ed1ab_0    conda-forge
     jupyter-server-proxy      4.3.0              pyhd8ed1ab_0    conda-forge
     jupyter_client            8.6.2              pyhd8ed1ab_0    conda-forge
     jupyter_core              5.7.2            py39hf3d152e_0    conda-forge
     jupyter_events            0.10.0             pyhd8ed1ab_0    conda-forge
     jupyter_server            2.14.2             pyhd8ed1ab_0    conda-forge
     jupyter_server_terminals  0.5.3              pyhd8ed1ab_0    conda-forge
     jupyterlab_pygments       0.3.0              pyhd8ed1ab_1    conda-forge
     kealib                    1.5.3                hee9dde6_1    conda-forge
     keyutils                  1.6.1                h166bdaf_0    conda-forge
     kiwisolver                1.4.5            py39h7633fee_1    conda-forge
     krb5                      1.21.3               h659f571_0    conda-forge
     lazy_loader               0.4                pyhd8ed1ab_0    conda-forge
     lcms2                     2.16                 hb7c19ff_0    conda-forge
     ld_impl_linux-64          2.40                 hf3520f5_7    conda-forge
     lerc                      4.0.0                h27087fc_0    conda-forge
     libabseil                 20240116.2      cxx17_he02047a_1    conda-forge
     libaec                    1.1.3                h59595ed_0    conda-forge
     libarchive                3.7.4                hfca40fe_0    conda-forge
     libarrow                  16.1.0          h03aeac6_20_cpu    conda-forge
     libarrow-acero            16.1.0          he02047a_20_cpu    conda-forge
     libarrow-dataset          16.1.0          he02047a_20_cpu    conda-forge
libarrow-substrait        16.1.0          hc9a23c6_20_cpu    conda-forge
     libasprintf               0.22.5               he8f35ee_3    conda-forge
     libasprintf-devel         0.22.5               he8f35ee_3    conda-forge
     libblas                   3.9.0           23_linux64_openblas    conda-forge
     libbrotlicommon           1.1.0                hd590300_1    conda-forge
     libbrotlidec              1.1.0                hd590300_1    conda-forge
     libbrotlienc              1.1.0                hd590300_1    conda-forge
     libcblas                  3.9.0           23_linux64_openblas    conda-forge
     libcrc32c                 1.1.2                h9c3ff4c_0    conda-forge
     libcublas                 12.5.3.2             he02047a_0    conda-forge
     libcublas-dev             12.5.3.2             he02047a_0    conda-forge
     libcucim                  24.08.00        cuda12_240807_gc22f905_0    rapidsai
     libcudf                   24.08.02        cuda12_240814_ge776742c28_0    rapidsai
     libcudf_kafka             24.08.02        cuda12_240814_ge776742c28_0    rapidsai
     libcufft                  11.2.3.61            he02047a_0    conda-forge
     libcufile                 1.10.1.7             he02047a_0    conda-forge
     libcufile-dev             1.10.1.7             he02047a_0    conda-forge
     libcugraph                24.08.00        cuda12_240808_gfc880db0c_0    rapidsai
     libcugraph_etl            24.08.00        cuda12_240808_gfc880db0c_0    rapidsai
     libcugraphops             24.08.01        cuda12_240808_gefdc8f56_0    rapidsai
     libcuml                   24.08.00        cuda12_240808_gb17f2dbbe_0    rapidsai
     libcumlprims              24.08.00        cuda12_240808_g172c6cb_0    rapidsai
     libcurand                 10.3.6.82            he02047a_0    conda-forge
     libcurand-dev             10.3.6.82            he02047a_0    conda-forge
     libcurl                   8.9.1                hdb1bdb2_0    conda-forge
     libcusolver               11.6.3.83            he02047a_0    conda-forge
     libcusolver-dev           11.6.3.83            he02047a_0    conda-forge
     libcusparse               12.5.1.3             he02047a_0    conda-forge
     libcusparse-dev           12.5.1.3             he02047a_0    conda-forge
     libcuspatial              24.08.00        cuda12_240808_g601ae02f_0    rapidsai
     libcuvs                   24.08.00        cuda12_240808_g8f250ac_0    rapidsai
     libdeflate                1.19                 hd590300_0    conda-forge
     libedit                   3.1.20191231         he28a2e2_2    conda-forge
     libev                     4.33                 hd590300_2    conda-forge
     libevent                  2.1.12               hf998b51_1    conda-forge
     libexpat                  2.6.2                h59595ed_0    conda-forge
     libffi                    3.4.2                h7f98852_5    conda-forge
     libgcc-ng                 14.1.0               h77fa898_0    conda-forge
     libgdal                   3.8.1                h4b8bffa_3    conda-forge
libgettextpo              0.22.5               he02047a_3    conda-forge
     libgettextpo-devel        0.22.5               he02047a_3    conda-forge
     libgfortran-ng            14.1.0               h69a702a_0    conda-forge
     libgfortran5              14.1.0               hc5f4f2c_0    conda-forge
     libglib                   2.78.4               h783c2da_0    conda-forge
     libgomp                   14.1.0               h77fa898_0    conda-forge
     libgoogle-cloud           2.28.0               h26d7fe4_0    conda-forge
     libgoogle-cloud-storage   2.28.0               ha262f82_0    conda-forge
     libgrpc                   1.62.2               h15f2491_0    conda-forge
     libiconv                  1.17                 hd590300_2    conda-forge
     libjpeg-turbo             3.0.0                hd590300_1    conda-forge
     libkml                    1.3.0             hbbc8833_1020    conda-forge
     libkvikio                 24.08.02        cuda12_240807_g6b80a5c_0    rapidsai
     liblapack                 3.9.0           23_linux64_openblas    conda-forge
     libllvm14                 14.0.6               hcd5def8_4    conda-forge
     libmamba                  1.5.8                had39da4_0    conda-forge
     libmambapy                1.5.8            py39h10defb6_0    conda-forge
     libnetcdf                 4.9.2           nompi_h135f659_114    conda-forge
     libnghttp2                1.58.0               h47da74e_1    conda-forge
     libnl                     3.10.0               h4bc722e_0    conda-forge
     libnsl                    2.0.1                hd590300_0    conda-forge
     libntlm                   1.4               h7f98852_1002    conda-forge
     libnvjitlink              12.5.82              he02047a_0    conda-forge
     libnvjpeg                 12.3.2.81            he02047a_0    conda-forge
     libopenblas               0.3.27          pthreads_hac2b453_1    conda-forge
     libparquet                16.1.0          haa1307c_20_cpu    conda-forge
     libpng                    1.6.43               h2797004_0    conda-forge
     libpq                     16.4                 h482b261_0    conda-forge
     libprotobuf               4.25.3               h08a7969_0    conda-forge
     libraft                   24.08.01        cuda12_240807_g185da16b_0    rapidsai
     libraft-headers           24.08.01        cuda12_240807_g185da16b_0    rapidsai
     libraft-headers-only      24.08.01        cuda12_240807_g185da16b_0    rapidsai
     librdkafka                1.9.2                ha5a0de0_2    conda-forge
     libre2-11                 2023.09.01           h5a48ba9_2    conda-forge
     librmm                    24.08.02        cuda12_240807_g5aa8ebe6_0    rapidsai
     librttopo                 1.1.0               h8917695_15    conda-forge
     libsodium                 1.0.18               h36c2ea0_1    conda-forge
     libsolv                   0.7.30               h3509ff9_0    conda-forge
     libspatialite             5.1.0                h72606ae_3    conda-forge
libsqlite                 3.46.0               hde9e2c9_0    conda-forge
     libssh2                   1.11.0               h0841786_0    conda-forge
     libstdcxx-ng              14.1.0               hc0a3c3a_0    conda-forge
     libthrift                 0.20.0               hb90f79a_0    conda-forge
     libtiff                   4.6.0                ha9c0a0a_2    conda-forge
     libucxx                   0.39.01         cuda12_240807_g19a66b4_0    rapidsai
     libutf8proc               2.8.0                h166bdaf_0    conda-forge
     libuuid                   2.38.1               h0b41bf4_0    conda-forge
     libuv                     1.48.0               hd590300_0    conda-forge
     libwebp-base              1.4.0                hd590300_0    conda-forge
     libxcb                    1.15                 h0b41bf4_0    conda-forge
     libxcrypt                 4.4.36               hd590300_1    conda-forge
     libxgboost                2.1.1           rapidsai_h207c33c_4    rapidsai
     libxml2                   2.12.7               h4c95cb1_3    conda-forge
     libzip                    1.10.1               h2629f0a_3    conda-forge
     libzlib                   1.3.1                h4ab18f5_1    conda-forge
     linkify-it-py             2.0.3              pyhd8ed1ab_0    conda-forge
     llvmlite                  0.43.0           py39h81bab63_0    conda-forge
     locket                    1.0.0              pyhd8ed1ab_0    conda-forge
     lz4                       4.3.3            py39h79d96da_0    conda-forge
     lz4-c                     1.9.4                hcb278e6_0    conda-forge
     lzo                       2.10              hd590300_1001    conda-forge
     mamba                     1.5.8            py39hc5d2bb1_0    conda-forge
     mapclassify               2.8.0              pyhd8ed1ab_0    conda-forge
     markdown                  3.6                pyhd8ed1ab_0    conda-forge
     markdown-it-py            3.0.0              pyhd8ed1ab_0    conda-forge
     markupsafe                2.1.5            py39hd1e30aa_0    conda-forge
     matplotlib-base           3.9.1            py39h0565ad7_2    conda-forge
     matplotlib-inline         0.1.7              pyhd8ed1ab_0    conda-forge
     mdit-py-plugins           0.4.1              pyhd8ed1ab_0    conda-forge
     mdurl                     0.1.2              pyhd8ed1ab_0    conda-forge
     menuinst                  2.1.2            py39hf3d152e_0    conda-forge
     minizip                   4.0.7                h401b404_0    conda-forge
     mistune                   3.0.2              pyhd8ed1ab_0    conda-forge
     msgpack-python            1.0.8            py39h95fdab5_0    conda-forge
     multidict                 6.0.5            py39hd1e30aa_0    conda-forge
     multipledispatch          0.6.0              pyhd8ed1ab_1    conda-forge
     munkres                   1.1.4              pyh9f0ad1d_0    conda-forge
     nbclient                  0.10.0             pyhd8ed1ab_0    conda-forge
     nbconvert-core            7.16.4             pyhd8ed1ab_1    conda-forge
     nbformat                  5.10.4             pyhd8ed1ab_0    conda-forge
     nccl                      2.22.3.1             hbc370b7_1    conda-forge
     ncurses                   6.5                  h59595ed_0    conda-forge
     networkx                  3.2.1              pyhd8ed1ab_0    conda-forge
     nodejs                    22.6.0               h6d9b948_0    conda-forge
     nspr                      4.35                 h27087fc_0    conda-forge
     nss                       3.103                h593d115_0    conda-forge
     numba                     0.60.0           py39h0320e7d_0    conda-forge
     numexpr                   2.10.1                   pypi_0    pypi
     numpy                     1.26.4           py39h474f0d3_0    conda-forge
     nvcomp                    3.0.6                h10b603f_0    conda-forge
     nvtx                      0.2.10           py39hd1e30aa_0    conda-forge
     nx-cugraph                24.08.00                 py39_0    rapidsai
     openjpeg                  2.5.2                h488ebb8_0    conda-forge
     openssl                   3.3.1                h4bc722e_2    conda-forge
     orc                       2.0.1                h17fec99_1    conda-forge
     osqp                      0.6.7.post1              pypi_0    pypi
     overrides                 7.7.0              pyhd8ed1ab_0    conda-forge
     packaging                 24.1               pyhd8ed1ab_0    conda-forge
     pandas                    2.2.2            py39hfc16268_1    conda-forge
     pandocfilters             1.5.0              pyhd8ed1ab_0    conda-forge
     panel                     1.4.5              pyhd8ed1ab_0    conda-forge
     param                     2.1.1              pyhff2d567_0    conda-forge
     parso                     0.8.4              pyhd8ed1ab_0    conda-forge
     partd                     1.4.2              pyhd8ed1ab_0    conda-forge
     pcre2                     10.42                hcad00b1_0    conda-forge
     pexpect                   4.9.0              pyhd8ed1ab_0    conda-forge
     pickleshare               0.7.5                   py_1003    conda-forge
     pillow                    10.3.0           py39h90c7501_0    conda-forge
     pip                       24.2               pyhd8ed1ab_0    conda-forge
     pixman                    0.43.2               h59595ed_0    conda-forge
     pkgutil-resolve-name      1.3.10             pyhd8ed1ab_1    conda-forge
     platformdirs              4.2.2              pyhd8ed1ab_0    conda-forge
     pluggy                    1.5.0              pyhd8ed1ab_0    conda-forge
     poppler                   23.12.0              h590f24d_0    conda-forge
     poppler-data              0.4.12               hd8ed1ab_0    conda-forge
     postgresql                16.4                 ha8faf9a_0    conda-forge
     proj                      9.3.0                h1d62c97_2    conda-forge
     prometheus_client         0.20.0             pyhd8ed1ab_0    conda-forge
     prompt-toolkit            3.0.47             pyha770c72_0    conda-forge
     psutil                    6.0.0            py39hd3abc70_0    conda-forge
     pthread-stubs             0.4               h36c2ea0_1001    conda-forge
     ptyprocess                0.7.0              pyhd3deb0d_0    conda-forge
     pure_eval                 0.2.3              pyhd8ed1ab_0    conda-forge
     py-xgboost                2.1.1           rapidsai_pyhf7e6b63_4    rapidsai
     pyaml                     24.7.0                   pypi_0    pypi
     pyarrow                   16.1.0           py39he8689d4_5    conda-forge
     pyarrow-core              16.1.0          py39hb8ba4b6_5_cpu    conda-forge
     pyarrow-hotfix            0.6                pyhd8ed1ab_0    conda-forge
     pybind11-abi              4                    hd8ed1ab_3    conda-forge
     pycosat                   0.6.6            py39hd1e30aa_0    conda-forge
     pycparser                 2.22               pyhd8ed1ab_0    conda-forge
     pyct                      0.5.0              pyhd8ed1ab_0    conda-forge
     pyee                      8.1.0              pyhd8ed1ab_0    conda-forge
     pygments                  2.18.0             pyhd8ed1ab_0    conda-forge
     pylibcugraph              24.08.00        cuda12_py39_240808_gfc880db0c_0    rapidsai
     pylibraft                 24.08.01        cuda12_py39_240807_g185da16b_0    rapidsai
     pynvjitlink               0.3.0            py39hd269673_0    rapidsai
     pynvml                    11.4.1             pyhd8ed1ab_0    conda-forge
     pyogrio                   0.7.2            py39hcfcd403_1    conda-forge
     pyparsing                 3.1.2              pyhd8ed1ab_0    conda-forge
     pyppeteer                 1.0.2              pyhd8ed1ab_0    conda-forge
     pyproj                    3.6.1            py39hce394fd_4    conda-forge
     pysocks                   1.7.1              pyha2e5f31_6    conda-forge
     python                    3.9.19          h0755675_0_cpython    conda-forge
     python-confluent-kafka    1.9.2            py39hb9d737c_2    conda-forge
     python-dateutil           2.9.0              pyhd8ed1ab_0    conda-forge
     python-fastjsonschema     2.20.0             pyhd8ed1ab_0    conda-forge
     python-json-logger        2.0.7              pyhd8ed1ab_0    conda-forge
     python-tzdata             2024.1             pyhd8ed1ab_0    conda-forge
     python_abi                3.9                      4_cp39    conda-forge
     pytz                      2024.1             pyhd8ed1ab_0    conda-forge
     pyviz_comms               3.0.3              pyhd8ed1ab_0    conda-forge
     pywavelets                1.6.0            py39hd92a3bb_0    conda-forge
     pyyaml                    6.0.2            py39hcd6043d_0    conda-forge
     pyzmq                     26.1.0           py39hbdd323a_0    conda-forge
     qdldl                     0.1.7.post4              pypi_0    pypi
     qhull                     2020.2               h434a139_5    conda-forge
     raft-dask                 24.08.01        cuda12_py39_240807_g185da16b_0    rapidsai
     rapids                    24.08.00        cuda12_py39_240808_g86654f0_0    rapidsai
     rapids-dask-dependency    24.08.00                   py_0    rapidsai
     rapids-xgboost            24.08.00        cuda12_py39_240808_g86654f0_0    rapidsai
     rdma-core                 53.0                 he02047a_0    conda-forge
     re2                       2023.09.01           h7f4b329_2    conda-forge
     readline                  8.2                  h8228510_1    conda-forge
     referencing               0.35.1             pyhd8ed1ab_0    conda-forge
     reproc                    14.2.4.post0         hd590300_1    conda-forge
     reproc-cpp                14.2.4.post0         h59595ed_1    conda-forge
     requests                  2.32.3             pyhd8ed1ab_0    conda-forge
     rfc3339-validator         0.1.4              pyhd8ed1ab_0    conda-forge
     rfc3986-validator         0.1.1              pyh9f0ad1d_0    conda-forge
     rich                      13.7.1             pyhd8ed1ab_0    conda-forge
     rmm                       24.08.02        cuda12_py39_240807_g5aa8ebe6_0    rapidsai
     rpds-py                   0.20.0           py39h5cde264_0    conda-forge
     ruamel.yaml               0.18.6           py39hd1e30aa_0    conda-forge
     ruamel.yaml.clib          0.2.8            py39hd1e30aa_0    conda-forge
     s2n                       1.5.0                h3400bea_0    conda-forge
     scikit-image              0.19.3           py39h4661b88_2    conda-forge
     scikit-learn              1.5.1            py39hf7b0125_0    conda-forge
     scikit-optimize           0.10.2                   pypi_0    pypi
     scikit-survival           0.23.0                   pypi_0    pypi
     scipy                     1.13.1           py39haf93ffa_0    conda-forge
     send2trash                1.8.3              pyh0d859eb_0    conda-forge
     setuptools                72.1.0             pyhd8ed1ab_0    conda-forge
     shapely                   2.0.4            py39h5a575da_1    conda-forge
     simpervisor               1.0.0              pyhd8ed1ab_0    conda-forge
     six                       1.16.0             pyh6c4a22f_0    conda-forge
     snappy                    1.2.1                ha2e4443_0    conda-forge
     sniffio                   1.3.1              pyhd8ed1ab_0    conda-forge
     sortedcontainers          2.4.0              pyhd8ed1ab_0    conda-forge
     soupsieve                 2.5                pyhd8ed1ab_1    conda-forge
     spdlog                    1.12.0               hd2e6256_2    conda-forge
     sqlite                    3.46.0               h6d4b2fc_0    conda-forge
     stack_data                0.6.2              pyhd8ed1ab_0    conda-forge
     streamz                   0.6.4              pyh6c4a22f_0    conda-forge
     tblib                     3.0.0              pyhd8ed1ab_0    conda-forge
     terminado                 0.18.1             pyh0d859eb_0    conda-forge
     threadpoolctl             3.5.0              pyhc1e730c_0    conda-forge
     tifffile                  2020.6.3                   py_0    conda-forge
     tiledb                    2.18.2               h99f50a1_1    conda-forge
     tinycss2                  1.3.0              pyhd8ed1ab_0    conda-forge
     tk                        8.6.13          noxft_h4845f30_101    conda-forge
     toolz                     0.12.1             pyhd8ed1ab_0    conda-forge
     tornado                   6.4.1            py39hd3abc70_0    conda-forge
     tqdm                      4.66.5             pyhd8ed1ab_0    conda-forge
     traitlets                 5.14.3             pyhd8ed1ab_0    conda-forge
     treelite                  4.3.0            py39hcb22a9e_0    conda-forge
     types-python-dateutil     2.9.0.20240316     pyhd8ed1ab_0    conda-forge
     typing-extensions         4.12.2               hd8ed1ab_0    conda-forge
     typing_extensions         4.12.2             pyha770c72_0    conda-forge
     typing_utils              0.1.0              pyhd8ed1ab_0    conda-forge
     tzcode                    2024a                h3f72095_0    conda-forge
     tzdata                    2024a                h0c530f3_0    conda-forge
     uc-micro-py               1.0.3              pyhd8ed1ab_0    conda-forge
     ucx                       1.15.0               hda83522_8    conda-forge
     ucx-proc                  1.0.0                       gpu    rapidsai
     ucx-py                    0.39.02         py39_240807_g314778e_0    rapidsai
     ucxx                      0.39.01         cuda12_py3.9_240807_g19a66b4_0    rapidsai
     unicodedata2              15.1.0           py39hd1e30aa_0    conda-forge
     uri-template              1.3.0              pyhd8ed1ab_0    conda-forge
     uriparser                 0.9.8                hac33072_0    conda-forge
     urllib3                   1.26.19            pyhd8ed1ab_0    conda-forge
     wcwidth                   0.2.13             pyhd8ed1ab_0    conda-forge
     webcolors                 24.8.0             pyhd8ed1ab_0    conda-forge
     webencodings              0.5.1              pyhd8ed1ab_2    conda-forge
     websocket-client          1.8.0              pyhd8ed1ab_0    conda-forge
     websockets                10.4             py39hb9d737c_1    conda-forge
     wheel                     0.44.0             pyhd8ed1ab_0    conda-forge
     xarray                    2024.7.0           pyhd8ed1ab_0    conda-forge
     xerces-c                  3.2.5                hac6953d_0    conda-forge
     xgboost                   2.1.1           rapidsai_pyh26c0410_4    rapidsai
     xorg-kbproto              1.0.7             h7f98852_1002    conda-forge
     xorg-libice               1.1.1                hd590300_0    conda-forge
     xorg-libsm                1.2.4                h7391055_0    conda-forge
     xorg-libx11               1.8.9                h8ee46fc_0    conda-forge
     xorg-libxau               1.0.11               hd590300_0    conda-forge
     xorg-libxdmcp             1.1.3                h7f98852_0    conda-forge
     xorg-libxext              1.3.4                h0b41bf4_2    conda-forge
     xorg-libxrender           0.9.11               hd590300_0    conda-forge
     xorg-renderproto          0.11.1            h7f98852_1002    conda-forge
     xorg-xextproto            7.3.0             h0b41bf4_1003    conda-forge
     xorg-xproto               7.0.31            h7f98852_1007    conda-forge
     xyzservices               2024.6.0           pyhd8ed1ab_0    conda-forge
     xz                        5.2.6                h166bdaf_0    conda-forge
     yaml                      0.2.5                h7f98852_2    conda-forge
     yaml-cpp                  0.8.0                h59595ed_0    conda-forge
     yarl                      1.9.4            py39hd1e30aa_0    conda-forge
     zeromq                    4.3.5                h75354e8_4    conda-forge
     zict                      3.0.0              pyhd8ed1ab_0    conda-forge
     zipp                      3.20.0             pyhd8ed1ab_0    conda-forge
     zlib                      1.3.1                h4ab18f5_1    conda-forge
     zstandard                 0.23.0           py39h623c9ba_0    conda-forge
     zstd                      1.5.6                ha6fb4c9_0    conda-forge

Could you please help me see what the possible errors are and how I can fix them next?