ChuckHastings
commented
3 years ago I cannot precisely execute your sample script, however I was able to eventually come to a failure condition that is consistent with your issue. Let me describe the steps I took to validate that my adaptation was reasonable, then I will describe what I think the issue and resolution would be.
1) I assume you're running in a Jupiter notebook (based on the description of the error and the syntax of the reproducing steps you provided)
2) osmnx is not installed by default in the docker containers. I assumed that if I simply added pip install osmnx that this would install an acceptable version
3) I received an html 400 error when executing the ox.graph_from_place call as provided above. I commented out the custom filter assuming that something in that definition might not be quite right. This allowed me to download a graph, although the graph is likely somewhat larger than you intend. Not sure why I couldn't execute your code as is... perhaps a configuration or version issue with osmnx
With these changes/assumptions, I was able to reproduce the kernel crashing that you describe. Note, as your print statements suggest, you arrive at the call to from_cudf_edgelist. I added another print statement after that call to verify what I suspected which is that you fail during the call to from_cudf_edgelist.
The error I was seeing is due to the fact that you set renumber=False. If you set renumber=True or if you do not specify the renumber parameter then the code should work and the memory utilization should be very small (at least for this graph).
Let me describe what's happening a bit. Graph libraries tend to fall into two categories. Graph libraries (like networkx which you include in your notebook) which are focused on a simple user experience and hide details within the implementation, and graph libraries which are focused on performance. Tools like networkx either choose data structures that will work for any type of user data or they internally translate into more efficient representations without demonstrating that to the user. High performance graph libraries generally require the user to specify the graph in a form that is ready for their consumption.
cuGraph attempts to satisfy both sets of user needs, and renumbering is one of the areas where we expose underlying implementation to users so that if they know that their data is already optimally numbered then they can save the cost of renumbering their data. There are a collection of well known/used data sets in the graph analytic community that are already numbered in a way that makes executing graph analytics efficient. For those types of data sets setting renumber=False will save unnecessary overhead.
The data I see in your cd_edge Dataframe is not appropriate for bypassing the renumbering step. Renumbering is going to take your vertices (there are 30,925 in the unfiltered data set I downloaded... probably smaller with your filters activated) and renumber them to be in the half-open range [0, 30925). All data structures used within the SSSP algorithm are then arrays that are of size 30,925 (about 240 KB of GPU memory each). Without renumbering, the vertex ids will be used as is. The largest vertex id in the unfiltered input set is 8362075761. Consequently each temporary array created in SSSP without renumbering (haven't looked to see how many temporaries SSSP uses) would be of size 8,362,075,762 (about 64 GB of GPU memory) which far exceeds the memory capacity of your GPU.
https://docs.rapids.ai/api/cugraph/stable/api.html#cugraph.structure.graph.Graph.from_cudf_edgelist provides the documentation for this call. There is a somewhat simpler description than I described above. Please let me know if this makes sense and resolves your issue. We can certainly look into clarifying something in the documentation if you find it confusing or incomplete.
iceboy910447
Describe the bug Dear rapids developer,
I encountered a memory spike during usage of SSSP and unexpected death of the python kernel. Spike from 2GB to above 10GB at the start of the function. The function was working during prior usage with rapids 0.16 and CUDA 10.1, even with graphs several times bigger than this one
Could you please doublecheck?
Thanks, iceboy910447
Steps/Code to reproduce bug
import nvstrings
import numpy as np import pandas as pd import cudf, cuml import dask_cudf import io, requests import math import gc import time
Plotting
import matplotlib.pyplot as plt import seaborn as sns
Learning
import cuspatial import cugraph import pandas as pd pd.version #1.1.4 import rmm rmm.mr.set_current_device_resource(rmm.mr.ManagedMemoryResource()) cudf.set_allocator(allocator="managed") import multiprocessing as mp import numpy as np import networkx as nx import osmnx as ox import requests import matplotlib import matplotlib.cm as cm import matplotlib.colors as colors %matplotlib inline ox.config(use_cache=True, log_console=True,timeout=1000) ox.version
G = ox.graph_from_place('Manhattan, New York, USA',simplify=True,truncate_by_edge=True,custom_filter='["area"!~"yes"]["highway"~"motorway|trunk|primary|secondary|tertiary|residential|motorway_link|trunk_link|primary_link|secondary_link|tertiary_link"]["highway"!~"service"]') #4503 Nodes, 9668 Edges print("Build edgelist") pd_edge = nx.to_pandas_edgelist(G) # generating a pandas dataframe containing all edgeinformations based on the given Graph print(pd_edge.dtypes) print("Build edgelist 2") pd_edge_zwei = pd_edge[["source","target","length"]] # simplification of the dataframe print(pd_edge_zwei.source.min()) print(pd_edge_zwei.target.min())
print(pd_edge_zwei.length.max()) print("Build edgelist 3") cd_edge = cudf.from_pandas(pd_edge_zwei) cd_edge.length = cd_edge.length.astype("float32") print(cd_edge.dtypes) G_zwei = cugraph.Graph() # building a cugraph Graph from the edgelist dataframe
G_zwei.from_cudf_edgelist(cd_edge, source = 'source',destination = 'target', edge_attr = 'travel_time', renumber = False)
G_zwei.add_edge_list(cd_edge['source'],cd_edge['target'],cd_edge['length'])
print("sleep wird durchgeführt") time.sleep(10) print("sleep wird beendet") G_zwei.from_cudf_edgelist(cd_edge,source='source', destination='target', edge_attr='length', renumber=False) distances = cugraph.sssp(G_zwei,42422026) # Single Source Shortest Path Calculation for one source node distances.head()
Expected behavior A clear and concise description of what you expected to happen.
Environment overview (please complete the following information) docker rapids 0.17 ubuntu 20.04 cuda 11.2
docker pull rapidsai/rapidsai-core:cuda11.0-runtime-ubuntu18.04-py3.8 docker run --gpus all --rm -it -p 8888:8888 -p 8787:8787 -p 8786:8786 \ rapidsai/rapidsai-core:cuda11.0-runtime-ubuntu18.04-py3.8
Environment details Please run and paste the output of the
cugraph/print_env.shscript here, to gather any other relevant environment detailsClick here to see environment details
**git*** commit 892a8e1c4b13ba8cc9fd706461dd4a505dc27e97 (grafted, HEAD, origin/branch-0.16) Author: Ray Douglass <3107146+raydouglass@users.noreply.github.com> Date: Wed Oct 21 15:39:58 2020 -0400 Merge pull request #1240 from jakirkham/req_ucx_proc Require `ucx-proc=*=gpu` **git submodules*** ***OS Information*** DISTRIB_ID=Ubuntu DISTRIB_RELEASE=18.04 DISTRIB_CODENAME=bionic DISTRIB_DESCRIPTION="Ubuntu 18.04.5 LTS" NAME="Ubuntu" VERSION="18.04.5 LTS (Bionic Beaver)" ID=ubuntu ID_LIKE=debian PRETTY_NAME="Ubuntu 18.04.5 LTS" VERSION_ID="18.04" HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" VERSION_CODENAME=bionic UBUNTU_CODENAME=bionic Linux db3a54a94461 5.8.0-38-generic #43~20.04.1-Ubuntu SMP Tue Jan 12 16:39:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux ***GPU Information*** Sat Jan 23 18:57:51 2021 +-----------------------------------------------------------------------------+ | NVIDIA-SMI 460.32.03 Driver Version: 460.32.03 CUDA Version: 11.2 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |===============================+======================+======================| | 0 GeForce RTX 208... Off | 00000000:65:00.0 On | N/A | | 0% 29C P8 36W / 250W | 1842MiB / 11018MiB | 28% Default | | | | N/A | +-------------------------------+----------------------+----------------------+ +-----------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=============================================================================| +-----------------------------------------------------------------------------+ ***CPU*** Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 20 On-line CPU(s) list: 0-19 Thread(s) per core: 2 Core(s) per socket: 10 Socket(s): 1 NUMA node(s): 1 Vendor ID: GenuineIntel CPU family: 6 Model: 85 Model name: Intel(R) Core(TM) i9-9900X CPU @ 3.50GHz Stepping: 4 CPU MHz: 1200.003 CPU max MHz: 4500.0000 CPU min MHz: 1200.0000 BogoMIPS: 6999.82 Virtualization: VT-x L1d cache: 32K L1i cache: 32K L2 cache: 1024K L3 cache: 19712K NUMA node0 CPU(s): 0-19 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single pti ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req md_clear flush_l1d ***CMake*** ***g++*** ***nvcc*** ***Python*** /opt/conda/envs/rapids/bin/python Python 3.8.6 ***Environment Variables*** PATH : /opt/conda/envs/rapids/bin:/opt/conda/condabin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin LD_LIBRARY_PATH : /opt/conda/envs/rapids/lib:/usr/local/nvidia/lib:/usr/local/nvidia/lib64 NUMBAPRO_NVVM : NUMBAPRO_LIBDEVICE : CONDA_PREFIX : /opt/conda/envs/rapids PYTHON_PATH : ***conda packages*** /opt/conda/condabin/conda # packages in environment at /opt/conda/envs/rapids: # # Name Version Build Channel _libgcc_mutex 0.1 conda_forge conda-forge _openmp_mutex 4.5 1_llvm conda-forge abseil-cpp 20200225.2 he1b5a44_2 conda-forge aiohttp 3.7.3 py38h25fe258_0 conda-forge appdirs 1.4.4 pyh9f0ad1d_0 conda-forge argon2-cffi 20.1.0 py38h25fe258_2 conda-forge arrow-cpp 1.0.1 py38h9444e92_16_cuda conda-forge arrow-cpp-proc 2.0.0 cuda conda-forge async-timeout 3.0.1 py_1000 conda-forge async_generator 1.10 py_0 conda-forge attrs 20.3.0 pyhd3deb0d_0 conda-forge aws-c-common 0.4.59 h36c2ea0_1 conda-forge aws-c-event-stream 0.1.6 had2084c_6 conda-forge aws-checksums 0.1.10 h4e93380_0 conda-forge aws-sdk-cpp 1.8.70 h57dc084_1 conda-forge backcall 0.2.0 pyh9f0ad1d_0 conda-forge backports 1.0 py_2 conda-forge backports.functools_lru_cache 1.6.1 py_0 conda-forge blas 2.17 openblas conda-forge bleach 3.2.1 pyh9f0ad1d_0 conda-forge bokeh 2.2.3 py38h578d9bd_0 conda-forge boost 1.72.0 py38h1e42940_1 conda-forge boost-cpp 1.72.0 h8e57a91_0 conda-forge boto3 1.16.34 pyhd8ed1ab_0 conda-forge botocore 1.19.34 pyhd8ed1ab_0 conda-forge brotli 1.0.9 he1b5a44_3 conda-forge brotlipy 0.7.0 py38h8df0ef7_1001 conda-forge bzip2 1.0.8 h516909a_3 conda-forge c-ares 1.17.1 h36c2ea0_0 conda-forge ca-certificates 2020.12.5 ha878542_0 conda-forge cairo 1.16.0 hcf35c78_1003 conda-forge certifi 2020.12.5 py38h578d9bd_0 conda-forge cffi 1.14.4 py38ha312104_0 conda-forge cfitsio 3.470 h2e3daa1_7 conda-forge chardet 3.0.4 py38h924ce5b_1008 conda-forge click 7.1.2 pyh9f0ad1d_0 conda-forge click-plugins 1.1.1 py_0 conda-forge cligj 0.7.1 pyhd8ed1ab_0 conda-forge cloudpickle 1.6.0 py_0 conda-forge colorcet 2.0.1 py_0 conda-forge cryptography 3.2.1 py38h7699a38_0 conda-forge cudatoolkit 11.0.221 h6bb024c_0 nvidia cudf 0.17.0a201209 cuda_11.0_py38_gbd321d1e93_382 rapidsai-nightly cudf_kafka 0.17.0a201209 py38_gbd321d1e93_382 rapidsai-nightly cudnn 8.0.0 cuda11.0_0 nvidia cugraph 0.17.0a201210 py38_ge205fd07_289 rapidsai-nightly cuml 0.17.0a201210 cuda11.0_py38_g2c0aacf44_173 rapidsai-nightly cupy 8.0.0 py38hb7c6141_0 rapidsai-nightly curl 7.71.1 he644dc0_8 conda-forge cusignal 0.17.0a201210 py37_g44e57f5_47 rapidsai-nightly cuspatial 0.17.0a201210 py38_g9aeebc8_23 rapidsai-nightly custreamz 0.17.0a201209 py38_gbd321d1e93_382 rapidsai-nightly cuxfilter 0.17.0a201203 py38_g01ee2c8_96 rapidsai-nightly cycler 0.10.0 py_2 conda-forge cyrus-sasl 2.1.27 h063b49f_1 conda-forge cython 0.29.21 py38h348cfbe_1 conda-forge cytoolz 0.11.0 py38h25fe258_1 conda-forge dask 2.30.0 py_0 conda-forge dask-core 2.30.0 py_0 conda-forge dask-cuda 0.17.0a201207 py38_56 rapidsai-nightly dask-cudf 0.17.0a201209 py38_gbd321d1e93_382 rapidsai-nightly dask-glm 0.2.0 py_1 conda-forge dask-labextension 3.0.0 py_0 conda-forge dask-ml 1.7.0 py_0 conda-forge datashader 0.11.1 pyh9f0ad1d_0 conda-forge datashape 0.5.4 py_1 conda-forge decorator 4.4.2 py_0 conda-forge defusedxml 0.6.0 py_0 conda-forge descartes 1.1.0 pypi_0 pypi distributed 2.30.1 py38h578d9bd_0 conda-forge dlpack 0.3 he1b5a44_1 conda-forge entrypoints 0.3 pyhd8ed1ab_1003 conda-forge expat 2.2.9 he1b5a44_2 conda-forge fa2 0.3.5 py38h1e0a361_0 conda-forge faiss-proc 1.0.0 cuda rapidsai-nightly fastavro 1.2.0 py38h25fe258_0 conda-forge fastrlock 0.5 py38h950e882_1 conda-forge filterpy 1.4.5 py_1 conda-forge fiona 1.8.13 py38h033e0f6_1 conda-forge fontconfig 2.13.1 h86ecdb6_1001 conda-forge freetype 2.10.4 h7ca028e_0 conda-forge freexl 1.0.5 h516909a_1002 conda-forge fsspec 0.8.4 py_0 conda-forge gdal 3.0.4 py38h172510d_10 conda-forge geopandas 0.8.1 py_0 conda-forge geos 3.8.1 he1b5a44_0 conda-forge geotiff 1.6.0 h05acad5_0 conda-forge gettext 0.19.8.1 hf34092f_1004 conda-forge gflags 2.2.2 he1b5a44_1004 conda-forge giflib 5.2.1 h36c2ea0_2 conda-forge git 2.29.2 pl5320h78be0e9_1 conda-forge glib 2.66.3 h58526e2_0 conda-forge glog 0.4.0 h49b9bf7_3 conda-forge gpuci-tools 0.2.1 5 gpuci grpc-cpp 1.33.2 h1870a98_1 conda-forge hdf4 4.2.13 hf30be14_1003 conda-forge hdf5 1.10.6 nompi_h7c3c948_1111 conda-forge heapdict 1.0.1 py_0 conda-forge holoviews 1.14.0 pyhd3deb0d_0 conda-forge icu 64.2 he1b5a44_1 conda-forge idna 2.10 pyh9f0ad1d_0 conda-forge importlib-metadata 3.1.1 pyhd8ed1ab_0 conda-forge importlib_metadata 3.1.1 hd8ed1ab_0 conda-forge iniconfig 1.1.1 pyh9f0ad1d_0 conda-forge ipykernel 5.4.0 py38h81c977d_0 conda-forge ipython 7.15.0 py38h32f6830_0 conda-forge ipython_genutils 0.2.0 py_1 conda-forge ipywidgets 7.5.1 pyh9f0ad1d_1 conda-forge jedi 0.17.2 py38h578d9bd_1 conda-forge jinja2 2.11.2 pyh9f0ad1d_0 conda-forge jmespath 0.10.0 pyh9f0ad1d_0 conda-forge joblib 0.17.0 py_0 conda-forge jpeg 9d h36c2ea0_0 conda-forge json-c 0.13.1 hbfbb72e_1002 conda-forge json5 0.9.5 pyh9f0ad1d_0 conda-forge jsonschema 3.2.0 py_2 conda-forge jupyter-server-proxy 1.5.0 py_0 conda-forge jupyter_client 6.1.7 py_0 conda-forge jupyter_core 4.7.0 py38h578d9bd_0 conda-forge jupyterlab 2.1.5 py_0 conda-forge jupyterlab-nvdashboard 0.5.0a200819 py_0 rapidsai-nightly jupyterlab_pygments 0.1.2 pyh9f0ad1d_0 conda-forge jupyterlab_server 1.2.0 py_0 conda-forge kealib 1.4.13 h33137a7_1 conda-forge kiwisolver 1.3.1 py38h82cb98a_0 conda-forge krb5 1.17.2 h926e7f8_0 conda-forge lcms2 2.11 hcbb858e_1 conda-forge ld_impl_linux-64 2.35.1 hed1e6ac_0 conda-forge libblas 3.8.0 17_openblas conda-forge libcblas 3.8.0 17_openblas conda-forge libcudf 0.17.0a201210 cuda11.0_g00ca24625e_383 rapidsai-nightly libcudf_kafka 0.17.0a201209 gbd321d1e93_382 rapidsai-nightly libcugraph 0.17.0a201210 cuda11.0_ge205fd07_289 rapidsai-nightly libcuml 0.17.0a201210 cuda11.0_g2c0aacf44_173 rapidsai-nightly libcumlprims 0.17.0a201030 cuda11.0_g1fa28a5_8 rapidsai-nightly libcurl 7.71.1 hcdd3856_8 conda-forge libcuspatial 0.17.0a201210 cuda11.0_g9aeebc8_23 rapidsai-nightly libdap4 3.20.6 h1d1bd15_1 conda-forge libedit 3.1.20191231 he28a2e2_2 conda-forge libev 4.33 h516909a_1 conda-forge libevent 2.1.10 hcdb4288_3 conda-forge libfaiss 1.6.3 h328c4c8_1_cuda rapidsai-nightly libffi 3.2.1 he1b5a44_1007 conda-forge libgcc-ng 7.5.0 h5dbcf3e_17 conda-forge libgcrypt 1.8.7 h36c2ea0_0 conda-forge libgdal 3.0.4 he6a97d6_10 conda-forge libgfortran-ng 7.5.0 hae1eefd_17 conda-forge libgfortran4 7.5.0 hae1eefd_17 conda-forge libglib 2.66.3 hbe7bbb4_0 conda-forge libgpg-error 1.39 he1b5a44_0 conda-forge libgsasl 1.8.0 2 conda-forge libhwloc 2.3.0 h5e5b7d1_1 conda-forge libiconv 1.16 h516909a_0 conda-forge libkml 1.3.0 hd79254b_1012 conda-forge liblapack 3.8.0 17_openblas conda-forge liblapacke 3.8.0 17_openblas conda-forge libllvm10 10.0.1 he513fc3_3 conda-forge libnetcdf 4.7.4 nompi_hefab0ff_106 conda-forge libnghttp2 1.41.0 h8cfc5f6_2 conda-forge libntlm 1.4 h516909a_1002 conda-forge libopenblas 0.3.10 pthreads_hb3c22a3_5 conda-forge libpng 1.6.37 h21135ba_2 conda-forge libpq 12.3 h255efa7_3 conda-forge libprotobuf 3.13.0.1 h8b12597_0 conda-forge librdkafka 1.5.2 h40bdf00_0 conda-forge librmm 0.17.0a201210 cuda11.0_gb8c8310_60 rapidsai-nightly libsodium 1.0.18 h36c2ea0_1 conda-forge libspatialindex 1.9.3 he1b5a44_3 conda-forge libspatialite 4.3.0a h2482549_1038 conda-forge libssh2 1.9.0 hab1572f_5 conda-forge libstdcxx-ng 7.5.0 h2ae2ef3_17 conda-forge libthrift 0.13.0 h5aa387f_6 conda-forge libtiff 4.1.0 h4f3a223_6 conda-forge libutf8proc 2.6.0 h36c2ea0_0 conda-forge libuuid 2.32.1 h14c3975_1000 conda-forge libuv 1.40.0 hd18ef5c_0 conda-forge libwebp 1.1.0 h76fa15c_4 conda-forge libwebp-base 1.1.0 h36c2ea0_3 conda-forge libxcb 1.13 h14c3975_1002 conda-forge libxgboost 1.3.0dev.rapidsai0.17 cuda11.0_0 rapidsai-nightly libxml2 2.9.10 hee79883_0 conda-forge llvm-openmp 11.0.0 hfc4b9b4_1 conda-forge llvmlite 0.34.0 py38h4f45e52_2 conda-forge locket 0.2.0 py_2 conda-forge lz4-c 1.9.2 he1b5a44_3 conda-forge markdown 3.3.3 pyh9f0ad1d_0 conda-forge markupsafe 1.1.1 py38h8df0ef7_2 conda-forge matplotlib-base 3.3.3 py38h5c7f4ab_0 conda-forge mistune 0.8.4 py38h25fe258_1002 conda-forge more-itertools 8.6.0 pyhd8ed1ab_0 conda-forge msgpack-python 1.0.0 py38h82cb98a_2 conda-forge multidict 4.7.5 py38h1e0a361_2 conda-forge multipledispatch 0.6.0 py_0 conda-forge munch 2.5.0 py_0 conda-forge nbclient 0.5.1 py_0 conda-forge nbconvert 6.0.7 py38h578d9bd_3 conda-forge nbformat 5.0.8 py_0 conda-forge nccl 2.7.8.1 h4962215_100 nvidia ncurses 6.2 h58526e2_4 conda-forge nest-asyncio 1.4.3 pyhd8ed1ab_0 conda-forge networkx 2.5 py_0 conda-forge nodejs 12.4.0 he1b5a44_0 conda-forge notebook 6.1.5 py38h578d9bd_0 conda-forge numba 0.51.2 py38hc5bc63f_0 conda-forge numpy 1.19.4 py38hf0fd68c_1 conda-forge nvtx 0.2.1 py38h1e0a361_2 conda-forge olefile 0.46 pyh9f0ad1d_1 conda-forge openjpeg 2.3.1 h981e76c_3 conda-forge openssl 1.1.1h h516909a_0 conda-forge orc 1.6.5 hd3605a7_0 conda-forge osmnx 1.0.1 pypi_0 pypi packaging 20.7 pyhd3deb0d_0 conda-forge pandas 1.1.4 py38h0ef3d22_0 conda-forge pandoc 2.11.2 h36c2ea0_0 conda-forge pandocfilters 1.4.2 py_1 conda-forge panel 0.10.2 pyhd8ed1ab_0 conda-forge param 1.10.0 py_0 conda-forge parquet-cpp 1.5.1 2 conda-forge parso 0.7.1 pyh9f0ad1d_0 conda-forge partd 1.1.0 py_0 conda-forge patsy 0.5.1 py_0 conda-forge pcre 8.44 he1b5a44_0 conda-forge perl 5.32.0 h36c2ea0_0 conda-forge pexpect 4.8.0 pyh9f0ad1d_2 conda-forge pickleshare 0.7.5 py_1003 conda-forge pillow 8.0.1 py38h70fbd49_0 conda-forge pip 20.3.1 pyhd8ed1ab_0 conda-forge pixman 0.38.0 h516909a_1003 conda-forge pluggy 0.13.1 py38h924ce5b_3 conda-forge poppler 0.87.0 h4190859_1 conda-forge poppler-data 0.4.10 0 conda-forge postgresql 12.3 hc2f5b80_3 conda-forge proj 7.0.0 h966b41f_5 conda-forge prometheus_client 0.9.0 pyhd3deb0d_0 conda-forge prompt-toolkit 3.0.8 pyha770c72_0 conda-forge protobuf 3.13.0.1 py38hadf7658_1 conda-forge psutil 5.7.3 py38h8df0ef7_0 conda-forge pthread-stubs 0.4 h36c2ea0_1001 conda-forge ptyprocess 0.6.0 py_1001 conda-forge py 1.9.0 pyh9f0ad1d_0 conda-forge py-xgboost 1.3.0dev.rapidsai0.17 cuda11.0py38_0 rapidsai-nightly pyarrow 1.0.1 py38h3e2403a_16_cuda conda-forge pycparser 2.20 pyh9f0ad1d_2 conda-forge pyct 0.4.6 py_0 conda-forge pyct-core 0.4.6 py_0 conda-forge pydeck 0.5.0 pyh9f0ad1d_0 conda-forge pyee 7.0.4 pyh9f0ad1d_0 conda-forge pygments 2.7.3 pyhd8ed1ab_0 conda-forge pynvml 8.0.4 py_1 conda-forge pyopenssl 20.0.0 pyhd8ed1ab_0 conda-forge pyparsing 2.4.7 pyh9f0ad1d_0 conda-forge pyppeteer 0.2.2 py_1 conda-forge pyproj 2.6.1.post1 py38h7521cb9_0 conda-forge pyrsistent 0.17.3 py38h25fe258_1 conda-forge pysocks 1.7.1 py38h924ce5b_2 conda-forge pytest 6.1.2 py38h578d9bd_0 conda-forge python 3.8.6 h852b56e_0_cpython conda-forge python-confluent-kafka 1.5.0 py38h1e0a361_0 conda-forge python-dateutil 2.8.1 py_0 conda-forge python_abi 3.8 1_cp38 conda-forge pytz 2020.4 pyhd8ed1ab_0 conda-forge pyviz_comms 0.7.6 pyh9f0ad1d_0 conda-forge pyyaml 5.3.1 py38h8df0ef7_1 conda-forge pyzmq 20.0.0 py38h1d1b12f_1 conda-forge rapids 0.17.0a201210 cuda11.0_py38_gdc7a92a_161 rapidsai-nightly rapids-xgboost 0.17.0a201210 cuda11.0_py38_gdc7a92a_161 rapidsai-nightly re2 2020.11.01 h58526e2_0 conda-forge readline 8.0 he28a2e2_2 conda-forge requests 2.25.0 pyhd3deb0d_0 conda-forge rmm 0.17.0a201210 cuda_11.0_py38_gb8c8310_60 rapidsai-nightly rtree 0.9.4 py38h02d302b_2 conda-forge s3fs 0.4.2 py_0 conda-forge s3transfer 0.3.3 py_3 conda-forge scikit-learn 0.23.1 py38h3a94b23_0 conda-forge scipy 1.5.1 py38h18bccfc_0 conda-forge seaborn 0.11.0 ha770c72_1 conda-forge seaborn-base 0.11.0 pyhd8ed1ab_1 conda-forge send2trash 1.5.0 py_0 conda-forge setuptools 49.6.0 py38h924ce5b_2 conda-forge shapely 1.7.1 py38hc7361b7_1 conda-forge simpervisor 0.3 py_1 conda-forge six 1.15.0 pyh9f0ad1d_0 conda-forge snappy 1.1.8 he1b5a44_3 conda-forge sortedcontainers 2.3.0 pyhd8ed1ab_0 conda-forge spdlog 1.7.0 hc9558a2_2 conda-forge sqlite 3.33.0 h4cf870e_1 conda-forge statsmodels 0.12.1 py38h0b5ebd8_1 conda-forge streamz 0.6.1 pyhd3deb0d_1 conda-forge tbb 2020.2 hc9558a2_0 conda-forge tblib 1.6.0 py_0 conda-forge terminado 0.9.1 py38h32f6830_1 conda-forge testpath 0.4.4 py_0 conda-forge threadpoolctl 2.1.0 pyh5ca1d4c_0 conda-forge tiledb 1.7.7 h8efa9f0_3 conda-forge tk 8.6.10 hed695b0_1 conda-forge toml 0.10.2 pyhd8ed1ab_0 conda-forge toolz 0.11.1 py_0 conda-forge tornado 6.1 py38h25fe258_0 conda-forge tqdm 4.54.1 pyhd8ed1ab_0 conda-forge traitlets 5.0.5 py_0 conda-forge treelite 0.93 py38hadf7658_3 conda-forge treelite-runtime 0.93 pypi_0 pypi typing-extensions 3.7.4.3 0 conda-forge typing_extensions 3.7.4.3 py_0 conda-forge tzcode 2020a h516909a_0 conda-forge ucx 1.8.1+g6b29558 cuda11.0_0 rapidsai-nightly ucx-proc 1.0.0 gpu rapidsai-nightly ucx-py 0.17.0a201210 py38_g6b29558_27 rapidsai-nightly umap-learn 0.4.6 py38h32f6830_0 conda-forge urllib3 1.25.11 py_0 conda-forge wcwidth 0.2.5 pyh9f0ad1d_2 conda-forge webencodings 0.5.1 py_1 conda-forge websockets 8.1 py38h1e0a361_2 conda-forge wheel 0.36.1 pyhd3deb0d_0 conda-forge widgetsnbextension 3.5.1 py38h578d9bd_4 conda-forge xarray 0.16.2 pyhd8ed1ab_0 conda-forge xerces-c 3.2.2 h8412b87_1004 conda-forge xgboost 1.3.0dev.rapidsai0.17 cuda11.0py38_0 rapidsai-nightly xorg-kbproto 1.0.7 h14c3975_1002 conda-forge xorg-libice 1.0.10 h516909a_0 conda-forge xorg-libsm 1.2.3 h84519dc_1000 conda-forge xorg-libx11 1.6.12 h516909a_0 conda-forge xorg-libxau 1.0.9 h14c3975_0 conda-forge xorg-libxdmcp 1.1.3 h516909a_0 conda-forge xorg-libxext 1.3.4 h516909a_0 conda-forge xorg-libxrender 0.9.10 h516909a_1002 conda-forge xorg-renderproto 0.11.1 h14c3975_1002 conda-forge xorg-xextproto 7.3.0 h14c3975_1002 conda-forge xorg-xproto 7.0.31 h14c3975_1007 conda-forge xz 5.2.5 h516909a_1 conda-forge yaml 0.2.5 h516909a_0 conda-forge yarl 1.6.3 py38h25fe258_0 conda-forge zeromq 4.3.3 h58526e2_3 conda-forge zict 2.0.0 py_0 conda-forge zipp 3.4.0 py_0 conda-forge zlib 1.2.11 h516909a_1010 conda-forge zstd 1.4.5 h6597ccf_2 conda-forge **Additional context** Free VRAM before function start 9GB