beckernick
commented
3 years ago In your first section of code, CuPy is using its own memory allocator which by default uses a memory pool. You can read more about it here, but the key point is that using a memory pool significantly increases performance in this example.
When you import cudf, the memory allocator for CuPy is changed to instead use the RAPIDS Memory Manager. This is usually a good decision, as we ideally want to use the same available memory pool so we don't compete for memory between cuDF and CuPy. However, by default, cuDF does not use a memory pool.
If you run
import cudf
cudf.set_allocator(pool=True)rather than just importing cudf, things should go fast again. This will create a memory pool sized to about half of your GPU's total memory.
I'm going to close this issue as answered, but please feel free to re-open if things don't work as expected!
Importing cuDF at any point significantly slows down cuPy operations (even if cuDF is not used). I think that this is an issue with cuDF rather than cuPy, but let me know if I should post it to the cuPy github instead.
Code to reproduce the error:
Output:
The slowdown is not limited to the function above or matrix multiplication (e.g. the time of argsorting 2M floats goes from 773 us to 2780 us). Its magnitude depends on the executed function and the size of the cuPy arrays used. I've seen everything between +10% to +1000%.
The same can be observed if I use two separate scripts for profiling (one with only cuPy imported and the other with both cuDF and cuPy imported - the order of the imports doesn't matter).
Environment overview (please complete the following information)
Environment details
Click here to see environment details
**git*** print_env.sh: 10: [: unexpected operator Not inside a git repository ***OS Information*** ID="ec2" VERSION="20200928-407" PRETTY_NAME="Debian GNU/Linux 11 (bullseye)" NAME="Debian GNU/Linux" VERSION_ID="11" VERSION="11 (bullseye)" VERSION_CODENAME=bullseye ID=debian HOME_URL="https://www.debian.org/" SUPPORT_URL="https://www.debian.org/support" BUG_REPORT_URL="https://bugs.debian.org/" Linux ip-192-168-204-153 5.10.0-7-cloud-amd64 #1 SMP Debian 5.10.40-1 (2021-05-28) x86_64 GNU/Linux ***GPU Information*** Tue Jul 13 18:29:23 2021 +-----------------------------------------------------------------------------+ | NVIDIA-SMI 465.19.01 Driver Version: 465.19.01 CUDA Version: 11.3 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |===============================+======================+======================| | 0 NVIDIA Tesla V1... On | 00000000:00:1E.0 Off | 0 | | N/A 38C P0 26W / 300W | 0MiB / 16160MiB | 0% Default | | | | N/A | +-------------------------------+----------------------+----------------------+ +-----------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=============================================================================| | No running processes found | +-----------------------------------------------------------------------------+ ***CPU*** Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian Address sizes: 46 bits physical, 48 bits virtual CPU(s): 8 On-line CPU(s) list: 0-7 Thread(s) per core: 2 Core(s) per socket: 4 Socket(s): 1 NUMA node(s): 1 Vendor ID: GenuineIntel CPU family: 6 Model: 79 Model name: Intel(R) Xeon(R) CPU E5-2686 v4 @ 2.30GHz Stepping: 1 CPU MHz: 2723.948 CPU max MHz: 3000.0000 CPU min MHz: 1200.0000 BogoMIPS: 4600.18 Hypervisor vendor: Xen Virtualization type: full L1d cache: 128 KiB L1i cache: 128 KiB L2 cache: 1 MiB L3 cache: 45 MiB NUMA node0 CPU(s): 0-7 Vulnerability Itlb multihit: KVM: Mitigation: VMX unsupported Vulnerability L1tf: Mitigation; PTE Inversion Vulnerability Mds: Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown Vulnerability Meltdown: Mitigation; PTI Vulnerability Spec store bypass: Vulnerable Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Vulnerability Spectre v2: Mitigation; Full generic retpoline, STIBP disabled, RSB filling Vulnerability Srbds: Not affected Vulnerability Tsx async abort: Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch cpuid_fault invpcid_single pti fsgsbase bmi1 hle avx2 smep bmi2 erms invpcid rtm rdseed adx xsaveopt ***CMake*** /usr/bin/cmake cmake version 3.18.4 CMake suite maintained and supported by Kitware (kitware.com/cmake). ***g++*** /usr/bin/g++ g++ (Debian 10.2.1-6) 10.2.1 20210110 Copyright (C) 2020 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. ***nvcc*** /usr/local/cuda-11.2/bin/nvcc nvcc: NVIDIA (R) Cuda compiler driver Copyright (c) 2005-2020 NVIDIA Corporation Built on Mon_Nov_30_19:08:53_PST_2020 Cuda compilation tools, release 11.2, V11.2.67 Build cuda_11.2.r11.2/compiler.29373293_0 ***Python*** /db_vol/hb_work/miniconda3/envs/rapids-21.06/bin/python Python 3.8.10 ***Environment Variables*** PATH : /db_vol/hb_work/miniconda3/envs/rapids-21.06/bin:/db_vol/hb_work/miniconda3/condabin:/usr/local/cuda-11.2/bin:/db_vol/hb_work/.pyenv/shims:/db_vol/hb_work/.pyenv/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games LD_LIBRARY_PATH : /opt/OpenBLAS/lib/:/usr/local/cuda-11.2/lib64/:/usr/lib/: NUMBAPRO_NVVM : NUMBAPRO_LIBDEVICE : CONDA_PREFIX : /db_vol/hb_work/miniconda3/envs/rapids-21.06 PYTHON_PATH : ***conda packages*** conda is /db_vol/hb_work/miniconda3/condabin/conda /db_vol/hb_work/miniconda3/condabin/conda # packages in environment at /db_vol/hb_work/miniconda3/envs/rapids-21.06: # # Name Version Build Channel _libgcc_mutex 0.1 conda_forge conda-forge _openmp_mutex 4.5 1_gnu conda-forge abseil-cpp 20210324.2 h9c3ff4c_0 conda-forge argon2-cffi 20.1.0 py38h497a2fe_2 conda-forge arrow-cpp 1.0.1 py38hf0991f3_43_cuda conda-forge arrow-cpp-proc 3.0.0 cuda conda-forge async_generator 1.10 py_0 conda-forge attrs 21.2.0 pyhd8ed1ab_0 conda-forge aws-c-cal 0.5.11 h95a6274_0 conda-forge aws-c-common 0.6.2 h7f98852_0 conda-forge aws-c-event-stream 0.2.7 h3541f99_13 conda-forge aws-c-io 0.10.5 hfb6a706_0 conda-forge aws-checksums 0.1.11 ha31a3da_7 conda-forge aws-sdk-cpp 1.8.186 hb4091e7_3 conda-forge backcall 0.2.0 pyh9f0ad1d_0 conda-forge backports 1.0 py_2 conda-forge backports.functools_lru_cache 1.6.4 pyhd8ed1ab_0 conda-forge blazingsql 21.6.0 pypi_0 pypi bleach 3.3.0 pyh44b312d_0 conda-forge bokeh 2.3.2 py38h578d9bd_0 conda-forge boost-cpp 1.72.0 h312852a_5 conda-forge brotli 1.0.9 h7f98852_5 conda-forge brotli-bin 1.0.9 h7f98852_5 conda-forge brotlipy 0.7.0 py38h497a2fe_1001 conda-forge bzip2 1.0.8 h7f98852_4 conda-forge c-ares 1.17.1 h7f98852_1 conda-forge ca-certificates 2021.5.30 ha878542_0 conda-forge cachetools 4.2.2 pyhd8ed1ab_0 conda-forge certifi 2021.5.30 py38h578d9bd_0 conda-forge cffi 1.14.5 py38ha65f79e_0 conda-forge chardet 4.0.0 py38h578d9bd_1 conda-forge click 8.0.1 py38h578d9bd_0 conda-forge cloudpickle 1.6.0 py_0 conda-forge cryptography 3.4.7 py38ha5dfef3_0 conda-forge cudatoolkit 11.2.72 h2bc3f7f_0 nvidia cudf 21.06.01 cuda_11.2_py38_g101fc0fda4_2 rapidsai cupy 9.2.0 py38ha69542f_0 conda-forge cyrus-sasl 2.1.27 h230043b_2 conda-forge cytoolz 0.11.0 py38h497a2fe_3 conda-forge dask 2021.5.0 pyhd8ed1ab_0 conda-forge dask-core 2021.5.0 pyhd8ed1ab_0 conda-forge dask-cuda 21.06.00 py38_0 rapidsai dask-cudf 21.06.01 py38_g101fc0fda4_2 rapidsai debugpy 1.3.0 py38h709712a_0 conda-forge decorator 5.0.9 pyhd8ed1ab_0 conda-forge defusedxml 0.7.1 pyhd8ed1ab_0 conda-forge distributed 2021.5.0 py38h578d9bd_0 conda-forge dlpack 0.5 h9c3ff4c_0 conda-forge entrypoints 0.3 pyhd8ed1ab_1003 conda-forge fastavro 1.4.2 py38h497a2fe_0 conda-forge fastrlock 0.6 py38h709712a_1 conda-forge freetype 2.10.4 h0708190_1 conda-forge fsspec 2021.6.1 pyhd8ed1ab_0 conda-forge future 0.18.2 py38h578d9bd_3 conda-forge gflags 2.2.2 he1b5a44_1004 conda-forge glog 0.5.0 h48cff8f_0 conda-forge google-cloud-cpp 1.28.0 ha08a4db_1 conda-forge greenlet 1.1.0 py38h709712a_0 conda-forge grpc-cpp 1.38.1 h36ce80c_0 conda-forge heapdict 1.0.1 py_0 conda-forge icu 68.1 h58526e2_0 conda-forge idna 2.10 pyh9f0ad1d_0 conda-forge importlib-metadata 4.6.1 py38h578d9bd_0 conda-forge ipykernel 6.0.1 py38hd0cf306_0 conda-forge ipython 7.25.0 py38hd0cf306_1 conda-forge ipython_genutils 0.2.0 py_1 conda-forge ipywidgets 7.6.3 pyhd3deb0d_0 conda-forge jbig 2.1 h7f98852_2003 conda-forge jedi 0.18.0 py38h578d9bd_2 conda-forge jinja2 3.0.1 pyhd8ed1ab_0 conda-forge jpeg 9d h36c2ea0_0 conda-forge jpype1 1.3.0 py38h1fd1430_0 conda-forge jsonschema 3.2.0 pyhd8ed1ab_3 conda-forge jupyter_client 6.1.12 pyhd8ed1ab_0 conda-forge jupyter_core 4.7.1 py38h578d9bd_0 conda-forge jupyterlab_pygments 0.1.2 pyh9f0ad1d_0 conda-forge jupyterlab_widgets 1.0.0 pyhd8ed1ab_1 conda-forge krb5 1.19.1 hcc1bbae_0 conda-forge lcms2 2.12 hddcbb42_0 conda-forge ld_impl_linux-64 2.36.1 hea4e1c9_0 conda-forge lerc 2.2.1 h9c3ff4c_0 conda-forge libblas 3.9.0 9_openblas conda-forge libbrotlicommon 1.0.9 h7f98852_5 conda-forge libbrotlidec 1.0.9 h7f98852_5 conda-forge libbrotlienc 1.0.9 h7f98852_5 conda-forge libcblas 3.9.0 9_openblas conda-forge libcrc32c 1.1.1 h9c3ff4c_2 conda-forge libcudf 21.06.01 cuda11.2_g101fc0fda4_2 rapidsai libcurl 7.77.0 h2574ce0_0 conda-forge libdeflate 1.7 h7f98852_5 conda-forge libedit 3.1.20191231 he28a2e2_2 conda-forge libev 4.33 h516909a_1 conda-forge libevent 2.1.10 hcdb4288_3 conda-forge libffi 3.3 h58526e2_2 conda-forge libgcc-ng 9.3.0 h2828fa1_19 conda-forge libgfortran-ng 9.3.0 hff62375_19 conda-forge libgfortran5 9.3.0 hff62375_19 conda-forge libgomp 9.3.0 h2828fa1_19 conda-forge libhwloc 2.3.0 h5e5b7d1_1 conda-forge libiconv 1.16 h516909a_0 conda-forge liblapack 3.9.0 9_openblas conda-forge libllvm10 10.0.1 he513fc3_3 conda-forge libnghttp2 1.43.0 h812cca2_0 conda-forge libntlm 1.4 h7f98852_1002 conda-forge libopenblas 0.3.15 pthreads_h8fe5266_1 conda-forge libpng 1.6.37 h21135ba_2 conda-forge libpq 13.3 hd57d9b9_0 conda-forge libprotobuf 3.16.0 h780b84a_0 conda-forge librmm 21.06.00 cuda11.2_gee432a0_0 rapidsai libsodium 1.0.18 h36c2ea0_1 conda-forge libssh2 1.9.0 ha56f1ee_6 conda-forge libstdcxx-ng 9.3.0 h6de172a_19 conda-forge libthrift 0.14.2 he6d91bd_1 conda-forge libtiff 4.3.0 hf544144_1 conda-forge libutf8proc 2.6.1 h7f98852_0 conda-forge libwebp-base 1.2.0 h7f98852_2 conda-forge libxml2 2.9.12 h72842e0_0 conda-forge llvmlite 0.36.0 py38h4630a5e_0 conda-forge locket 0.2.0 py_2 conda-forge lz4-c 1.9.3 h9c3ff4c_0 conda-forge markupsafe 2.0.1 py38h497a2fe_0 conda-forge matplotlib-inline 0.1.2 pyhd8ed1ab_2 conda-forge mistune 0.8.4 py38h497a2fe_1004 conda-forge msgpack-python 1.0.2 py38h1fd1430_1 conda-forge nbclient 0.5.3 pyhd8ed1ab_0 conda-forge nbconvert 6.1.0 py38h578d9bd_0 conda-forge nbformat 5.1.3 pyhd8ed1ab_0 conda-forge ncurses 6.2 h58526e2_4 conda-forge nest-asyncio 1.5.1 pyhd8ed1ab_0 conda-forge netifaces 0.10.9 py38h497a2fe_1003 conda-forge nlohmann_json 3.9.1 h9c3ff4c_1 conda-forge notebook 6.4.0 pyha770c72_0 conda-forge numba 0.53.1 py38h8b71fd7_1 conda-forge numpy 1.21.0 py38h9894fe3_0 conda-forge nvtx 0.2.3 py38h497a2fe_0 conda-forge olefile 0.46 pyh9f0ad1d_1 conda-forge openjdk 8.0.282 h7f98852_0 conda-forge openjpeg 2.4.0 hb52868f_1 conda-forge openssl 1.1.1k h7f98852_0 conda-forge orc 1.6.9 h58a87f1_0 conda-forge packaging 21.0 pyhd8ed1ab_0 conda-forge pandas 1.2.5 py38h1abd341_0 conda-forge pandoc 2.14.0.3 h7f98852_0 conda-forge pandocfilters 1.4.2 py_1 conda-forge parquet-cpp 1.5.1 2 conda-forge parso 0.8.2 pyhd8ed1ab_0 conda-forge partd 1.2.0 pyhd8ed1ab_0 conda-forge pexpect 4.8.0 pyh9f0ad1d_2 conda-forge pickleshare 0.7.5 py_1003 conda-forge pillow 8.3.0 py38h8e6f84c_0 conda-forge pip 21.1.3 pyhd8ed1ab_0 conda-forge prometheus_client 0.11.0 pyhd8ed1ab_0 conda-forge prompt-toolkit 3.0.19 pyha770c72_0 conda-forge protobuf 3.16.0 py38h709712a_0 conda-forge psutil 5.8.0 py38h497a2fe_1 conda-forge ptyprocess 0.7.0 pyhd3deb0d_0 conda-forge pyarrow 1.0.1 py38hb53058b_43_cuda conda-forge pycparser 2.20 pyh9f0ad1d_2 conda-forge pygments 2.9.0 pyhd8ed1ab_0 conda-forge pyhive 0.6.4 pyhd8ed1ab_0 conda-forge pynvml 11.0.0 pyhd8ed1ab_0 conda-forge pyopenssl 20.0.1 pyhd8ed1ab_0 conda-forge pyparsing 2.4.7 pyh9f0ad1d_0 conda-forge pyrsistent 0.17.3 py38h497a2fe_2 conda-forge pysocks 1.7.1 py38h578d9bd_3 conda-forge python 3.8.10 h49503c6_1_cpython conda-forge python-dateutil 2.8.1 py_0 conda-forge python_abi 3.8 2_cp38 conda-forge pytz 2021.1 pyhd8ed1ab_0 conda-forge pyyaml 5.4.1 py38h497a2fe_0 conda-forge pyzmq 22.1.0 py38h2035c66_0 conda-forge re2 2021.06.01 h9c3ff4c_0 conda-forge readline 8.1 h46c0cb4_0 conda-forge requests 2.25.1 pyhd3deb0d_0 conda-forge rmm 21.06.00 cuda_11.2_py38_gee432a0_0 rapidsai s2n 1.0.10 h9b69904_0 conda-forge sasl 0.3.1 py38h709712a_0 conda-forge send2trash 1.7.1 pyhd8ed1ab_0 conda-forge setuptools 49.6.0 py38h578d9bd_3 conda-forge six 1.16.0 pyh6c4a22f_0 conda-forge snappy 1.1.8 he1b5a44_3 conda-forge sortedcontainers 2.4.0 pyhd8ed1ab_0 conda-forge spdlog 1.8.5 h4bd325d_0 conda-forge sqlalchemy 1.4.20 py38h497a2fe_0 conda-forge sqlite 3.36.0 h9cd32fc_0 conda-forge tblib 1.7.0 pyhd8ed1ab_0 conda-forge terminado 0.10.1 py38h578d9bd_0 conda-forge testpath 0.5.0 pyhd8ed1ab_0 conda-forge thrift 0.13.0 py38h709712a_2 conda-forge thrift_sasl 0.3.0 py38h1e0a361_1002 conda-forge tk 8.6.10 h21135ba_1 conda-forge toolz 0.11.1 py_0 conda-forge tornado 6.1 py38h497a2fe_1 conda-forge tqdm 4.61.2 pyhd8ed1ab_0 conda-forge traitlets 5.0.5 py_0 conda-forge typing_extensions 3.10.0.0 pyha770c72_0 conda-forge ucx 1.9.0+gcd9efd3 cuda11.2_0 rapidsai ucx-proc 1.0.0 gpu rapidsai ucx-py 0.20.0 py38_gcd9efd3_0 rapidsai urllib3 1.26.6 pyhd8ed1ab_0 conda-forge wcwidth 0.2.5 pyh9f0ad1d_2 conda-forge webencodings 0.5.1 py_1 conda-forge wheel 0.36.2 pyhd3deb0d_0 conda-forge widgetsnbextension 3.5.1 py38h578d9bd_4 conda-forge xz 5.2.5 h516909a_1 conda-forge yaml 0.2.5 h516909a_0 conda-forge zeromq 4.3.4 h9c3ff4c_0 conda-forge zict 2.0.0 py_0 conda-forge zipp 3.5.0 pyhd8ed1ab_0 conda-forge zlib 1.2.11 h516909a_1010 conda-forge zstd 1.5.0 ha95c52a_0 conda-forge