search

rapidsai / rmm

RAPIDS Memory Manager
https://docs.rapids.ai/api/rmm/stable/
Apache License 2.0
492 stars 198 forks source link

[BUG] RMM logging appears not to work with `DeviceBuffer`s #281

Closed jakirkham closed 4 years ago

jakirkham commented 4 years ago

Describe the bug

When trying to use RMM's logging with rmm_alloc and rmm_free in Python, this works fine. However when trying to use logging with RMM's DeviceBuffer in Python, logging appears not to capture any activity.

Steps/Code to reproduce bug

Working case with the log using rmm_alloc and rmm_free:

In [1]: import rmm                                                              

In [2]: rmm.reinitialize(pool_allocator=True, 
   ...:                  initial_pool_size=int(1 * 2**20), 
   ...:                  logging=True)                                          
Out[2]: 0

In [3]: ptr = rmm._lib.rmm_alloc(10, 0)                                         

In [4]: rmm.csv_log()                                                           
Out[4]: 'Event Type,Device ID,Address,Stream,Size (bytes),Free Memory,Total Memory,Current Allocs,Start,End,Elapsed,Location\nAlloc,0,0x7f4b99a00000,0,10,1048064,1048576,1,13.8011,13.8011,3.711e-06,/datasets/jkirkham/miniconda/envs/rapids13dev/lib/python3.7/site-packages/IPython/core/interactiveshell.py:3331\n'

In [5]: rmm._lib.rmm_free(ptr, 0)                                               

In [6]: rmm.csv_log()                                                           
Out[6]: 'Event Type,Device ID,Address,Stream,Size (bytes),Free Memory,Total Memory,Current Allocs,Start,End,Elapsed,Location\nAlloc,0,0x7f4b99a00000,0,10,1048064,1048576,1,13.8011,13.8011,3.711e-06,/datasets/jkirkham/miniconda/envs/rapids13dev/lib/python3.7/site-packages/IPython/core/interactiveshell.py:3331\nFree,0,0x7f4b99a00000,0,0,1048576,1048576,0,30.8792,30.8792,3.602e-06,/datasets/jkirkham/miniconda/envs/rapids13dev/lib/python3.7/site-packages/IPython/core/interactiveshell.py:3331\n'

Not working case with DeviceBuffer:

In [1]: import rmm                                                              

In [2]: rmm.reinitialize(pool_allocator=True, 
   ...:                  initial_pool_size=int(1 * 2**20), 
   ...:                  logging=True)                                          
Out[2]: 0

In [3]: rdb = rmm.DeviceBuffer(size=32)                                         

In [4]: rmm.csv_log()                                                           
Out[4]: 'Event Type,Device ID,Address,Stream,Size (bytes),Free Memory,Total Memory,Current Allocs,Start,End,Elapsed,Location\n'

Expected behavior

The allocation would still be tracked in the DeviceBuffer case.

Environment details (please complete the following information):

``` **git*** commit dcca68b122284383716d0b49e47db0970cd085c4 (HEAD -> branch-0.13, rapidsai/branch-0.13) Merge: 8ed78b9 e777301 Author: Keith Kraus Date: Wed Feb 12 18:17:04 2020 -0500 Merge pull request #280 from jakirkham/skip_malloc_offset Drop last host `malloc` and `free` calls in Cython ***OS Information*** DGX_NAME="DGX Server" DGX_PRETTY_NAME="NVIDIA DGX Server" DGX_SWBUILD_DATE="2018-03-20" DGX_SWBUILD_VERSION="3.1.6" DGX_COMMIT_ID="1b0f58ecbf989820ce745a9e4836e1de5eea6cfd" DGX_SERIAL_NUMBER=QTFCOU8310024 DGX_OTA_VERSION="3.1.7" DGX_OTA_DATE="Thu Sep 27 20:07:53 PDT 2018" DISTRIB_ID=Ubuntu DISTRIB_RELEASE=16.04 DISTRIB_CODENAME=xenial DISTRIB_DESCRIPTION="Ubuntu 16.04.5 LTS" NAME="Ubuntu" VERSION="16.04.5 LTS (Xenial Xerus)" ID=ubuntu ID_LIKE=debian PRETTY_NAME="Ubuntu 16.04.5 LTS" VERSION_ID="16.04" HOME_URL="http://www.ubuntu.com/" SUPPORT_URL="http://help.ubuntu.com/" BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/" VERSION_CODENAME=xenial UBUNTU_CODENAME=xenial Linux dgx16 4.4.0-135-generic #161-Ubuntu SMP Mon Aug 27 10:45:01 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux ***GPU Information*** Wed Feb 12 16:50:19 2020 +-----------------------------------------------------------------------------+ | NVIDIA-SMI 440.33.01 Driver Version: 440.33.01 CUDA Version: 10.2 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | |===============================+======================+======================| | 0 Tesla V100-SXM2... On | 00000000:06:00.0 Off | 0 | | N/A 39C P0 45W / 300W | 0MiB / 32510MiB | 0% Default | +-------------------------------+----------------------+----------------------+ | 1 Tesla V100-SXM2... On | 00000000:07:00.0 Off | 0 | | N/A 41C P0 45W / 300W | 0MiB / 32510MiB | 0% Default | +-------------------------------+----------------------+----------------------+ | 2 Tesla V100-SXM2... On | 00000000:0A:00.0 Off | 0 | | N/A 41C P0 44W / 300W | 0MiB / 32510MiB | 0% Default | +-------------------------------+----------------------+----------------------+ | 3 Tesla V100-SXM2... On | 00000000:0B:00.0 Off | 0 | | N/A 39C P0 45W / 300W | 0MiB / 32510MiB | 0% Default | +-------------------------------+----------------------+----------------------+ | 4 Tesla V100-SXM2... On | 00000000:85:00.0 Off | 0 | | N/A 41C P0 45W / 300W | 0MiB / 32510MiB | 0% Default | +-------------------------------+----------------------+----------------------+ | 5 Tesla V100-SXM2... On | 00000000:86:00.0 Off | 0 | | N/A 41C P0 43W / 300W | 0MiB / 32510MiB | 0% Default | +-------------------------------+----------------------+----------------------+ | 6 Tesla V100-SXM2... On | 00000000:89:00.0 Off | 0 | | N/A 43C P0 44W / 300W | 0MiB / 32510MiB | 0% Default | +-------------------------------+----------------------+----------------------+ | 7 Tesla V100-SXM2... On | 00000000:8A:00.0 Off | 0 | | N/A 40C P0 43W / 300W | 0MiB / 32510MiB | 0% Default | +-------------------------------+----------------------+----------------------+ +-----------------------------------------------------------------------------+ | Processes: GPU Memory | | GPU PID Type Process name Usage | |=============================================================================| | No running processes found | +-----------------------------------------------------------------------------+ ***CPU*** Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 80 On-line CPU(s) list: 0-79 Thread(s) per core: 2 Core(s) per socket: 20 Socket(s): 2 NUMA node(s): 2 Vendor ID: GenuineIntel CPU family: 6 Model: 79 Model name: Intel(R) Xeon(R) CPU E5-2698 v4 @ 2.20GHz Stepping: 1 CPU MHz: 2752.578 CPU max MHz: 3600.0000 CPU min MHz: 1200.0000 BogoMIPS: 4391.49 Virtualization: VT-x L1d cache: 32K L1i cache: 32K L2 cache: 256K L3 cache: 51200K NUMA node0 CPU(s): 0-19,40-59 NUMA node1 CPU(s): 20-39,60-79 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb invpcid_single intel_pt ssbd ibrs ibpb stibp kaiser tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm rdseed adx smap xsaveopt cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts flush_l1d ***CMake*** /datasets/jkirkham/miniconda/envs/rapids13dev/bin/cmake cmake version 3.16.3 CMake suite maintained and supported by Kitware (kitware.com/cmake). ***g++*** /datasets/jkirkham/miniconda/envs/rapids13dev/bin/g++ g++ (crosstool-NG 1.23.0.450-d54ae) 7.3.0 Copyright (C) 2017 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. ***nvcc*** /datasets/jkirkham/miniconda/envs/rapids13dev/bin/nvcc nvcc: NVIDIA (R) Cuda compiler driver Copyright (c) 2005-2019 NVIDIA Corporation Built on Wed_Oct_23_19:24:38_PDT_2019 Cuda compilation tools, release 10.2, V10.2.89 ***Python*** /datasets/jkirkham/miniconda/envs/rapids13dev/bin/python Python 3.7.6 ***Environment Variables*** PATH : /datasets/jkirkham/miniconda/envs/rapids13dev/bin:/datasets/jkirkham/miniconda/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/local/cuda/bin LD_LIBRARY_PATH : NUMBAPRO_NVVM : NUMBAPRO_LIBDEVICE : CONDA_PREFIX : /datasets/jkirkham/miniconda/envs/rapids13dev PYTHON_PATH : ***conda packages*** /datasets/jkirkham/miniconda/condabin/conda # packages in environment at /datasets/jkirkham/miniconda/envs/rapids13dev: # # Name Version Build Channel _libgcc_mutex 0.1 conda_forge conda-forge _openmp_mutex 4.5 0_gnu conda-forge aiohttp 3.6.2 py37h516909a_0 conda-forge alabaster 0.7.12 py_0 conda-forge alembic 1.4.0 py_0 conda-forge appdirs 1.4.3 py_1 conda-forge arrow 0.15.5 py37_0 conda-forge arrow-cpp 0.15.0 py37h090bef1_2 conda-forge aspy.yaml 1.3.0 py_0 conda-forge async-timeout 3.0.1 py_1000 conda-forge async_generator 1.10 py_0 conda-forge attrs 19.3.0 py_0 conda-forge autoconf 2.69 pl526h14c3975_9 conda-forge automake 1.16.1 pl526_1003 conda-forge babel 2.8.0 py_0 conda-forge backcall 0.1.0 py_0 conda-forge binutils-meta 1.0.4 0 conda-forge binutils_impl_linux-64 2.33.1 h53a641e_8 conda-forge binutils_linux-64 2.33.1 h9595d00_16 conda-forge black 19.10b0 py37_0 conda-forge blas 2.14 openblas conda-forge bleach 3.1.0 py_0 conda-forge blinker 1.4 py_1 conda-forge bokeh 1.4.0 py37_0 conda-forge boost 1.70.0 py37h9de70de_1 conda-forge boost-cpp 1.70.0 h8e57a91_2 conda-forge brotli 1.0.7 he1b5a44_1000 conda-forge bzip2 1.0.8 h516909a_2 conda-forge c-ares 1.15.0 h516909a_1001 conda-forge c-compiler 1.0.4 h516909a_0 conda-forge ca-certificates 2019.11.28 hecc5488_0 conda-forge cached-property 1.5.1 py_0 conda-forge cairo 1.16.0 hfb77d84_1002 conda-forge certifi 2019.11.28 py37_0 conda-forge certipy 0.1.3 py_0 conda-forge cffi 1.13.2 py37h8022711_0 conda-forge cfgv 3.0.0 py_0 conda-forge chardet 3.0.4 py37_1003 conda-forge click 7.0 py_0 conda-forge cloudpickle 1.2.2 py_1 conda-forge cmake 3.16.3 h28c56e5_0 conda-forge compilers 1.0.4 0 conda-forge configurable-http-proxy 4.2.0 node13_he01fd0c_2 conda-forge cryptography 2.8 py37h72c5cf5_1 conda-forge cudatoolkit 10.1.243 h6bb024c_0 defaults cudf 0.13.0a200212 py37_1692 rapidsai-nightly cudnn 7.6.5 cuda10.1_0 defaults cuml 0.13.0a200212 cuda10.1_py37_181 rapidsai-nightly cupy 7.1.1 py37h0c141eb_1 conda-forge cxx-compiler 1.0.4 hc9558a2_0 conda-forge cython 0.29.15 py37he1b5a44_0 conda-forge cytoolz 0.10.1 py37h516909a_0 conda-forge dask 2.10.1 py_0 conda-forge dask-core 2.10.1 py_0 conda-forge dask-cuda 0.13.0b200212 py37_12 rapidsai-nightly dask-cudf 0.13.0a200212 py37_1692 rapidsai-nightly dask-labextension 1.1.0 py_0 conda-forge dbus 1.13.6 he372182_0 conda-forge decorator 4.4.1 py_0 conda-forge defusedxml 0.6.0 py_0 conda-forge distributed 2.10.0 py_0 conda-forge dlpack 0.2 he1b5a44_1 conda-forge docutils 0.16 py37_0 conda-forge double-conversion 3.1.5 he1b5a44_2 conda-forge editdistance 0.5.3 py37he1b5a44_0 conda-forge entrypoints 0.3 py37_1000 conda-forge expat 2.2.9 he1b5a44_2 conda-forge fastavro 0.22.9 py37h516909a_0 conda-forge fastrlock 0.4 py37he1b5a44_1000 conda-forge flake8 3.7.9 py37_0 conda-forge flatbuffers 1.11.0 he1b5a44_0 conda-forge fontconfig 2.13.1 h86ecdb6_1001 conda-forge fortran-compiler 1.0.4 he991be0_0 conda-forge freetype 2.10.0 he983fc9_1 conda-forge fribidi 1.0.5 h516909a_1002 conda-forge fsspec 0.6.2 py_0 conda-forge gcc_impl_linux-64 7.3.0 hd420e75_5 conda-forge gcc_linux-64 7.3.0 h553295d_16 conda-forge gettext 0.19.8.1 hc5be6a0_1002 conda-forge gflags 2.2.2 he1b5a44_1002 conda-forge gfortran_impl_linux-64 7.3.0 hdf63c60_5 conda-forge gfortran_linux-64 7.3.0 h553295d_16 conda-forge glib 2.58.3 py37h6f030ca_1002 conda-forge glog 0.4.0 he1b5a44_1 conda-forge graphite2 1.3.13 hf484d3e_1000 conda-forge graphviz 2.42.3 h0511662_0 conda-forge grpc-cpp 1.23.0 h18db393_0 conda-forge gst-plugins-base 1.14.5 h0935bb2_2 conda-forge gstreamer 1.14.5 h36ae1b5_2 conda-forge gxx_impl_linux-64 7.3.0 hdf63c60_5 conda-forge gxx_linux-64 7.3.0 h553295d_16 conda-forge harfbuzz 2.4.0 h9f30f68_3 conda-forge heapdict 1.0.1 py_0 conda-forge hypothesis 5.5.1 py_0 conda-forge icu 64.2 he1b5a44_1 conda-forge identify 1.4.11 py_0 conda-forge idna 2.8 py37_1000 conda-forge imagesize 1.2.0 py_0 conda-forge importlib_metadata 1.5.0 py37_0 conda-forge ipykernel 5.1.4 py37h5ca1d4c_0 conda-forge ipython 7.12.0 py37h5ca1d4c_0 conda-forge ipython_genutils 0.2.0 py_1 conda-forge ipywidgets 7.5.1 py_0 conda-forge isort 4.3.21 py37_0 conda-forge jedi 0.16.0 py37_0 conda-forge jinja2 2.11.1 py_0 conda-forge joblib 0.14.1 py_0 conda-forge jpeg 9c h14c3975_1001 conda-forge json5 0.9.0 py_0 conda-forge jsonschema 3.2.0 py37_0 conda-forge jupyter 1.0.0 py_2 conda-forge jupyter-server-proxy 1.2.0 py_0 conda-forge jupyter_client 5.3.4 py37_1 conda-forge jupyter_console 6.0.0 py_0 conda-forge jupyter_core 4.6.2 py37_0 conda-forge jupyter_telemetry 0.0.5 py_0 conda-forge jupyterhub 1.1.0 py37_2 conda-forge jupyterhub-base 1.1.0 py37_2 conda-forge jupyterlab 1.2.6 py_0 conda-forge jupyterlab-nvdashboard 0.1.11200212 py_12 rapidsai-nightly jupyterlab_server 1.0.6 py_0 conda-forge krb5 1.16.4 h2fd8d38_0 conda-forge ld_impl_linux-64 2.33.1 h53a641e_8 conda-forge libblas 3.8.0 14_openblas conda-forge libcblas 3.8.0 14_openblas conda-forge libclang 9.0.1 default_hde54327_0 conda-forge libcudf 0.13.0a200212 cuda10.1_1692 rapidsai-nightly libcuml 0.13.0a200212 cuda10.1_181 rapidsai-nightly libcumlprims 0.13.0a200212 cuda10.1_9 rapidsai-nightly libcurl 7.68.0 hda55be3_0 conda-forge libedit 3.1.20170329 hf8c457e_1001 conda-forge libevent 2.1.10 h72c5cf5_0 conda-forge libffi 3.2.1 he1b5a44_1006 conda-forge libgcc-ng 9.2.0 h24d8f2e_2 conda-forge libgfortran-ng 7.3.0 hdf63c60_5 conda-forge libgomp 9.2.0 h24d8f2e_2 conda-forge libhwloc 2.1.0 h3c4fd83_0 conda-forge libiconv 1.15 h516909a_1005 conda-forge liblapack 3.8.0 14_openblas conda-forge liblapacke 3.8.0 14_openblas conda-forge libllvm8 8.0.1 hc9558a2_0 conda-forge libllvm9 9.0.1 hc9558a2_0 conda-forge libnvstrings 0.13.0a200212 cuda10.1_1692 rapidsai-nightly libopenblas 0.3.7 h5ec1e0e_6 conda-forge libpng 1.6.37 hed695b0_0 conda-forge libprotobuf 3.8.0 h8b12597_0 conda-forge libsodium 1.0.17 h516909a_0 conda-forge libssh2 1.8.2 h22169c7_2 conda-forge libstdcxx-ng 9.2.0 hdf63c60_2 conda-forge libtiff 4.1.0 hfc65ed5_0 conda-forge libtool 2.4.6 h14c3975_1002 conda-forge libuuid 2.32.1 h14c3975_1000 conda-forge libuv 1.34.0 h516909a_0 conda-forge libxcb 1.13 h14c3975_1002 conda-forge libxkbcommon 0.10.0 he1b5a44_0 conda-forge libxml2 2.9.10 hee79883_0 conda-forge llvmlite 0.31.0 py37h8b12597_0 conda-forge locket 0.2.0 py_2 conda-forge lz4-c 1.8.3 he1b5a44_1001 conda-forge m4 1.4.18 h14c3975_1001 conda-forge make 4.2.1 h14c3975_2004 conda-forge mako 1.1.0 py_0 conda-forge markupsafe 1.1.1 py37h516909a_0 conda-forge mccabe 0.6.1 py_1 conda-forge mistune 0.8.4 py37h516909a_1000 conda-forge more-itertools 8.2.0 py_0 conda-forge msgpack-python 0.6.2 py37hc9558a2_0 conda-forge multidict 4.7.4 py37h516909a_0 conda-forge mypy_extensions 0.4.3 py37_0 conda-forge nbconvert 5.6.1 py37_0 conda-forge nbformat 5.0.4 py_0 conda-forge nbsphinx 0.5.1 py_0 conda-forge nccl 2.5.7.1 h51cf6c1_0 conda-forge ncurses 6.1 hf484d3e_1002 conda-forge nodeenv 1.3.5 py_0 conda-forge nodejs 13.8.0 h10a4023_0 conda-forge notebook 6.0.3 py37_0 conda-forge nspr 4.25 he1b5a44_0 conda-forge nss 3.47 he751ad9_0 conda-forge numba 0.48.0 py37hb3f55d8_0 conda-forge numpy 1.18.1 py37h95a1406_0 conda-forge numpydoc 0.9.2 py_0 conda-forge nvcc_linux-64 10.2 hc6a2c23_6 conda-forge nvstrings 0.13.0a200212 py37_1692 rapidsai-nightly oauthlib 3.0.1 py_0 conda-forge olefile 0.46 py_0 conda-forge openssl 1.1.1d h516909a_0 conda-forge packaging 20.1 py_0 conda-forge pamela 1.0.0 py_0 conda-forge pandas 0.25.3 py37hb3f55d8_0 conda-forge pandoc 2.9.1.1 0 conda-forge pandocfilters 1.4.2 py_1 conda-forge pango 1.42.4 ha030887_1 conda-forge parquet-cpp 1.5.1 2 conda-forge parso 0.6.1 py_0 conda-forge partd 1.1.0 py_0 conda-forge pathspec 0.7.0 py_0 conda-forge pcre 8.44 he1b5a44_0 conda-forge perl 5.26.2 h516909a_1006 conda-forge pexpect 4.8.0 py37_0 conda-forge pickle5 0.0.9 py37h516909a_0 conda-forge pickleshare 0.7.5 py37_1000 conda-forge pillow 7.0.0 py37hefe7db6_0 conda-forge pip 20.0.2 py_2 conda-forge pixman 0.38.0 h516909a_1003 conda-forge pkg-config 0.29.2 h516909a_1006 conda-forge pluggy 0.13.0 py37_0 conda-forge pre_commit 1.18.1 py37_0 conda-forge prometheus_client 0.7.1 py_0 conda-forge prompt_toolkit 2.0.10 py_0 conda-forge psutil 5.6.7 py37h516909a_0 conda-forge pthread-stubs 0.4 h14c3975_1001 conda-forge ptyprocess 0.6.0 py_1001 conda-forge py 1.8.1 py_0 conda-forge pyarrow 0.15.0 py37h8b68381_1 conda-forge pycodestyle 2.5.0 py_0 conda-forge pycparser 2.19 py37_1 conda-forge pycurl 7.43.0.5 py37h16ce93b_0 conda-forge pyflakes 2.1.1 py_0 conda-forge pygments 2.5.2 py_0 conda-forge pyjwt 1.7.1 py_0 conda-forge pynvml 8.0.4 py_0 conda-forge pyopenssl 19.1.0 py37_0 conda-forge pyparsing 2.4.6 py_0 conda-forge pyqt 5.12.3 py37hcca6a23_1 conda-forge pyqt5-sip 4.19.18 pypi_0 pypi pyqtwebengine 5.12.1 pypi_0 pypi pyrsistent 0.15.7 py37h516909a_0 conda-forge pysocks 1.7.1 py37_0 conda-forge pytest 5.3.5 py37_1 conda-forge pytest-asyncio 0.10.0 py37_1000 conda-forge python 3.7.6 h357f687_2 conda-forge python-dateutil 2.8.1 py_0 conda-forge python-editor 1.0.4 py_0 conda-forge python-graphviz 0.13.2 py_0 conda-forge python-json-logger 0.1.11 py_0 conda-forge pytz 2019.3 py_0 conda-forge pyyaml 5.3 py37h516909a_0 conda-forge pyzmq 18.1.1 py37h1768529_0 conda-forge qt 5.12.5 hd8c4c69_1 conda-forge qtconsole 4.6.0 py_0 conda-forge rapidjson 1.1.0 he1b5a44_1002 conda-forge re2 2020.01.01 he1b5a44_0 conda-forge readline 8.0 hf8c457e_0 conda-forge regex 2020.1.8 py37h516909a_0 conda-forge requests 2.22.0 py37_1 conda-forge rhash 1.3.6 h14c3975_1001 conda-forge rmm 0.13.0 pypi_0 pypi ruamel.yaml 0.16.6 py37h516909a_0 conda-forge ruamel.yaml.clib 0.2.0 py37h516909a_0 conda-forge scikit-learn 0.22.1 py37hcdab131_1 conda-forge scipy 1.4.1 py37h921218d_0 conda-forge send2trash 1.5.0 py_0 conda-forge setuptools 45.2.0 py37_0 conda-forge simpervisor 0.3 py_1 conda-forge six 1.14.0 py37_0 conda-forge snappy 1.1.8 he1b5a44_1 conda-forge snowballstemmer 2.0.0 py_0 conda-forge sortedcontainers 2.1.0 py_0 conda-forge sphinx 2.4.1 py_0 conda-forge sphinx_rtd_theme 0.4.3 py_0 conda-forge sphinxcontrib-applehelp 1.0.1 py_0 conda-forge sphinxcontrib-devhelp 1.0.1 py_0 conda-forge sphinxcontrib-htmlhelp 1.0.2 py_0 conda-forge sphinxcontrib-jsmath 1.0.1 py_0 conda-forge sphinxcontrib-qthelp 1.0.2 py_0 conda-forge sphinxcontrib-serializinghtml 1.1.3 py_0 conda-forge sphinxcontrib-websupport 1.1.2 py_0 conda-forge sqlalchemy 1.3.13 py37h516909a_0 conda-forge sqlite 3.30.1 hcee41ef_0 conda-forge streamz 0.5.2 py_0 conda-forge tblib 1.6.0 py_0 conda-forge terminado 0.8.3 py37_0 conda-forge testpath 0.4.4 py_0 conda-forge thrift-cpp 0.12.0 hf3afdfd_1004 conda-forge tk 8.6.10 hed695b0_0 conda-forge toml 0.10.0 py_0 conda-forge toolz 0.10.0 py_0 conda-forge tornado 6.0.3 py37h516909a_3 conda-forge traitlets 4.3.3 py37_0 conda-forge typed-ast 1.4.1 py37h516909a_0 conda-forge typing_extensions 3.7.4.1 py37_0 conda-forge ucx 1.7.0+g9d06c3a cuda10.1_0 rapidsai-nightly ucx-proc 1.0.0 gpu rapidsai-nightly ucx-py 0.13.0a200212+g9d06c3a py37_32 rapidsai-nightly uriparser 0.9.3 he1b5a44_1 conda-forge urllib3 1.25.7 py37_0 conda-forge valgrind 3.15.0 he513fc3_0 conda-forge virtualenv 16.7.5 py_0 conda-forge wcwidth 0.1.8 py_0 conda-forge webencodings 0.5.1 py_1 conda-forge wheel 0.34.2 py_1 conda-forge widgetsnbextension 3.5.1 py37_0 conda-forge xorg-kbproto 1.0.7 h14c3975_1002 conda-forge xorg-libice 1.0.10 h516909a_0 conda-forge xorg-libsm 1.2.3 h84519dc_1000 conda-forge xorg-libx11 1.6.9 h516909a_0 conda-forge xorg-libxau 1.0.9 h14c3975_0 conda-forge xorg-libxdmcp 1.1.3 h516909a_0 conda-forge xorg-libxext 1.3.4 h516909a_0 conda-forge xorg-libxpm 3.5.13 h516909a_0 conda-forge xorg-libxrender 0.9.10 h516909a_1002 conda-forge xorg-libxt 1.1.5 h516909a_1003 conda-forge xorg-renderproto 0.11.1 h14c3975_1002 conda-forge xorg-xextproto 7.3.0 h14c3975_1002 conda-forge xorg-xproto 7.0.31 h14c3975_1007 conda-forge xz 5.2.4 h14c3975_1001 conda-forge yaml 0.2.2 h516909a_1 conda-forge yarl 1.3.0 py37h516909a_1000 conda-forge zeromq 4.3.2 he1b5a44_2 conda-forge zict 1.0.0 py_0 conda-forge zipp 2.2.0 py_0 conda-forge zlib 1.2.11 h516909a_1006 conda-forge zstd 1.4.3 h3b9ef0a_0 conda-forge ```

Additional context

Came up when debugging issue ( https://github.com/rapidsai/ucx-py/issues/402 ).

Given DeviceBuffer is a thin wrapper around rmm::device_buffer, am guessing this behavior begins there.

cc @quasiben (who also saw this earlier)

harrism commented 4 years ago

Argh. rmm::device_buffer uses device_memory_resource, and none of these seem to use rmmAlloc / rmmFree (rather, the other way around). Logging is implemented in rmm API, not in the memory resources.

jakirkham commented 4 years ago

Fixed with PR ( https://github.com/rapidsai/rmm/pull/297 ).