search

rapidsai / cudf

cuDF - GPU DataFrame Library
https://docs.rapids.ai/api/cudf/stable/
Apache License 2.0
8.33k stars 888 forks source link

[BUG] CSV reader crashs on successive calls #5086

Closed rlratzel closed 4 years ago

rlratzel commented 4 years ago

Describe the bug cugraph has an issue where a failing pytest is not allowing proper cleanup to happen, which results in a series of calls I think I recreated in the python script below. When that series of calls is made, Python crashes with a seg fault.

Steps/Code to reproduce bug Run this script:

import rmm
import cudf

def add_edge_list_to_adj_list(managed, pool, graph_file):
    rmm.reinitialize(
        managed_memory=managed,
        pool_allocator=pool,
        initial_pool_size=2 << 27
    )
    assert(rmm.is_initialized())
    print('Reading ' + str(graph_file) + '...')
    return cudf.read_csv(graph_file, delimiter=' ',
                         dtype=['int32', 'int32', 'float32'], header=None)

if __name__ == "__main__":
    #ds="./datasets/netscience.csv"
    ds="./datasets/dolphins.csv"
    cu_M = add_edge_list_to_adj_list(True, True, ds)
    cu_M = add_edge_list_to_adj_list(True, True, ds)

with this CSV: https://github.com/rapidsai/cugraph/blob/branch-0.14/datasets/dolphins.csv

Expected behavior No seg fault

Environment overview (please complete the following information) 0.14 nightly devel Docker image from night of May 2 (dated early May 3).

Environment details

Click here to see environment details

     **git***
     commit ea8ff9a9c676b7a6f70bcc6d4a63e002d3657ad9 (HEAD)
     Author: Kumar Aatish 
     Date:   Sat May 2 01:15:31 2020 -0400

     Moved coo_to_df to graph_new

     Added other fused type based convenience functions
     **git submodules***

     ***OS Information***
     DISTRIB_ID=Ubuntu
     DISTRIB_RELEASE=18.04
     DISTRIB_CODENAME=bionic
     DISTRIB_DESCRIPTION="Ubuntu 18.04.3 LTS"
     NAME="Ubuntu"
     VERSION="18.04.3 LTS (Bionic Beaver)"
     ID=ubuntu
     ID_LIKE=debian
     PRETTY_NAME="Ubuntu 18.04.3 LTS"
     VERSION_ID="18.04"
     HOME_URL="https://www.ubuntu.com/"
     SUPPORT_URL="https://help.ubuntu.com/"
     BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
     PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
     VERSION_CODENAME=bionic
     UBUNTU_CODENAME=bionic
     Linux 60df6149199d 5.0.0-32-generic #34~18.04.2-Ubuntu SMP Thu Oct 10 10:36:02 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

     ***GPU Information***
     Mon May  4 15:35:59 2020
     +-----------------------------------------------------------------------------+
     | NVIDIA-SMI 430.50       Driver Version: 430.50       CUDA Version: 10.1     |
     |-------------------------------+----------------------+----------------------+
     | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
     | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
     |===============================+======================+======================|
     |   0  Quadro RTX 8000     Off  | 00000000:15:00.0 Off |                  Off |
     | 33%   48C    P8    46W / 260W |      1MiB / 48601MiB |      0%      Default |
     +-------------------------------+----------------------+----------------------+

     +-----------------------------------------------------------------------------+
     | Processes:                                                       GPU Memory |
     |  GPU       PID   Type   Process name                             Usage      |
     |=============================================================================|
     |  No running processes found                                                 |
     +-----------------------------------------------------------------------------+

     ***CPU***
     Architecture:        x86_64
     CPU op-mode(s):      32-bit, 64-bit
     Byte Order:          Little Endian
     CPU(s):              12
     On-line CPU(s) list: 0-11
     Thread(s) per core:  2
     Core(s) per socket:  6
     Socket(s):           1
     NUMA node(s):        1
     Vendor ID:           GenuineIntel
     CPU family:          6
     Model:               85
     Model name:          Intel(R) Xeon(R) Gold 6128 CPU @ 3.40GHz
     Stepping:            4
     CPU MHz:             1200.071
     CPU max MHz:         3700.0000
     CPU min MHz:         1200.0000
     BogoMIPS:            6800.00
     Virtualization:      VT-x
     L1d cache:           32K
     L1i cache:           32K
     L2 cache:            1024K
     L3 cache:            19712K
     NUMA node0 CPU(s):   0-11
     Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single pti ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req pku ospke md_clear flush_l1d

     ***CMake***
     /opt/conda/envs/rapids/bin/cmake
     cmake version 3.14.5

     CMake suite maintained and supported by Kitware (kitware.com/cmake).

     ***g++***
     /usr/bin/g++
     g++ (Ubuntu 7.4.0-1ubuntu1~18.04.1) 7.4.0
     Copyright (C) 2017 Free Software Foundation, Inc.
     This is free software; see the source for copying conditions.  There is NO
     warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

     ***nvcc***
     /usr/local/cuda/bin/nvcc
     nvcc: NVIDIA (R) Cuda compiler driver
     Copyright (c) 2005-2019 NVIDIA Corporation
     Built on Sun_Jul_28_19:07:16_PDT_2019
     Cuda compilation tools, release 10.1, V10.1.243

     ***Python***
     /opt/conda/envs/rapids/bin/python
     Python 3.7.6

     ***Environment Variables***
     PATH                            : /opt/conda/envs/rapids/bin:/opt/conda/condabin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/conda/bin:/conda/bin:/conda/bin
     LD_LIBRARY_PATH                 : /opt/conda/envs/rapids/lib:/usr/local/nvidia/lib:/usr/local/nvidia/lib64:/usr/local/cuda/lib64:/usr/local/lib
     NUMBAPRO_NVVM                   :
     NUMBAPRO_LIBDEVICE              :
     CONDA_PREFIX                    : /opt/conda/envs/rapids
     PYTHON_PATH                     :

     ***conda packages***
     /opt/conda/envs/rapids/bin/conda
     # packages in environment at /opt/conda/envs/rapids:
     #
     # Name                    Version                   Build  Channel
     _libgcc_mutex             0.1                 conda_forge    conda-forge
     _openmp_mutex             4.5                      1_llvm    conda-forge
     aiohttp                   3.6.2            py37h516909a_0    conda-forge
     alabaster                 0.7.12                     py_0    conda-forge
     appdirs                   1.4.3                      py_1    conda-forge
     arrow-cpp                 0.15.0           py37h5ac5442_0    conda-forge
     asn1crypto                1.3.0                    py37_0    conda-forge
     async-timeout             3.0.1                   py_1000    conda-forge
     attrs                     19.3.0                     py_0    conda-forge
     aws-sam-translator        1.22.0           py37hc8dfbb8_0    conda-forge
     aws-xray-sdk              0.95                       py_0    conda-forge
     babel                     2.8.0                      py_0    conda-forge
     backcall                  0.1.0                      py_0    conda-forge
     backports                 1.0                        py_2    conda-forge
     backports.tempfile        1.0                        py_0    conda-forge
     backports.weakref         1.0.post1       py37hc8dfbb8_1001    conda-forge
     beautifulsoup4            4.9.0            py37hc8dfbb8_0    conda-forge
     black                     19.10b0                  py37_0    conda-forge
     blas                      2.14                   openblas    conda-forge
     bleach                    3.1.4              pyh9f0ad1d_0    conda-forge
     blinker                   1.4                        py_1    conda-forge
     bokeh                     1.4.0            py37hc8dfbb8_1    conda-forge
     boost-cpp                 1.70.0               h8e57a91_2    conda-forge
     boto                      2.49.0                     py_0    conda-forge
     boto3                     1.13.1             pyh9f0ad1d_0    conda-forge
     botocore                  1.16.1             pyh9f0ad1d_0    conda-forge
     brotli                    1.0.7             he1b5a44_1001    conda-forge
     brotlipy                  0.7.0           py37h8f50634_1000    conda-forge
     bzip2                     1.0.8                h516909a_2    conda-forge
     c-ares                    1.15.0            h516909a_1001    conda-forge
     ca-certificates           2020.4.5.1           hecc5488_0    conda-forge
     cachetools                3.1.1                      py_0    conda-forge
     cairo                     1.16.0            hcf35c78_1003    conda-forge
     certifi                   2020.4.5.1       py37hc8dfbb8_0    conda-forge
     cffi                      1.14.0           py37hd463f26_0    conda-forge
     cfitsio                   3.470                hb60a0a2_2    conda-forge
     cfn-lint                  0.24.4                   py37_0    conda-forge
     chardet                   3.0.4           py37hc8dfbb8_1006    conda-forge
     clang                     8.0.1                hc9558a2_2    conda-forge
     clang-tools               8.0.1                hc9558a2_2    conda-forge
     clangxx                   8.0.1                         2    conda-forge
     click                     7.1.2              pyh9f0ad1d_0    conda-forge
     cloudpickle               1.4.1                      py_0    conda-forge
     cmake                     3.14.5               hf94ab9c_0    conda-forge
     cmake_setuptools          0.1.3                      py_0    rapidsai
     cmarkgfm                  0.4.2            py37h516909a_2    conda-forge
     commonmark                0.9.1                      py_0    conda-forge
     conda                     4.7.12                   py37_1    conda-forge
     conda-build               3.18.11                  py37_0    conda-forge
     conda-package-handling    1.6.0            py37h8f50634_2    conda-forge
     conda-verify              3.1.1           py37hc8dfbb8_1001    conda-forge
     cookies                   2.2.1                      py_0    conda-forge
     coverage                  5.1              py37h8f50634_0    conda-forge
     cryptography              2.9.2            py37hb09aad4_0    conda-forge
     cudatoolkit               10.1.243             h6bb024c_0    nvidia
     cudf                      0.14.0a0+4051.g5f95c9745          pypi_0    pypi
     cudnn                     7.6.0                cuda10.1_0    nvidia
     cugraph                   0.14.0a0+361.g7352ebfd          pypi_0    pypi
     cuml                      0.14.0a0+2233.ge901e2b49          pypi_0    pypi
     cupy                      7.3.0            py37h0632833_0    conda-forge
     curl                      7.68.0               hf8cf82a_0    conda-forge
     cusignal                  0.14.0a0+238.gbed2053          pypi_0    pypi
     cuspatial                 0.14.0a0+199.g8990c6a          pypi_0    pypi
     cuxfilter                 0.14.0a0+85.g9958887          pypi_0    pypi
     cycler                    0.10.0                     py_2    conda-forge
     cyrus-sasl                2.1.27               he38ecfd_0    conda-forge
     cython                    0.29.17          py37h3340039_0    conda-forge
     cytoolz                   0.10.1           py37h516909a_0    conda-forge
     dask                      2.15.0                     py_0    conda-forge
     dask-core                 2.15.0                     py_0    conda-forge
     dask-cudf                 0.14.0a0+4051.g5f95c9745          pypi_0    pypi
     dask-glm                  0.2.0                      py_1    conda-forge
     dask-labextension         2.0.2                      py_0    conda-forge
     dask-ml                   1.2.0                      py_0    conda-forge
     dask-xgboost              0.1.5                    pypi_0    pypi
     dbus                      1.13.6               he372182_0    conda-forge
     decorator                 4.4.2                      py_0    conda-forge
     defusedxml                0.6.0                      py_0    conda-forge
     distributed               2.15.2           py37hc8dfbb8_0    conda-forge
     dlpack                    0.2                  he1b5a44_1    conda-forge
     docker-py                 4.2.0            py37hc8dfbb8_0    conda-forge
     docker-pycreds            0.4.0                      py_0    conda-forge
     docutils                  0.15.2                   py37_0    conda-forge
     double-conversion         3.1.5                he1b5a44_2    conda-forge
     doxygen                   1.8.18               hd1b7508_0    conda-forge
     ecdsa                     0.13                       py_0    conda-forge
     entrypoints               0.3             py37hc8dfbb8_1001    conda-forge
     enum34                    1.1.10           py37hc8dfbb8_1    conda-forge
     expat                     2.2.9                he1b5a44_2    conda-forge
     fastavro                  0.23.3           py37h8f50634_0    conda-forge
     fastrlock                 0.4             py37h3340039_1001    conda-forge
     feather-format            0.4.0                   py_1003    conda-forge
     filelock                  3.0.10                     py_0    conda-forge
     flake8                    3.7.9            py37hc8dfbb8_1    conda-forge
     flask                     1.1.2              pyh9f0ad1d_0    conda-forge
     flatbuffers               1.10.0            hf484d3e_1002    conda-forge
     fontconfig                2.13.1            h86ecdb6_1001    conda-forge
     freetype                  2.10.1               he06d7ca_0    conda-forge
     freexl                    1.0.5             h14c3975_1002    conda-forge
     fribidi                   1.0.9                h516909a_0    conda-forge
     fsspec                    0.7.3                      py_0    conda-forge
     future                    0.18.2           py37hc8dfbb8_1    conda-forge
     gcsfs                     0.6.1                      py_0    conda-forge
     gdal                      2.4.3            py37h5f563d9_9    conda-forge
     geos                      3.7.2                he1b5a44_2    conda-forge
     geotiff                   1.5.1                h32362d2_6    conda-forge
     gettext                   0.19.8.1          hc5be6a0_1002    conda-forge
     gflags                    2.2.2             he1b5a44_1002    conda-forge
     giflib                    5.1.7                h516909a_1    conda-forge
     git                       2.26.0          pl526hf241897_0    conda-forge
     glib                      2.64.2               h6f030ca_0    conda-forge
     glob2                     0.7                        py_0    conda-forge
     glog                      0.4.0                h49b9bf7_3    conda-forge
     gmp                       6.2.0                he1b5a44_2    conda-forge
     google-auth               1.14.1             pyh9f0ad1d_0    conda-forge
     google-auth-oauthlib      0.4.1                      py_2    conda-forge
     graphite2                 1.3.13            he1b5a44_1001    conda-forge
     graphviz                  2.40.1               h0f2764d_1    conda-forge
     grpc-cpp                  1.23.0               h18db393_0    conda-forge
     gst-plugins-base          1.14.5               h0935bb2_2    conda-forge
     gstreamer                 1.14.5               h36ae1b5_2    conda-forge
     harfbuzz                  2.4.0                h9f30f68_3    conda-forge
     hdf4                      4.2.13            hf30be14_1003    conda-forge
     hdf5                      1.10.5          nompi_h3c11f04_1104    conda-forge
     heapdict                  1.0.1                      py_0    conda-forge
     httpretty                 1.0.2                      py_0    conda-forge
     hypothesis                5.10.4                     py_0    conda-forge
     icu                       64.2                 he1b5a44_1    conda-forge
     idna                      2.8                   py37_1000    conda-forge
     imagesize                 1.2.0                      py_0    conda-forge
     importlib-metadata        1.6.0            py37hc8dfbb8_0    conda-forge
     importlib_metadata        1.6.0                         0    conda-forge
     ipykernel                 5.2.1            py37h43977f1_0    conda-forge
     ipython                   7.3.0            py37h24bf2e0_0    conda-forge
     ipython_genutils          0.2.0                      py_1    conda-forge
     isort                     4.3.21           py37hc8dfbb8_1    conda-forge
     itsdangerous              1.1.0                      py_0    conda-forge
     jedi                      0.17.0           py37hc8dfbb8_0    conda-forge
     jeepney                   0.4.3                      py_0    conda-forge
     jinja2                    2.11.2             pyh9f0ad1d_0    conda-forge
     jmespath                  0.9.5                      py_0    conda-forge
     joblib                    0.14.1                     py_0    conda-forge
     jpeg                      9c                h14c3975_1001    conda-forge
     json-c                    0.13.1            h14c3975_1001    conda-forge
     json5                     0.9.0                      py_0    conda-forge
     jsondiff                  1.1.2                      py_0    conda-forge
     jsonpatch                 1.24                       py_0    conda-forge
     jsonpickle                1.4.1              pyh9f0ad1d_0    conda-forge
     jsonpointer               2.0                        py_0    conda-forge
     jsonschema                3.2.0            py37hc8dfbb8_1    conda-forge
     jupyter-server-proxy      1.3.2                      py_0    conda-forge
     jupyter_client            6.1.3                      py_0    conda-forge
     jupyter_core              4.6.3            py37hc8dfbb8_1    conda-forge
     jupyterlab                1.2.14                     py_0    conda-forge
     jupyterlab-nvdashboard    0.2.1                    pypi_0    pypi
     jupyterlab_server         1.1.1                      py_0    conda-forge
     kealib                    1.4.13               hec59c27_0    conda-forge
     keyring                   21.2.1           py37hc8dfbb8_0    conda-forge
     kiwisolver                1.2.0            py37h99015e2_0    conda-forge
     krb5                      1.16.4               h2fd8d38_0    conda-forge
     lapack                    3.6.1                ha44fe06_2    conda-forge
     ld_impl_linux-64          2.34                 h53a641e_0    conda-forge
     libarchive                3.3.3             hb44662c_1005    conda-forge
     libblas                   3.8.0               14_openblas    conda-forge
     libcblas                  3.8.0               14_openblas    conda-forge
     libcumlprims              0.14.0a200503       cuda10.1_26    rapidsai-nightly
     libcurl                   7.68.0               hda55be3_0    conda-forge
     libcypher-parser          0.6.2                         1    rapidsai
     libdap4                   3.20.4               hd3bb157_0    conda-forge
     libedit                   3.1.20170329      hf8c457e_1001    conda-forge
     libevent                  2.1.10               h72c5cf5_0    conda-forge
     libffi                    3.2.1             he1b5a44_1007    conda-forge
     libgcc-ng                 7.3.0                h24d8f2e_5    conda-forge
     libgcrypt                 1.8.4             hf484d3e_1000    conda-forge
     libgdal                   2.4.3                h2f07a13_9    conda-forge
     libgfortran               3.0.0                         1    conda-forge
     libgfortran-ng            7.3.0                hdf63c60_5    conda-forge
     libgpg-error              1.36                 he1b5a44_0    conda-forge
     libgsasl                  1.8.0             h19a2143_1004    conda-forge
     libhwloc                  2.1.0                h3c4fd83_0    conda-forge
     libiconv                  1.15              h516909a_1006    conda-forge
     libkml                    1.3.0             h4fcabce_1010    conda-forge
     liblapack                 3.8.0               14_openblas    conda-forge
     liblapacke                3.8.0               14_openblas    conda-forge
     liblief                   0.9.0                hf8a498c_1    conda-forge
     libllvm8                  8.0.1                hc9558a2_0    conda-forge
     libnetcdf                 4.7.1           nompi_h94020b1_102    conda-forge
     libntlm                   1.4               h516909a_1002    conda-forge
     libopenblas               0.3.7                h5ec1e0e_7    conda-forge
     libpng                    1.6.37               hed695b0_1    conda-forge
     libpq                     11.5                 hd9ab2ff_2    conda-forge
     libprotobuf               3.8.0                h8b12597_0    conda-forge
     librdkafka                1.2.2                hb2b7465_0    conda-forge
     libsodium                 1.0.17               h516909a_0    conda-forge
     libspatialite             4.3.0a            h4f6d029_1032    conda-forge
     libssh2                   1.8.2                h22169c7_2    conda-forge
     libstdcxx-ng              7.3.0                hdf63c60_5    conda-forge
     libtiff                   4.0.10            h57b8799_1003    conda-forge
     libtool                   2.4.6             h14c3975_1002    conda-forge
     libuuid                   2.32.1            h14c3975_1000    conda-forge
     libuv                     1.34.0               h516909a_0    conda-forge
     libxcb                    1.13              h14c3975_1002    conda-forge
     libxml2                   2.9.10               hee79883_0    conda-forge
     lightgbm                  2.3.1            py37he1b5a44_0    conda-forge
     llvm-openmp               10.0.0               hc9558a2_0    conda-forge
     llvmlite                  0.31.0           py37h5202443_1    conda-forge
     locket                    0.2.0                      py_2    conda-forge
     lz4-c                     1.8.3             he1b5a44_1001    conda-forge
     lzo                       2.10              h14c3975_1000    conda-forge
     make                      4.3                  h516909a_0    conda-forge
     markupsafe                1.1.1            py37h8f50634_1    conda-forge
     matplotlib                3.2.1                         0    conda-forge
     matplotlib-base           3.2.1            py37h30547a4_0    conda-forge
     mccabe                    0.6.1                      py_1    conda-forge
     mistune                   0.8.4           py37h8f50634_1001    conda-forge
     mock                      3.0.5            py37hc8dfbb8_1    conda-forge
     more-itertools            8.2.0                      py_0    conda-forge
     moto                      1.3.14                     py_0    conda-forge
     msgpack-python            1.0.0            py37h99015e2_1    conda-forge
     multidict                 4.7.5            py37h8f50634_1    conda-forge
     multipledispatch          0.6.0                      py_0    conda-forge
     mypy_extensions           0.4.3            py37hc8dfbb8_1    conda-forge
     nbconvert                 5.6.1            py37hc8dfbb8_1    conda-forge
     nbformat                  5.0.6                      py_0    conda-forge
     nbsphinx                  0.6.1              pyh9f0ad1d_0    conda-forge
     nccl                      2.5.7.1              h51cf6c1_0    conda-forge
     ncurses                   6.1               hf484d3e_1002    conda-forge
     networkx                  2.4                        py_1    conda-forge
     nodejs                    13.13.0              hf5d1a2b_0    conda-forge
     nomkl                     1.0                  h5ca1d4c_0    conda-forge
     notebook                  6.0.3                    py37_0    conda-forge
     numba                     0.48.0           py37hb3f55d8_0    conda-forge
     numpy                     1.18.1           py37h8960a57_1    conda-forge
     numpydoc                  0.9.2                      py_0    conda-forge
     nvstrings-cuda101         0.0.0.dev0               pypi_0    pypi
     oauthlib                  3.0.1                      py_0    conda-forge
     olefile                   0.46                       py_0    conda-forge
     openjpeg                  2.3.1                h21c5421_1    conda-forge
     openssl                   1.1.1g               h516909a_0    conda-forge
     packaging                 20.1                       py_0    conda-forge
     pandas                    0.25.3           py37hb3f55d8_0    conda-forge
     pandoc                    1.19.2                        0    conda-forge
     pandocfilters             1.4.2                      py_1    conda-forge
     pango                     1.42.4               h7062337_4    conda-forge
     parquet-cpp               1.5.1                         2    conda-forge
     parso                     0.7.0              pyh9f0ad1d_0    conda-forge
     partd                     1.1.0                      py_0    conda-forge
     patchelf                  0.10                 he1b5a44_0    conda-forge
     pathspec                  0.8.0              pyh9f0ad1d_0    conda-forge
     patsy                     0.5.1                      py_0    conda-forge
     pcre                      8.44                 he1b5a44_0    conda-forge
     perl                      5.26.2            h516909a_1006    conda-forge
     pexpect                   4.8.0            py37hc8dfbb8_1    conda-forge
     pickleshare               0.7.5           py37hc8dfbb8_1001    conda-forge
     pillow                    6.2.1            py37h6b7be26_0    conda-forge
     pip                       20.1               pyh9f0ad1d_0    conda-forge
     pixman                    0.38.0            h516909a_1003    conda-forge
     pkginfo                   1.5.0.1                    py_0    conda-forge
     pluggy                    0.13.0                   py37_0    conda-forge
     poppler                   0.67.0               h14e79db_8    conda-forge
     poppler-data              0.4.9                         1    conda-forge
     postgresql                11.5                 hc63931a_2    conda-forge
     proj                      6.2.1                hc80f0dc_0    conda-forge
     prometheus_client         0.7.1                      py_0    conda-forge
     prompt-toolkit            2.0.10                   pypi_0    pypi
     prompt_toolkit            3.0.5                         0    conda-forge
     protobuf                  3.8.0            py37he1b5a44_2    conda-forge
     psutil                    5.7.0            py37h8f50634_1    conda-forge
     pthread-stubs             0.4               h14c3975_1001    conda-forge
     ptyprocess                0.6.0                   py_1001    conda-forge
     py                        1.8.1                      py_0    conda-forge
     py-lief                   0.9.0            py37he1b5a44_1    conda-forge
     pyarrow                   0.15.0           py37h8b68381_1    conda-forge
     pyasn1                    0.4.8                      py_0    conda-forge
     pyasn1-modules            0.2.7                      py_0    conda-forge
     pycodestyle               2.5.0                      py_0    conda-forge
     pycosat                   0.6.3           py37h8f50634_1004    conda-forge
     pycparser                 2.20                       py_0    conda-forge
     pyflakes                  2.1.1                      py_0    conda-forge
     pygments                  2.6.1                      py_0    conda-forge
     pyjwt                     1.7.1                      py_0    conda-forge
     pynvml                    8.0.4                    pypi_0    pypi
     pyopenssl                 19.1.0                     py_1    conda-forge
     pyparsing                 2.4.7              pyh9f0ad1d_0    conda-forge
     pyqt                      5.9.2            py37hcca6a23_4    conda-forge
     pyrsistent                0.16.0           py37h8f50634_0    conda-forge
     pysocks                   1.7.1            py37hc8dfbb8_1    conda-forge
     pytest                    5.4.1            py37hc8dfbb8_0    conda-forge
     pytest-cov                2.8.1                      py_0    conda-forge
     python                    3.7.6           h8356626_5_cpython    conda-forge
     python-dateutil           2.8.1                      py_0    conda-forge
     python-jose               3.1.0              pyh9f0ad1d_0    conda-forge
     python-libarchive-c       2.9                      py37_0    conda-forge
     python_abi                3.7                     1_cp37m    conda-forge
     pytz                      2020.1             pyh9f0ad1d_0    conda-forge
     pyyaml                    5.3.1            py37h8f50634_0    conda-forge
     pyzmq                     19.0.0           py37hac76be4_1    conda-forge
     qt                        5.9.7                h0c104cb_3    conda-forge
     rapidjson                 1.1.0             he1b5a44_1002    conda-forge
     rapids-build-env          0.14.0          cuda10.1_py37_150    rapidsai-nightly
     rapids-doc-env            0.14.0                 py37_151    rapidsai-nightly
     rapids-notebook-env       0.14.0          cuda10.1_py37_150    rapidsai-nightly
     re2                       2020.05.01           he1b5a44_0    conda-forge
     readline                  8.0                  hf8c457e_0    conda-forge
     readme_renderer           24.0                       py_0    conda-forge
     recommonmark              0.6.0                      py_0    conda-forge
     regex                     2020.4.4         py37h8f50634_0    conda-forge
     requests                  2.23.0             pyh8c360ce_2    conda-forge
     requests-oauthlib         1.2.0                      py_0    conda-forge
     requests-toolbelt         0.9.1                      py_0    conda-forge
     responses                 0.10.14            pyh9f0ad1d_0    conda-forge
     rhash                     1.3.6             h14c3975_1001    conda-forge
     ripgrep                   12.0.1               h516909a_1    conda-forge
     rmm                       0.14.0                   pypi_0    pypi
     rsa                       4.0                        py_0    conda-forge
     ruamel_yaml               0.15.80         py37h8f50634_1001    conda-forge
     s3fs                      0.4.2                      py_0    conda-forge
     s3transfer                0.3.3            py37hc8dfbb8_1    conda-forge
     scikit-learn              0.21.3           py37hcdab131_0    conda-forge
     scipy                     1.3.0            py37h921218d_1    conda-forge
     seaborn                   0.10.1                     py_0    conda-forge
     secretstorage             3.1.2            py37hc8dfbb8_1    conda-forge
     send2trash                1.5.0                      py_0    conda-forge
     setuptools                46.1.3           py37hc8dfbb8_0    conda-forge
     simpervisor               0.3                        py_1    conda-forge
     sip                       4.19.8          py37hf484d3e_1000    conda-forge
     six                       1.14.0                     py_1    conda-forge
     snappy                    1.1.8                he1b5a44_1    conda-forge
     snowballstemmer           2.0.0                      py_0    conda-forge
     sortedcontainers          2.1.0                      py_0    conda-forge
     soupsieve                 1.9.4            py37hc8dfbb8_1    conda-forge
     sphinx                    3.0.3                      py_0    conda-forge
     sphinx_rtd_theme          0.4.3                      py_0    conda-forge
     sphinxcontrib-applehelp   1.0.2                      py_0    conda-forge
     sphinxcontrib-devhelp     1.0.2                      py_0    conda-forge
     sphinxcontrib-htmlhelp    1.0.3                      py_0    conda-forge
     sphinxcontrib-jsmath      1.0.1                      py_0    conda-forge
     sphinxcontrib-qthelp      1.0.3                      py_0    conda-forge
     sphinxcontrib-serializinghtml 1.1.4                      py_0    conda-forge
     sphinxcontrib-websupport  1.2.2              pyh9f0ad1d_0    conda-forge
     sqlite                    3.30.1               hcee41ef_0    conda-forge
     sshpubkeys                3.1.0                      py_0    conda-forge
     statsmodels               0.11.1           py37h8f50634_1    conda-forge
     streamz                   0.5.3              pyh9f0ad1d_0    conda-forge
     tbb                       2020.1               hc9558a2_0    conda-forge
     tblib                     1.6.0                      py_0    conda-forge
     terminado                 0.8.3            py37hc8dfbb8_1    conda-forge
     testpath                  0.4.4                      py_0    conda-forge
     thrift-cpp                0.12.0            hf3afdfd_1004    conda-forge
     tk                        8.6.10               hed695b0_0    conda-forge
     toml                      0.10.0                     py_0    conda-forge
     toolz                     0.10.0                     py_0    conda-forge
     tornado                   6.0.4            py37h8f50634_1    conda-forge
     tqdm                      4.45.0             pyh9f0ad1d_1    conda-forge
     traitlets                 4.3.3            py37hc8dfbb8_1    conda-forge
     twine                     3.1.1                    py37_0    conda-forge
     typed-ast                 1.4.1            py37h516909a_0    conda-forge
     typing_extensions         3.7.4.2                    py_0    conda-forge
     tzcode                    2020a                h516909a_0    conda-forge
     ucx                       1.7.0+g9d06c3a       cuda10.1_0    rapidsai
     ucx-py                    0.14.0a0+166.g7cf2144          pypi_0    pypi
     umap-learn                0.4.2            py37hc8dfbb8_0    conda-forge
     uriparser                 0.9.3                he1b5a44_1    conda-forge
     urllib3                   1.25.9                     py_0    conda-forge
     wcwidth                   0.1.9              pyh9f0ad1d_0    conda-forge
     webencodings              0.5.1                      py_1    conda-forge
     websocket-client          0.57.0           py37hc8dfbb8_1    conda-forge
     werkzeug                  1.0.1              pyh9f0ad1d_0    conda-forge
     wheel                     0.34.2                     py_1    conda-forge
     wrapt                     1.12.1           py37h8f50634_1    conda-forge
     xerces-c                  3.2.2             h8412b87_1004    conda-forge
     xgboost                   1.0.0-SNAPSHOT           pypi_0    pypi
     xmltodict                 0.12.0                     py_0    conda-forge
     xorg-kbproto              1.0.7             h14c3975_1002    conda-forge
     xorg-libice               1.0.10               h516909a_0    conda-forge
     xorg-libsm                1.2.3             h84519dc_1000    conda-forge
     xorg-libx11               1.6.9                h516909a_0    conda-forge
     xorg-libxau               1.0.9                h14c3975_0    conda-forge
     xorg-libxdmcp             1.1.3                h516909a_0    conda-forge
     xorg-libxext              1.3.4                h516909a_0    conda-forge
     xorg-libxpm               3.5.13               h516909a_0    conda-forge
     xorg-libxrender           0.9.10            h516909a_1002    conda-forge
     xorg-libxt                1.1.5             h516909a_1003    conda-forge
     xorg-renderproto          0.11.1            h14c3975_1002    conda-forge
     xorg-xextproto            7.3.0             h14c3975_1002    conda-forge
     xorg-xproto               7.0.31            h14c3975_1007    conda-forge
     xz                        5.2.5                h516909a_0    conda-forge
     yaml                      0.2.4                h516909a_0    conda-forge
     yarl                      1.3.0           py37h516909a_1000    conda-forge
     zeromq                    4.3.2                he1b5a44_2    conda-forge
     zict                      2.0.0                    pypi_0    pypi
     zipp                      3.1.0                      py_0    conda-forge
     zlib                      1.2.11            h516909a_1006    conda-forge
     zstd                      1.4.0                h3b9ef0a_0    conda-forge

Additional context

Thread 1 "python" received signal SIGSEGV, Segmentation fault.
0x0000000000000005 in ?? ()
(gdb) where
#0  0x0000000000000005 in ?? ()
#1  0x00007feb34813fc8 in rmm::mr::device_memory_resource::deallocate (stream=0x0, bytes=<optimized out>, p=<optimized out>, this=<optimized out>) at ../include/rmm/mr/device/device_memory_resource.hpp:104
#2  rmm::device_buffer::deallocate (this=0x5601bce5bca0) at ../include/rmm/device_buffer.hpp:413
#3  rmm::device_buffer::~device_buffer (this=0x5601bce5bca0, __in_chrg=<optimized out>) at ../include/rmm/device_buffer.hpp:253
#4  std::default_delete<rmm::device_buffer>::operator() (this=<optimized out>, __ptr=0x5601bce5bca0) at /usr/include/c++/7/bits/unique_ptr.h:78
#5  std::unique_ptr<rmm::device_buffer, std::default_delete<rmm::device_buffer> >::~unique_ptr (this=0x7fea76c4df68, __in_chrg=<optimized out>) at /usr/include/c++/7/bits/unique_ptr.h:268
#6  __Pyx_call_destructor<std::unique_ptr<rmm::device_buffer, std::default_delete<rmm::device_buffer> > > (x=...) at rmm/_lib/device_buffer.cpp:334
#7  __pyx_tp_dealloc_3rmm_4_lib_13device_buffer_DeviceBuffer (o=0x7fea76c4df50) at rmm/_lib/device_buffer.cpp:20784
#8  0x00005601b9a69987 in dict_dealloc () at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Objects/dictobject.c:1905
#9  0x00005601b9afdb60 in subtype_dealloc () at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Objects/typeobject.c:1263
#10 0x00007fea7c6499f7 in __pyx_tp_dealloc_4cudf_4_lib_6column_Column (o=0x7fea76b23b90) at cudf/_lib/column.cpp:24204
#11 0x00005601b9afdb7e in subtype_dealloc () at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Objects/typeobject.c:1278
#12 0x00005601b9a69ac7 in tupledealloc () at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Objects/tupleobject.c:246
#13 0x00005601b9a699b8 in dict_dealloc () at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Objects/dictobject.c:1905
#14 0x00005601b9afdb60 in subtype_dealloc () at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Objects/typeobject.c:1263
#15 0x00005601b9a59491 in free_keys_object (keys=<optimized out>) at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Objects/dictobject.c:559
#16 PyDict_Clear () at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Objects/dictobject.c:1644
#17 0x00007fea7c694787 in __pyx_tp_dealloc_4cudf_4_lib_5table_Table (o=0x7fea7677c9b0) at cudf/_lib/table.cpp:19003
...
rgsl888prabhu commented 4 years ago

This seems to be an issue in general, here is a one more repro, might be with similar scenario in cudf python as #4826. @shwina have you come across this issue anytime?

import rmm
import cudf
rmm.reinitialize(
    managed_memory=True,
    pool_allocator=True,
    initial_pool_size=2 << 27
)
assert(rmm.is_initialized())
df = cudf.DataFrame({"a":[1]})

(cudf_dev2) rgsl888@onepiece:~/Projects/backup/cudf$ cuda-memcheck python sam.py 
========= CUDA-MEMCHECK
========= Program hit cudaErrorCudartUnloading (error 4) due to "driver shutting down" on CUDA API call to cudaGetDevice. 
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame:/usr/lib/x86_64-linux-gnu/libcuda.so [0x3ac5a3]
=========     Host Frame:/home/rgsl888/anaconda3/envs/cudf_dev2/lib/python3.7/site-packages/rmm/_lib/../../../../libcudart.so.10.1 (cudaGetDevice + 0x186) [0x4b416]
=========     Host Frame:/home/rgsl888/anaconda3/envs/cudf_dev2/lib/python3.7/site-packages/rmm/_lib/../../../../librmm.so (_ZN5cnmem7ContextD1Ev + 0x2b) [0x568fb]
=========     Host Frame:/home/rgsl888/anaconda3/envs/cudf_dev2/lib/python3.7/site-packages/rmm/_lib/../../../../librmm.so (_ZN5cnmem7Context7releaseEv + 0x56) [0x56b56]
=========     Host Frame:/home/rgsl888/anaconda3/envs/cudf_dev2/lib/python3.7/site-packages/rmm/_lib/../../../../librmm.so (_ZN3rmm2mr29cnmem_managed_memory_resourceD0Ev + 0x20) [0x37e90]
=========     Host Frame:/home/rgsl888/anaconda3/envs/cudf_dev2/lib/python3.7/site-packages/rmm/_lib/../../../../librmm.so (_ZN3rmm7Manager8finalizeEv + 0x20f) [0x2ce9f]
=========     Host Frame:/home/rgsl888/anaconda3/envs/cudf_dev2/lib/python3.7/site-packages/rmm/_lib/../../../../librmm.so (_Z11rmmFinalizev + 0x24) [0x27f54]
=========     Host Frame:/home/rgsl888/anaconda3/envs/cudf_dev2/lib/python3.7/site-packages/rmm/_lib/lib.cpython-37m-x86_64-linux-gnu.so [0x7b96]
=========     Host Frame:/lib/x86_64-linux-gnu/libc.so.6 [0x43041]
=========     Host Frame:/lib/x86_64-linux-gnu/libc.so.6 [0x4313a]
=========     Host Frame:/lib/x86_64-linux-gnu/libc.so.6 (__libc_start_main + 0xee) [0x21b9e]
=========     Host Frame:python [0x1dcbd0]
=========
========= Program hit cudaErrorCudartUnloading (error 4) due to "driver shutting down" on CUDA API call to cudaSetDevice. 
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame:/usr/lib/x86_64-linux-gnu/libcuda.so [0x3ac5a3]
=========     Host Frame:/home/rgsl888/anaconda3/envs/cudf_dev2/lib/python3.7/site-packages/rmm/_lib/../../../../libcudart.so.10.1 (cudaSetDevice + 0x180) [0x4b5b0]
=========     Host Frame:/home/rgsl888/anaconda3/envs/cudf_dev2/lib/python3.7/site-packages/rmm/_lib/../../../../librmm.so (_ZN5cnmem7ContextD1Ev + 0x4d) [0x5691d]
=========     Host Frame:/home/rgsl888/anaconda3/envs/cudf_dev2/lib/python3.7/site-packages/rmm/_lib/../../../../librmm.so (_ZN5cnmem7Context7releaseEv + 0x56) [0x56b56]
=========     Host Frame:/home/rgsl888/anaconda3/envs/cudf_dev2/lib/python3.7/site-packages/rmm/_lib/../../../../librmm.so (_ZN3rmm2mr29cnmem_managed_memory_resourceD0Ev + 0x20) [0x37e90]
=========     Host Frame:/home/rgsl888/anaconda3/envs/cudf_dev2/lib/python3.7/site-packages/rmm/_lib/../../../../librmm.so (_ZN3rmm7Manager8finalizeEv + 0x20f) [0x2ce9f]
=========     Host Frame:/home/rgsl888/anaconda3/envs/cudf_dev2/lib/python3.7/site-packages/rmm/_lib/../../../../librmm.so (_Z11rmmFinalizev + 0x24) [0x27f54]
=========     Host Frame:/home/rgsl888/anaconda3/envs/cudf_dev2/lib/python3.7/site-packages/rmm/_lib/lib.cpython-37m-x86_64-linux-gnu.so [0x7b96]
=========     Host Frame:/lib/x86_64-linux-gnu/libc.so.6 [0x43041]
=========     Host Frame:/lib/x86_64-linux-gnu/libc.so.6 [0x4313a]
=========     Host Frame:/lib/x86_64-linux-gnu/libc.so.6 (__libc_start_main + 0xee) [0x21b9e]
=========     Host Frame:python [0x1dcbd0]
rgsl888prabhu commented 4 years ago

@rlratzel The issue in the example is rmm is being reinitialized, it doesn't inform cudf about memory being released/acquired, and cudf still holds the unique_ptr that it got which normally gets flushed out later. But, with rmm re-initialization, cudf and rmm both are out of sync which causes this issue. It would be better to allocate pool as module level fixture rather than initializing every-time.

For example, modified test case 

import rmm
import cudf
def add_edge_list_to_adj_list(graph_file):
    print('Reading ' + str(graph_file) + '...')
    return cudf.read_csv(graph_file, delimiter=' ',
                         dtype=['int32', 'int32', 'float32'], header=None)
if __name__ == "__main__":
    #ds="./datasets/netscience.csv"
    rmm.reinitialize(
        managed_memory=True,
        pool_allocator=True,
        initial_pool_size=2 << 27
    )
    assert(rmm.is_initialized())
    ds="./dolphins.csv"
    cu_M = add_edge_list_to_adj_list(ds)
    cu_M = add_edge_list_to_adj_list(ds)
    cu_M = add_edge_list_to_adj_list(ds)
    cu_M = add_edge_list_to_adj_list(ds)
kkraus14 commented 4 years ago

Closing as not an issue.

rlratzel commented 4 years ago

Thanks for the explanation @rgsl888prabhu .

We have the re-init normally take parameters in our tests to test cugraph under different RMM configurations (managed and pool True/False combinations), which is why it's there. For my own knowledge, is there a way we can manually inform cudf about the unique_ptr being invalidated somehow in a teardown (or just a safe way to reinit with different params)? Or, is this just unsupported? (I'm fine with that answer too) We're considering removing the parameterization of the RMM config otherwise and just testing with one config.

rgsl888prabhu commented 4 years ago

You can del the residual elements explicitly before moving to next tests. Don't know about any other way to trigger garbage collection.