search

BlazingDB / blazingsql

BlazingSQL is a lightweight, GPU accelerated, SQL engine for Python. Built on RAPIDS cuDF.
https://blazingsql.com
Apache License 2.0
1.93k stars 183 forks source link

[BUG] Some JOIN workloads seems to crash with RMM errors #152

Closed aucahuasi closed 4 years ago

aucahuasi commented 4 years ago

Describe the bug Sometimes running the queries with JOIN we see crashes like:

ERROR: CUDA Runtime call cudaStreamSynchronize(stream) in line 283 of file /conda/envs/bzsqlenv/blazingdb-ral/src/Interpreter/interpreter_cpp.cu failed with an illegal memory access was encountered (77).
terminate called after throwing an instance of 'thrust::system::system_error'
 what():  rmm_allocator::deallocate(): RMM_FREE: __global__ function call is not configured
Aborted (core dumped)

Steps/Code to reproduce bug

algebra = """LogicalProject(visit_nbr=[$0], store_nbr=[$1], scan_type=[$3], rpt_cd=[$4], upc_nbr=[$5], othr_income_ind=[$6], scan_rtl_amt=[$10], visit_dt=[$11], scan_cnt=[$12], vendor_nbr=[CASE($14, $17, null:INTEGER)], CASE=[CASE(=(CAST($3):INTEGER, 3), 38, $2)], *=[*($7, $8)], CASE12=[CASE(=(CAST($9):INTEGER, 1), $10, 0:DOUBLE)], CASE13=[CASE(AND(>=($11, 2018-01-27), <($11, 2019-01-26), =(CAST($3):INTEGER, 0), =(CAST($16):INTEGER, 4), <>(/($10, $7), $18), OR(=(CASE($14, $17, null:INTEGER), 467830), =(CASE($14, $17, null:INTEGER), 475632), =(CASE($14, $17, null:INTEGER), 735105)), =(CAST($9):INTEGER, 1)), 0:DOUBLE, <>(CAST($9):INTEGER, 1), 0:DOUBLE, $7)], CASE14=[CASE(AND(>=($11, 2018-01-27), <($11, 2019-01-26), =(CAST($3):INTEGER, 0), =(CAST($16):INTEGER, 4), <>(/($10, $7), $18), OR(=(CASE($14, $17, null:INTEGER), 467830), =(CASE($14, $17, null:INTEGER), 475632), =(CASE($14, $17, null:INTEGER), 735105))), 0:DOUBLE, $7)], CASE15=[CASE(=(CAST($13):INTEGER, 1), $10, 0:DOUBLE)], CASE16=[CASE(AND(>=($11, 2018-01-27), <($11, 2019-01-26), =(CAST($3):INTEGER, 0), =(CAST($16):INTEGER, 4), <>(/($10, $7), $18), OR(=(CASE($14, $17, null:INTEGER), 467830), =(CASE($14, $17, null:INTEGER), 475632), =(CASE($14, $17, null:INTEGER), 735105))), 0:DOUBLE, <>(CAST($13):INTEGER, 1), 0:DOUBLE, $7)], ==[=($1, 7368)], =18=[=($1, 241)], =19=[=($1, 3335)], =20=[=($1, 2466)], =21=[=($1, 717)], =22=[=($1, 2168)], =23=[=($1, 3816)], =24=[=($1, 5746)], =25=[=($1, 5915)], =26=[=($1, 245)], =27=[=($1, 997)], =28=[=($1, 4108)], >==[>=($11, 2019-01-01)])
 LogicalTableScan(table=[[main, res]])"""
res1 =bc.sql("select whatever it does not matter", algebra=algebra)

Expected behavior Execution without crash

Environment overview (please complete the following information)

Environment details Please run and paste the output of the print_env.sh script here, to gather any other relevant environment details

Click here to see environment details

     **git***
     commit 8c50b2c7ccfcd2aed9e19558e276d3952d9e7629 (HEAD -> develop, origin/feature/gpuci, origin/develop, origin/HEAD)
     Author: Percy Camilo Triveño Aucahuasi 
     Date:   Mon Dec 2 12:32:38 2019 -0500

     Add dask-cuda as dep
     **git submodules***

     ***OS Information***
     DISTRIB_ID=Ubuntu
     DISTRIB_RELEASE=16.04
     DISTRIB_CODENAME=xenial
     DISTRIB_DESCRIPTION="Ubuntu 16.04.4 LTS"
     NAME="Ubuntu"
     VERSION="16.04.4 LTS (Xenial Xerus)"
     ID=ubuntu
     ID_LIKE=debian
     PRETTY_NAME="Ubuntu 16.04.4 LTS"
     VERSION_ID="16.04"
     HOME_URL="http://www.ubuntu.com/"
     SUPPORT_URL="http://help.ubuntu.com/"
     BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
     VERSION_CODENAME=xenial
     UBUNTU_CODENAME=xenial
     Linux pctabz 4.15.0-70-generic #79~16.04.1-Ubuntu SMP Tue Nov 12 14:01:10 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

     ***GPU Information***
     Tue Dec  3 11:16:34 2019
     +-----------------------------------------------------------------------------+
     | NVIDIA-SMI 418.87.00    Driver Version: 418.87.00    CUDA Version: 10.1     |
     |-------------------------------+----------------------+----------------------+
     | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
     | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
     |===============================+======================+======================|
     |   0  GeForce GTX 105...  Off  | 00000000:01:00.0 Off |                  N/A |
     | N/A   51C    P0    N/A /  N/A |      0MiB /  4042MiB |      0%      Default |
     +-------------------------------+----------------------+----------------------+

     +-----------------------------------------------------------------------------+
     | Processes:                                                       GPU Memory |
     |  GPU       PID   Type   Process name                             Usage      |
     |=============================================================================|
     |  No running processes found                                                 |
     +-----------------------------------------------------------------------------+

     ***CPU***
     Architecture:          x86_64
     CPU op-mode(s):        32-bit, 64-bit
     Byte Order:            Little Endian
     CPU(s):                8
     On-line CPU(s) list:   0-7
     Thread(s) per core:    2
     Core(s) per socket:    4
     Socket(s):             1
     NUMA node(s):          1
     Vendor ID:             GenuineIntel
     CPU family:            6
     Model:                 158
     Model name:            Intel(R) Core(TM) i7-7700HQ CPU @ 2.80GHz
     Stepping:              9
     CPU MHz:               3479.683
     CPU max MHz:           3800,0000
     CPU min MHz:           800,0000
     BogoMIPS:              5616.00
     Virtualization:        VT-x
     L1d cache:             32K
     L1i cache:             32K
     L2 cache:              256K
     L3 cache:              6144K
     NUMA node0 CPU(s):     0-7
     Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb invpcid_single pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx rdseed adx smap clflushopt intel_pt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp md_clear flush_l1d

     ***CMake***
     /home/percy/Applications/anaconda/conda/envs/new3/bin/cmake
     cmake version 3.15.5

     CMake suite maintained and supported by Kitware (kitware.com/cmake).

     ***g++***
     /usr/bin/g++
     g++ (Ubuntu 5.4.0-6ubuntu1~16.04.12) 5.4.0 20160609
     Copyright (C) 2015 Free Software Foundation, Inc.
     This is free software; see the source for copying conditions.  There is NO
     warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

     ***nvcc***

     ***Python***
     /home/percy/Applications/anaconda/conda/envs/new3/bin/python
     Python 3.7.4

     ***Environment Variables***
     PATH                            : /home/percy/Applications/anaconda/conda/envs/new3/bin:/home/percy/Applications/anaconda/conda/condabin:/home/percy/Applications/gcloud/google-cloud-sdk/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin:/home/percy/Applications/docker-compose/current:/home/percy/Applications/kubectl:/home/percy/Applications/minikube:/home/percy/Applications/ctop:/home/percy/Applications/anaconda/conda/bin
     LD_LIBRARY_PATH                 :
     NUMBAPRO_NVVM                   :
     NUMBAPRO_LIBDEVICE              :
     CONDA_PREFIX                    : /home/percy/Applications/anaconda/conda/envs/new3
     PYTHON_PATH                     :

     ***conda packages***
     /home/percy/Applications/anaconda/conda/condabin/conda
     # packages in environment at /home/percy/Applications/anaconda/conda/envs/new3:
     #
     # Name                    Version                   Build  Channel
     _libgcc_mutex             0.1                        main
     arrow-cpp                 0.15.0           py37h090bef1_2    conda-forge
     blazingsql                0.6                      pypi_0    pypi
     bokeh                     1.4.0                    py37_0    conda-forge
     boost-cpp                 1.70.0               h8e57a91_2    conda-forge
     brotli                    1.0.7             he1b5a44_1000    conda-forge
     bsql-engine               0.6                      pypi_0    pypi
     bsql-toolchain            0.4.7                         0    blazingsql-nightly
     bsql-toolchain-aws-cpp    0.4.7                         0    blazingsql-nightly
     bsql-toolchain-gcp-cpp    0.4.7                         0    blazingsql-nightly
     bzip2                     1.0.8                h516909a_1    conda-forge
     c-ares                    1.15.0            h516909a_1001    conda-forge
     ca-certificates           2019.11.28           hecc5488_0    conda-forge
     certifi                   2019.11.28               py37_0    conda-forge
     cffi                      1.13.2           py37h8022711_0    conda-forge
     chardet                   3.0.4                 py37_1003    conda-forge
     click                     7.0                        py_0    conda-forge
     cloudpickle               1.2.2                      py_1    conda-forge
     cmake                     3.15.5               hf94ab9c_0    conda-forge
     cppzmq                    4.4.1                hc9558a2_0    conda-forge
     cryptography              2.8              py37h72c5cf5_0    conda-forge
     cudatoolkit               10.0.130                      0
     cudf                      0.11.0a191202         py37_3723    rapidsai-nightly/label/cuda10.0
     cudnn                     7.6.4                cuda10.0_0
     cupy                      6.5.0            py37h49a79c6_0    conda-forge
     curl                      7.67.0               hbc83047_0
     cyrus-sasl                2.1.26               h82bf5a1_4
     cython                    0.29.14          py37he1b5a44_0    conda-forge
     cytoolz                   0.10.1           py37h516909a_0    conda-forge
     dask                      2.8.1                      py_0    conda-forge
     dask-core                 2.8.1                      py_0    conda-forge
     dask-cuda                 0.11.0a0+16.gac37fed          pypi_0    pypi
     dask-cudf                 0.11.0a191202         py37_3723    rapidsai-nightly/label/cuda10.0
     distributed               2.8.1                      py_0    conda-forge
     dlpack                    0.2                  he1b5a44_1    conda-forge
     double-conversion         3.1.5                he1b5a44_2    conda-forge
     expat                     2.2.5             he1b5a44_1004    conda-forge
     fastavro                  0.22.7           py37h516909a_0    conda-forge
     fastrlock                 0.4             py37he1b5a44_1000    conda-forge
     freetype                  2.10.0               he983fc9_1    conda-forge
     fsspec                    0.6.1                      py_0    conda-forge
     future                    0.18.2                   py37_0    conda-forge
     gflags                    2.2.2             he1b5a44_1002    conda-forge
     glog                      0.4.0                he1b5a44_1    conda-forge
     gmock                     1.10.0                        1    conda-forge
     grpc-cpp                  1.23.0               h18db393_0    conda-forge
     gtest                     1.10.0               hc9558a2_1    conda-forge
     heapdict                  1.0.1                      py_0    conda-forge
     icu                       64.2                 he1b5a44_1    conda-forge
     idna                      2.8                   py37_1000    conda-forge
     jinja2                    2.10.3                     py_0    conda-forge
     jpeg                      9c                h14c3975_1001    conda-forge
     jpype1                    0.7              py37h9de70de_0    conda-forge
     krb5                      1.16.1               h173b8e3_7
     libblas                   3.8.0               14_openblas    conda-forge
     libcblas                  3.8.0               14_openblas    conda-forge
     libcudf                   0.11.0a191202     cuda10.0_3723    rapidsai-nightly/label/cuda10.0
     libcurl                   7.67.0               h20c2e04_0
     libdb                     6.1.26            hf484d3e_2000    conda-forge
     libedit                   3.1.20181209         hc058e9b_0
     libevent                  2.1.10               h72c5cf5_0    conda-forge
     libffi                    3.2.1                hd88cf55_4
     libgcc-ng                 9.1.0                hdf63c60_0
     libgfortran-ng            7.3.0                hdf63c60_2    conda-forge
     liblapack                 3.8.0               14_openblas    conda-forge
     libllvm8                  8.0.1                hc9558a2_0    conda-forge
     libntlm                   1.4               h14c3975_1002    conda-forge
     libnvstrings              0.11.0a191202     cuda10.0_3723    rapidsai-nightly/label/cuda10.0
     libopenblas               0.3.7                h5ec1e0e_4    conda-forge
     libpng                    1.6.37               hed695b0_0    conda-forge
     libprotobuf               3.8.0                h8b12597_0    conda-forge
     librmm                    0.11.0b191202       cuda10.0_58    rapidsai-nightly/label/cuda10.0
     libsodium                 1.0.17               h516909a_0    conda-forge
     libssh2                   1.8.2                h22169c7_2    conda-forge
     libstdcxx-ng              9.1.0                hdf63c60_0
     libtiff                   4.1.0                hfc65ed5_0    conda-forge
     libuv                     1.33.1               h516909a_0    conda-forge
     llvmlite                  0.30.0           py37h8b12597_1    conda-forge
     locket                    0.2.0                      py_2    conda-forge
     lz4-c                     1.8.3             he1b5a44_1001    conda-forge
     markupsafe                1.1.1            py37h516909a_0    conda-forge
     maven                     3.6.0                         0    conda-forge
     msgpack-python            0.6.2            py37hc9558a2_0    conda-forge
     nccl                      2.4.8.1              hd6f8bf8_1    conda-forge
     ncurses                   6.1                  he6710b0_1
     netifaces                 0.10.9          py37h516909a_1000    conda-forge
     numba                     0.46.0           py37hb3f55d8_1    conda-forge
     numpy                     1.17.3           py37h95a1406_0    conda-forge
     nvstrings                 0.11.0a191202         py37_3723    rapidsai-nightly/label/cuda10.0
     olefile                   0.46                       py_0    conda-forge
     openjdk                   8.0.192           h14c3975_1003    conda-forge
     openssl                   1.1.1d               h516909a_0    conda-forge
     packaging                 19.2                       py_0    conda-forge
     pandas                    0.24.2           py37hb3f55d8_1    conda-forge
     parquet-cpp               1.5.1                         2    conda-forge
     partd                     1.0.0                      py_0    conda-forge
     pillow                    6.2.1            py37h34e0f95_0
     pip                       19.3.1                   py37_0
     psutil                    5.6.7            py37h516909a_0    conda-forge
     pyarrow                   0.15.0           py37h8b68381_1    conda-forge
     pycparser                 2.19                     py37_1    conda-forge
     pyhive                    0.6.1                    py37_0
     pynvml                    8.0.3                      py_0    conda-forge
     pyopenssl                 19.1.0                   py37_0    conda-forge
     pyparsing                 2.4.5                      py_0    conda-forge
     pysocks                   1.7.1                    py37_0    conda-forge
     python                    3.7.4                h265db76_1
     python-dateutil           2.8.1                      py_0    conda-forge
     pytz                      2019.3                     py_0    conda-forge
     pyyaml                    5.1.2            py37h516909a_1    conda-forge
     rapidjson                 1.1.0             he1b5a44_1002    conda-forge
     re2                       2019.12.01           he1b5a44_0    conda-forge
     readline                  7.0                  h7b6447c_5
     requests                  2.22.0                   py37_1    conda-forge
     rhash                     1.3.6             h14c3975_1001    conda-forge
     rmm                       0.11.0b191202           py37_58    rapidsai-nightly/label/cuda10.0
     sasl                      0.2.1           py37hf484d3e_1001    conda-forge
     setuptools                42.0.1                   py37_0
     six                       1.13.0                   py37_0    conda-forge
     snappy                    1.1.7             he1b5a44_1002    conda-forge
     sortedcontainers          2.1.0                      py_0    conda-forge
     sqlalchemy                1.3.11           py37h516909a_0    conda-forge
     sqlite                    3.30.1               h7b6447c_0
     tblib                     1.4.0                      py_0    conda-forge
     thrift                    0.11.0          py37he1b5a44_1001    conda-forge
     thrift-cpp                0.12.0            hf3afdfd_1004    conda-forge
     thrift_sasl               0.3.0           py37h516909a_1001    conda-forge
     tk                        8.6.8                hbc83047_0
     toolz                     0.10.0                     py_0    conda-forge
     tornado                   6.0.3            py37h516909a_0    conda-forge
     uriparser                 0.9.3                he1b5a44_1    conda-forge
     urllib3                   1.25.7                   py37_0    conda-forge
     wheel                     0.33.6                   py37_0
     xz                        5.2.4                h14c3975_4
     yaml                      0.2.2                he1b5a44_0    conda-forge
     zeromq                    4.3.2                he1b5a44_2    conda-forge
     zict                      1.0.0                      py_0    conda-forge
     zlib                      1.2.11               h7b6447c_3
     zstd                      1.4.3                h3b9ef0a_0    conda-forge

Additional context

aucahuasi commented 4 years ago

Original, query:

If i have query like

SELECT part a, part b FROM scans_c AS a JOIN store_visit_c AS b ON a.store_nbr = b.store_nbr AND a.visit_nbr = b.visit_nbr but having the query like this.

SELECT part a FROM scans_c AS a JOIN store_visit_c AS b ON a.store_nbr = b.store_nbr AND a.visit_nbr = b.visit_nbr or

SELECT part b FROM scans_c AS a JOIN store_visit_c AS b ON a.store_nbr = b.store_nbr AND a.visit_nbr = b.visit_nbr works.

aucahuasi commented 4 years ago

More clues, If I reduce the amount of data it is running fine. I think it is running out of memory calculating the following CTE (visit_scans_c ). Though the result DF doesn’t use much space on the GPU memory. I guess the blazing engine is trying to evaluate all the columns in parallel and running into out of memory issues.

aucahuasi commented 4 years ago

Hi @mlahir1 I moved the issue from the deprecated RAL repo to here. So from now we are going to put all the info related with the issue you reported to @williamBlazing here ;)

Couple of questions:

cc @rommelDB @Christian8491 @williamBlazing @felipeblazing

mlahir1 commented 4 years ago

@aucahuasi , please find the answers inline.

  1. What is the size of your data? A. The size of the data is about 100 MB on disk. Length of scans_c: 4790297 Length of store_visit_c: 530020 If the query works dimension of visit_scans_c is supposed to be 4790297 rows X 20 columns
  2. How many gpus to you have in your computer? A. 4 GPU's in each node
  3. Are you able to use other nodes with gpus so we can start running some quires with dask? A. I have run the same query with dask and run into the same issue.

cc @rommelDB @Christian8491 @williamBlazing @felipeblazing

aucahuasi commented 4 years ago

@jeanp413 +1

Christian8491 commented 4 years ago

Hi @mlahir1 , could you please checkout to thechecking-inner-join branch and compile it. After that run your initial script with your "big" query and show the output message. Something like ---------BEFORE -- INNER JOIN shoul be printed and other messages possibly. Thanks!