kkraus14
commented
3 years ago This typically happens from creating a CUDA context before forking which is not allowed. Are you importing cudf in the top level somewhere in your application?
Closed GeeCastro closed 3 years ago
kkraus14
commented
3 years ago This typically happens from creating a CUDA context before forking which is not allowed. Are you importing cudf in the top level somewhere in your application?
GeeCastro
commented
3 years ago It worked with the small example above, thanks! I'll see if I can make sure we only load the cudf in that function for our project... If it's not an option, I guess I'll have to find a way to initialise cuda manually within that function?
kkraus14
commented
3 years ago It worked with the small example above, thanks! I'll see if I can make sure we only load the cudf in that function for our project... If it's not an option, I guess I'll have to find a way to initialise cuda manually within that function?
Importing cudf by default runs some code to validate things like driver version, toolkit version, device capability, etc. and this code initializes CUDA. You can bypass this by setting the environment variable: CUDF_NO_INITIALIZE or RAPIDS_NO_INITIALIZE
GeeCastro
commented
3 years ago setting export RAPIDS_NO_INITIALIZE=True worked, indeed. Thanks again! I guess it'll just initialise in a lazy fashion when the actual functions are called?
Describe the bug rapids.ai throws a
cudaErrorInitializationErrorwhen used in a celery worker.Steps/Code to reproduce bug
from python interpreter
It throws the following error on the celery worker.
Expected behavior load the dataframe without crashing. It works fine if not used with celery.
Environment overview (please complete the following information)
Environment details Please run and paste the output of the
cudf/print_env.shscript here, to gather any other relevant environment details cat env.txtClick here to see environment details
**git*** Not inside a git repository ***OS Information*** DISTRIB_ID=Ubuntu DISTRIB_RELEASE=18.04 DISTRIB_CODENAME=bionic DISTRIB_DESCRIPTION="Ubuntu 18.04.5 LTS" NAME="Ubuntu" VERSION="18.04.5 LTS (Bionic Beaver)" ID=ubuntu ID_LIKE=debian PRETTY_NAME="Ubuntu 18.04.5 LTS" VERSION_ID="18.04" HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" VERSION_CODENAME=bionic UBUNTU_CODENAME=bionic Linux dsbird.int.cfms.org.uk 4.15.0-128-generic #131-Ubuntu SMP Wed Dec 9 06:57:35 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux ***GPU Information*** Tue Jan 5 18:47:58 2021 +-----------------------------------------------------------------------------+ | NVIDIA-SMI 455.45.01 Driver Version: 455.45.01 CUDA Version: 11.1 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |===============================+======================+======================| | 0 Tesla T4 On | 00000000:05:00.0 Off | 0 | | N/A 32C P8 9W / 70W | 3MiB / 15109MiB | 0% Default | | | | N/A | +-------------------------------+----------------------+----------------------+ +-----------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=============================================================================| | No running processes found | +-----------------------------------------------------------------------------+ ***CPU*** Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 28 On-line CPU(s) list: 0-27 Thread(s) per core: 2 Core(s) per socket: 14 Socket(s): 1 NUMA node(s): 1 Vendor ID: GenuineIntel CPU family: 6 Model: 79 Model name: Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz Stepping: 1 CPU MHz: 1703.911 CPU max MHz: 3300.0000 CPU min MHz: 1200.0000 BogoMIPS: 4800.29 Virtualization: VT-x L1d cache: 32K L1i cache: 32K L2 cache: 256K L3 cache: 35840K NUMA node0 CPU(s): 0-27 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm rdt_a rdseed adx smap intel_pt xsaveopt cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts md_clear flush_l1d ***CMake*** ***g++*** /usr/bin/g++ g++ (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0 Copyright (C) 2017 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. ***nvcc*** ***Python*** /gpfs/cfms/home/gcastro/miniconda3_x86-64/envs/rapids17/bin/python Python 3.7.8 ***Environment Variables*** PATH : /gpfs/cfms/home/gcastro/miniconda3_x86-64/envs/rapids17/bin:/gpfs/cfms/home/gcastro/miniconda3_x86-64/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/gpfs/cfms/home/gcastro/.local/bin:/gpfs/cfms/home/gcastro/bin LD_LIBRARY_PATH : NUMBAPRO_NVVM : NUMBAPRO_LIBDEVICE : CONDA_PREFIX : /gpfs/cfms/home/gcastro/miniconda3_x86-64/envs/rapids17 PYTHON_PATH : ***conda packages*** /gpfs/cfms/home/gcastro/miniconda3_x86-64/condabin/conda # packages in environment at /gpfs/cfms/home/gcastro/miniconda3_x86-64/envs/rapids17: # # Name Version Build Channel _libgcc_mutex 0.1 conda_forge conda-forge _openmp_mutex 4.5 1_llvm conda-forge abseil-cpp 20200225.2 he1b5a44_2 conda-forge aiohttp 3.7.3 py37h4abf009_0 conda-forge alembic 1.4.3 pypi_0 pypi alsa-lib 1.2.3 h516909a_0 conda-forge amqp 2.6.0 pypi_0 pypi aniso8601 8.1.0 pyhd3deb0d_0 conda-forge appdirs 1.4.4 pyh9f0ad1d_0 conda-forge argon2-cffi 20.1.0 py37h4abf009_2 conda-forge arrow 0.17.0 py37hc8dfbb8_1 conda-forge arrow-cpp 1.0.1 py37h2318771_14_cuda conda-forge arrow-cpp-proc 2.0.0 cuda conda-forge async-timeout 3.0.1 py_1000 conda-forge async_generator 1.10 py_0 conda-forge attrs 20.3.0 pyhd3deb0d_0 conda-forge aws-c-common 0.4.59 h36c2ea0_1 conda-forge aws-c-event-stream 0.1.6 had2084c_6 conda-forge aws-checksums 0.1.10 h4e93380_0 conda-forge aws-sdk-cpp 1.8.63 h9b98462_0 conda-forge babel 2.9.0 pyhd3deb0d_0 conda-forge backcall 0.2.0 pyh9f0ad1d_0 conda-forge backports 1.0 py_2 conda-forge backports.functools_lru_cache 1.6.1 py_0 conda-forge bcrypt 3.2.0 py37h8f50634_1 conda-forge beautifulsoup4 4.9.3 pypi_0 pypi billiard 3.6.3.0 py37h8f50634_2 conda-forge blazingsql 0.17.0 pypi_0 pypi bleach 3.2.1 pyh9f0ad1d_0 conda-forge blinker 1.4 py_1 conda-forge bokeh 2.2.3 py37h89c1867_0 conda-forge boost 1.72.0 py37h48f8a5e_1 conda-forge boost-cpp 1.72.0 h8e57a91_0 conda-forge brotli 1.0.9 he1b5a44_3 conda-forge brotlipy 0.7.0 py37hb5d75c8_1001 conda-forge bzip2 1.0.8 h516909a_3 conda-forge c-ares 1.17.1 h36c2ea0_0 conda-forge ca-certificates 2020.12.5 ha878542_0 conda-forge cachelib 0.1.1 pypi_0 pypi cairo 1.16.0 hcf35c78_1003 conda-forge catboost 0.24.1 pypi_0 pypi celery 4.4.4 pyh9f0ad1d_0 conda-forge certifi 2020.12.5 py37h89c1867_0 conda-forge cffi 1.14.4 py37h11fe52a_0 conda-forge cfitsio 3.470 h2e3daa1_7 conda-forge chardet 3.0.4 py37he5f6b98_1008 conda-forge click 7.1.2 pyh9f0ad1d_0 conda-forge click-plugins 1.1.1 py_0 conda-forge cliff 3.5.0 pypi_0 pypi cligj 0.7.1 pyhd8ed1ab_0 conda-forge cloudpickle 1.6.0 py_0 conda-forge cmaes 0.7.0 pypi_0 pypi cmd2 1.4.0 pypi_0 pypi colorama 0.4.4 pypi_0 pypi colorcet 2.0.1 py_0 conda-forge colorlog 4.6.2 pypi_0 pypi cryptography 3.2.1 py37hc72a4ac_0 conda-forge cudatoolkit 11.0.221 h6bb024c_0 nvidia cudf 0.17.0a201209 cuda_11.0_py37_gbd321d1e93_382 rapidsai-nightly cudf_kafka 0.17.0a201209 py37_gbd321d1e93_382 rapidsai-nightly cudnn 8.0.0 cuda11.0_0 nvidia cugraph 0.17.0a201210 py37_ge205fd07_289 rapidsai-nightly cuml 0.17.0a201210 cuda11.0_py37_g2c0aacf44_173 rapidsai-nightly cupy 8.0.0 py37h0ce7dbb_0 rapidsai-nightly curl 7.71.1 he644dc0_8 conda-forge cusignal 0.17.0a201210 py37_g44e57f5_47 rapidsai-nightly cuspatial 0.17.0a201210 py37_g9aeebc8_23 rapidsai-nightly custreamz 0.17.0a201209 py37_gbd321d1e93_382 rapidsai-nightly cuxfilter 0.17.0a201210 py37_ge97755c_109 rapidsai-nightly cycler 0.10.0 py_2 conda-forge cyrus-sasl 2.1.27 h063b49f_1 conda-forge cython 0.29.21 pypi_0 pypi cytoolz 0.11.0 py37h4abf009_1 conda-forge dask 2.30.0 py_0 conda-forge dask-core 2.30.0 py_0 conda-forge dask-cuda 0.17.0a201207 py37_56 rapidsai-nightly dask-cudf 0.17.0a201209 py37_gbd321d1e93_382 rapidsai-nightly datashader 0.11.1 pyh9f0ad1d_0 conda-forge datashape 0.5.4 py_1 conda-forge decorator 4.4.2 py_0 conda-forge defusedxml 0.6.0 py_0 conda-forge distributed 2.30.1 py37h89c1867_0 conda-forge dlpack 0.3 he1b5a44_1 conda-forge dnspython 2.0.0 py_0 conda-forge email_validator 1.1.1 pyh9f0ad1d_0 conda-forge entrypoints 0.3 pyhd8ed1ab_1003 conda-forge et_xmlfile 1.0.1 py_1001 conda-forge expat 2.2.9 he1b5a44_2 conda-forge faiss-proc 1.0.0 cuda rapidsai-nightly faker 4.1.0 py_0 conda-forge fastavro 1.2.0 py37h4abf009_0 conda-forge fastrlock 0.5 py37h3340039_1 conda-forge filelock 3.0.12 pyh9f0ad1d_0 conda-forge fiona 1.8.13 py37h0492a4a_1 conda-forge flask 1.1.2 pyh9f0ad1d_0 conda-forge flask-admin 1.5.6 pyh9f0ad1d_0 conda-forge flask-alembic 2.0.1 pypi_0 pypi flask-babelex 0.9.4 py_0 conda-forge flask-limiter 1.3 pypi_0 pypi flask-login 0.5.0 pyh9f0ad1d_0 conda-forge flask-mail 0.9.1 py_2 conda-forge flask-marshmallow 0.11.0 py_0 conda-forge flask-principal 0.4.0 py_1 conda-forge flask-restful 0.3.8 py_0 conda-forge flask-script 2.0.6 py_0 conda-forge flask-security 3.0.0 py_0 conda-forge flask-session 0.3.2 pypi_0 pypi flask-sqlalchemy 2.4.2 pyh9f0ad1d_0 conda-forge flask-wtf 0.14.3 py_0 conda-forge fontconfig 2.13.1 h86ecdb6_1001 conda-forge freetype 2.10.4 h7ca028e_0 conda-forge freexl 1.0.5 h516909a_1002 conda-forge fsspec 0.8.4 py_0 conda-forge future 0.18.2 py37h89c1867_2 conda-forge gdal 3.0.4 py37h4b180d9_10 conda-forge geopandas 0.8.1 py_0 conda-forge geos 3.8.1 he1b5a44_0 conda-forge geotiff 1.6.0 h05acad5_0 conda-forge gettext 0.19.8.1 hf34092f_1004 conda-forge gflags 2.2.2 he1b5a44_1004 conda-forge giflib 5.2.1 h36c2ea0_2 conda-forge git 2.29.2 pl5320h78be0e9_1 conda-forge glib 2.66.3 h58526e2_0 conda-forge glog 0.4.0 h49b9bf7_3 conda-forge google 3.0.0 pypi_0 pypi google-cloud-cpp 1.16.0 he4a878c_2 conda-forge google-cloud-cpp-common 0.25.0 he83eced_7 conda-forge googleapis-cpp 0.10.0 h6b1abdc_4 conda-forge gpuci-tools 0.2.1 5 gpuci graphite2 1.3.13 h58526e2_1001 conda-forge grpc-cpp 1.32.0 h7997a97_1 conda-forge grpcio 1.34.0 pypi_0 pypi gtest 1.10.0 h0efe328_4 conda-forge harfbuzz 2.4.0 h9f30f68_3 conda-forge hdf4 4.2.13 hf30be14_1003 conda-forge hdf5 1.10.6 nompi_h7c3c948_1111 conda-forge heapdict 1.0.1 py_0 conda-forge icu 64.2 he1b5a44_1 conda-forge idna 2.10 pyh9f0ad1d_0 conda-forge imageio 2.9.0 py_0 conda-forge iml 0.6.2 pypi_0 pypi importlib-metadata 3.1.1 pyhd8ed1ab_0 conda-forge importlib_metadata 1.5.0 py37_0 conda-forge ipykernel 5.4.0 py37h888b3d9_0 conda-forge ipython 7.19.0 py37h888b3d9_0 conda-forge ipython_genutils 0.2.0 py_1 conda-forge ipywidgets 7.5.1 pyh9f0ad1d_1 conda-forge itsdangerous 1.1.0 py_0 conda-forge jansson 2.11 h516909a_1001 conda-forge jdcal 1.4.1 py_0 conda-forge jedi 0.17.2 py37h89c1867_1 conda-forge jinja2 2.11.2 pyh9f0ad1d_0 conda-forge jinja2-time 0.2.0 py_2 conda-forge joblib 0.17.0 py_0 conda-forge jpeg 9d h36c2ea0_0 conda-forge jpype1 1.1.2 py37h99015e2_0 conda-forge json-c 0.13.1 hbfbb72e_1002 conda-forge jsonschema 3.2.0 py_2 conda-forge jupyter-server-proxy 1.5.0 py_0 conda-forge jupyter_client 6.1.7 py_0 conda-forge jupyter_core 4.7.0 py37h89c1867_0 conda-forge jupyterlab_pygments 0.1.2 pyh9f0ad1d_0 conda-forge kealib 1.4.13 h33137a7_1 conda-forge kiwisolver 1.3.1 py37hc928c03_0 conda-forge kombu 4.6.11 py37hc8dfbb8_0 conda-forge krb5 1.17.2 h926e7f8_0 conda-forge lcms2 2.11 hcbb858e_1 conda-forge ld_impl_linux-64 2.35.1 hed1e6ac_0 conda-forge libblas 3.8.0 17_openblas conda-forge libcblas 3.8.0 17_openblas conda-forge libcrc32c 1.1.1 he1b5a44_2 conda-forge libcudf 0.17.0a201210 cuda11.0_g00ca24625e_383 rapidsai-nightly libcudf_kafka 0.17.0a201209 gbd321d1e93_382 rapidsai-nightly libcugraph 0.17.0a201210 cuda11.0_ge205fd07_289 rapidsai-nightly libcuml 0.17.0a201210 cuda11.0_g2c0aacf44_173 rapidsai-nightly libcumlprims 0.17.0a201030 cuda11.0_g1fa28a5_8 rapidsai-nightly libcurl 7.71.1 hcdd3856_8 conda-forge libcuspatial 0.17.0a201210 cuda11.0_g9aeebc8_23 rapidsai-nightly libdap4 3.20.6 h1d1bd15_1 conda-forge libedit 3.1.20191231 he28a2e2_2 conda-forge libev 4.33 h516909a_1 conda-forge libevent 2.1.10 hcdb4288_3 conda-forge libfaiss 1.6.3 h328c4c8_1_cuda rapidsai-nightly libffi 3.2.1 he1b5a44_1007 conda-forge libgcc-ng 7.5.0 h5dbcf3e_17 conda-forge libgcrypt 1.8.7 h36c2ea0_0 conda-forge libgdal 3.0.4 he6a97d6_10 conda-forge libgfortran-ng 7.5.0 hae1eefd_17 conda-forge libgfortran4 7.5.0 hae1eefd_17 conda-forge libglib 2.66.3 hbe7bbb4_0 conda-forge libgpg-error 1.39 he1b5a44_0 conda-forge libgsasl 1.8.0 2 conda-forge libhwloc 2.3.0 h5e5b7d1_1 conda-forge libiconv 1.16 h516909a_0 conda-forge libkml 1.3.0 hd79254b_1012 conda-forge liblapack 3.8.0 17_openblas conda-forge libllvm10 10.0.1 he513fc3_3 conda-forge libnetcdf 4.7.4 nompi_hefab0ff_106 conda-forge libnghttp2 1.41.0 h8cfc5f6_2 conda-forge libntlm 1.4 h516909a_1002 conda-forge libopenblas 0.3.10 pthreads_hb3c22a3_5 conda-forge libpng 1.6.37 h21135ba_2 conda-forge libpq 12.3 h255efa7_3 conda-forge libprotobuf 3.13.0.1 h8b12597_0 conda-forge librdkafka 1.5.2 h40bdf00_0 conda-forge librmm 0.17.0a201210 cuda11.0_gb8c8310_60 rapidsai-nightly libsodium 1.0.18 h36c2ea0_1 conda-forge libspatialindex 1.9.3 he1b5a44_3 conda-forge libspatialite 4.3.0a h2482549_1038 conda-forge libssh2 1.9.0 hab1572f_5 conda-forge libstdcxx-ng 7.5.0 h2ae2ef3_17 conda-forge libthrift 0.13.0 h5aa387f_6 conda-forge libtiff 4.1.0 h4f3a223_6 conda-forge libutf8proc 2.6.0 h36c2ea0_0 conda-forge libuuid 2.32.1 h14c3975_1000 conda-forge libuv 1.34.0 h516909a_0 conda-forge libwebp 1.1.0 h76fa15c_4 conda-forge libwebp-base 1.1.0 h36c2ea0_3 conda-forge libxcb 1.13 h14c3975_1002 conda-forge libxgboost 1.3.0dev.rapidsai0.17 cuda11.0_0 rapidsai-nightly libxml2 2.9.10 hee79883_0 conda-forge libxslt 1.1.33 h31b3aaa_0 conda-forge limits 1.5.1 pypi_0 pypi llvm-openmp 11.0.0 hfc4b9b4_1 conda-forge llvmlite 0.34.0 py37h5202443_2 conda-forge locket 0.2.0 py_2 conda-forge lxml 4.5.1 py37he3881c9_0 conda-forge lz4-c 1.9.2 he1b5a44_3 conda-forge mako 1.1.3 pypi_0 pypi markdown 3.2.2 py_0 conda-forge markupsafe 1.1.1 py37hb5d75c8_2 conda-forge marshmallow 2.20.5 py_0 conda-forge marshmallow-sqlalchemy 0.23.0 py_0 conda-forge matplotlib-base 3.3.3 py37h4f6019d_0 conda-forge mistune 0.8.4 py37h4abf009_1002 conda-forge modal 0.3.5 pypi_0 pypi more-itertools 8.6.0 pyhd8ed1ab_0 conda-forge msgpack-python 1.0.0 py37hc928c03_2 conda-forge multidict 4.7.5 py37h8f50634_2 conda-forge multipledispatch 0.6.0 py_0 conda-forge munch 2.5.0 py_0 conda-forge nbclient 0.5.1 py_0 conda-forge nbconvert 6.0.7 py37h89c1867_3 conda-forge nbformat 5.0.8 py_0 conda-forge nccl 2.7.8.1 h4962215_100 nvidia ncurses 6.2 h58526e2_4 conda-forge nest-asyncio 1.4.3 pyhd8ed1ab_0 conda-forge netifaces 0.10.9 py37h8f50634_1003 conda-forge networkx 2.5 py_0 conda-forge nodejs 13.13.0 hf5d1a2b_0 conda-forge notebook 6.1.5 py37h89c1867_0 conda-forge numba 0.51.2 py37h9fdb41a_0 conda-forge numpy 1.19.4 py37h7e9df27_1 conda-forge nvtx 0.2.1 py37h8f50634_2 conda-forge olefile 0.46 pyh9f0ad1d_1 conda-forge openjdk 11.0.8 hacce0ff_0 conda-forge openjpeg 2.3.1 h981e76c_3 conda-forge openpyxl 3.0.3 py_0 conda-forge openssl 1.1.1h h516909a_0 conda-forge optuna 2.0.0 pypi_0 pypi orc 1.6.5 hd3605a7_0 conda-forge packaging 20.7 pyhd3deb0d_0 conda-forge pandas 1.1.4 py37h10a2094_0 conda-forge pandoc 2.11.2 h36c2ea0_0 conda-forge pandocfilters 1.4.2 py_1 conda-forge panel 0.9.7 py_0 conda-forge param 1.10.0 py_0 conda-forge parquet-cpp 1.5.1 2 conda-forge parso 0.7.1 pyh9f0ad1d_0 conda-forge partd 1.1.0 py_0 conda-forge passlib 1.7.4 pyh9f0ad1d_0 conda-forge patsy 0.5.1 py_0 conda-forge pbr 5.5.1 pypi_0 pypi pcre 8.44 he1b5a44_0 conda-forge perl 5.32.0 h36c2ea0_0 conda-forge pexpect 4.8.0 pyh9f0ad1d_2 conda-forge pickle5 0.0.11 py37h8f50634_0 conda-forge pickleshare 0.7.5 py_1003 conda-forge pillow 8.0.1 py37h63a5d19_0 conda-forge pip 20.3.3 pyhd8ed1ab_0 conda-forge pixman 0.38.0 h516909a_1003 conda-forge plotly 4.9.0 pypi_0 pypi pluggy 0.13.1 py37he5f6b98_3 conda-forge poppler 0.87.0 h4190859_1 conda-forge poppler-data 0.4.10 0 conda-forge postgresql 12.3 hc2f5b80_3 conda-forge prettytable 0.7.2 pypi_0 pypi proj 7.0.0 h966b41f_5 conda-forge prometheus_client 0.9.0 pyhd3deb0d_0 conda-forge prompt-toolkit 3.0.8 pyha770c72_0 conda-forge protobuf 3.13.0.1 py37h745909e_1 conda-forge psutil 5.7.3 py37hb5d75c8_0 conda-forge psycopg2-binary 2.8.5 pypi_0 pypi pthread-stubs 0.4 h36c2ea0_1001 conda-forge ptyprocess 0.6.0 py_1001 conda-forge py 1.10.0 pyhd3deb0d_0 conda-forge py-spy 0.3.3 pypi_0 pypi py-xgboost 1.3.0dev.rapidsai0.17 cuda11.0py37_0 rapidsai-nightly pyarrow 1.0.1 py37hbeecfa9_14_cuda conda-forge pycparser 2.20 pyh9f0ad1d_2 conda-forge pyct 0.4.6 py_0 conda-forge pyct-core 0.4.6 py_0 conda-forge pydeck 0.5.0 pyh9f0ad1d_0 conda-forge pyee 7.0.4 pyh9f0ad1d_0 conda-forge pygments 2.7.3 pyhd8ed1ab_0 conda-forge pyhive 0.6.2 pyh9f0ad1d_0 conda-forge pynvml 8.0.4 py_1 conda-forge pyopenssl 20.0.0 pyhd8ed1ab_0 conda-forge pyparsing 2.4.7 pyh9f0ad1d_0 conda-forge pyperclip 1.8.1 pypi_0 pypi pyppeteer 0.2.2 py_1 conda-forge pyproj 2.6.1.post1 py37h34dd122_0 conda-forge pyrsistent 0.17.3 py37h4abf009_1 conda-forge pysocks 1.7.1 py37he5f6b98_2 conda-forge pytest 5.4.2 py37hc8dfbb8_0 conda-forge pytest-flask 1.0.0 py_0 conda-forge python 3.7.8 h8bdb77d_2_cpython conda-forge python-confluent-kafka 1.5.0 py37h8f50634_0 conda-forge python-dateutil 2.8.1 py_0 conda-forge python-editor 1.0.4 pypi_0 pypi python-frontmatter 0.5.0 pypi_0 pypi python-graphviz 0.16 pypi_0 pypi python_abi 3.7 1_cp37m conda-forge pytz 2020.4 pyhd8ed1ab_0 conda-forge pyviz_comms 0.7.6 pyh9f0ad1d_0 conda-forge pywavelets 1.1.1 py37h161383b_3 conda-forge pyyaml 5.1.2 py37h516909a_0 conda-forge pyzmq 20.0.0 py37h5a562af_1 conda-forge rapids 0.17.0a201210 cuda11.0_py37_gdc7a92a_161 rapidsai-nightly rapids-blazing 0.17.0a201210 cuda11.0_py37_gdc7a92a_161 rapidsai-nightly rapids-xgboost 0.17.0a201210 cuda11.0_py37_gdc7a92a_161 rapidsai-nightly ray 0.8.6 pypi_0 pypi re2 2020.10.01 he1b5a44_0 conda-forge readline 8.0 he28a2e2_2 conda-forge redis 3.4.1 pypi_0 pypi requests 2.25.0 pyhd3deb0d_0 conda-forge retrying 1.3.3 pypi_0 pypi rmm 0.17.0a201210 cuda_11.0_py37_gb8c8310_60 rapidsai-nightly rtree 0.9.4 py37h0b55af0_2 conda-forge sasl 0.2.1 py37h3340039_1002 conda-forge scikit-image 0.16.2 py37hb3f55d8_0 conda-forge scikit-learn 0.23.2 py37hddcf8d6_3 conda-forge scipy 1.5.3 py37h8911b10_0 conda-forge send2trash 1.5.0 py_0 conda-forge setuptools 49.6.0 py37he5f6b98_2 conda-forge shap 0.36.0 py37h9fdb41a_0 conda-forge shapely 1.7.1 py37hedb1597_1 conda-forge simpervisor 0.3 py_1 conda-forge six 1.15.0 pyh9f0ad1d_0 conda-forge slicer 0.0.7 pyhd8ed1ab_0 conda-forge snappy 1.1.8 he1b5a44_3 conda-forge sortedcontainers 2.3.0 pyhd8ed1ab_0 conda-forge soupsieve 2.1 pypi_0 pypi spdlog 1.7.0 hc9558a2_2 conda-forge speaklater 1.3 py_3 conda-forge sqlalchemy 1.3.20 py37h8f50634_0 conda-forge sqlite 3.33.0 h4cf870e_1 conda-forge statsmodels 0.12.1 py37ha21ca33_1 conda-forge stevedore 3.3.0 pypi_0 pypi streamz 0.6.1 pyhd3deb0d_1 conda-forge tabulate 0.8.7 pypi_0 pypi tbb 2020.2 hc9558a2_0 conda-forge tblib 1.6.0 py_0 conda-forge tensorboardx 2.1 pypi_0 pypi terminado 0.9.1 py37h89c1867_1 conda-forge testpath 0.4.4 py_0 conda-forge text-unidecode 1.3 py_0 conda-forge threadpoolctl 2.1.0 pyh5ca1d4c_0 conda-forge thrift 0.13.0 py37h3340039_2 conda-forge thrift_sasl 0.4.2 py37h8f50634_0 conda-forge tiledb 1.7.7 h8efa9f0_3 conda-forge tk 8.6.10 hed695b0_1 conda-forge toml 0.10.2 pyhd8ed1ab_0 conda-forge toolz 0.11.1 py_0 conda-forge tornado 6.1 py37h4abf009_0 conda-forge tox 3.15.0 py37hc8dfbb8_0 conda-forge tqdm 4.54.1 pyhd8ed1ab_0 conda-forge traitlets 5.0.5 py_0 conda-forge treelite 0.93 py37h745909e_3 conda-forge treelite-runtime 0.93 pypi_0 pypi typing-extensions 3.7.4.3 0 conda-forge typing_extensions 3.7.4.3 py_0 conda-forge tzcode 2020a h516909a_0 conda-forge tzlocal 2.1 pyh9f0ad1d_0 conda-forge ucx 1.8.1+g6b29558 cuda11.0_0 rapidsai-nightly ucx-proc 1.0.0 gpu rapidsai-nightly ucx-py 0.17.0a201210 py37_g6b29558_27 rapidsai-nightly urllib3 1.25.11 py_0 conda-forge uwsgi 2.0.18 py37h79bd928_3 conda-forge vine 1.3.0 py_0 conda-forge virtualenv 16.7.5 py_0 conda-forge wcwidth 0.2.5 pyh9f0ad1d_2 conda-forge webencodings 0.5.1 py_1 conda-forge websockets 8.1 py37h8f50634_2 conda-forge werkzeug 1.0.1 pyh9f0ad1d_0 conda-forge wheel 0.36.1 pyhd3deb0d_0 conda-forge widgetsnbextension 3.5.1 py37h89c1867_4 conda-forge wtforms 2.3.2 pyh9f0ad1d_0 conda-forge xarray 0.16.2 pyhd8ed1ab_0 conda-forge xerces-c 3.2.2 h8412b87_1004 conda-forge xgboost 1.3.0dev.rapidsai0.17 cuda11.0py37_0 rapidsai-nightly xorg-fixesproto 5.0 h14c3975_1002 conda-forge xorg-inputproto 2.3.2 h14c3975_1002 conda-forge xorg-kbproto 1.0.7 h14c3975_1002 conda-forge xorg-libice 1.0.10 h516909a_0 conda-forge xorg-libsm 1.2.3 h84519dc_1000 conda-forge xorg-libx11 1.6.12 h516909a_0 conda-forge xorg-libxau 1.0.9 h14c3975_0 conda-forge xorg-libxdmcp 1.1.3 h516909a_0 conda-forge xorg-libxext 1.3.4 h516909a_0 conda-forge xorg-libxfixes 5.0.3 h516909a_1004 conda-forge xorg-libxi 1.7.10 h516909a_0 conda-forge xorg-libxrender 0.9.10 h516909a_1002 conda-forge xorg-libxtst 1.2.3 h516909a_1002 conda-forge xorg-recordproto 1.14.2 h516909a_1002 conda-forge xorg-renderproto 0.11.1 h14c3975_1002 conda-forge xorg-xextproto 7.3.0 h14c3975_1002 conda-forge xorg-xproto 7.0.31 h14c3975_1007 conda-forge xz 5.2.5 h516909a_1 conda-forge yaml 0.1.7 h14c3975_1001 conda-forge yarl 1.6.3 py37h4abf009_0 conda-forge zeromq 4.3.3 h58526e2_3 conda-forge zict 2.0.0 py_0 conda-forge zipp 3.4.0 py_0 conda-forge zlib 1.2.11 h516909a_1010 conda-forge zstd 1.4.5 h6597ccf_2 conda-forgeAdditional context We get this problem from version 0.14. Setting
--pool soloin celery worker fixes it but is not something we want. It may mean that the error is related to child process not being initialised correctly... It could be because the cuda init happens before celery actually starts the child process.