TypeError: cannot pickle '_thread.RLock' object

[jdwillard19]

This following code does not execute for me and gives the stack trace shown after. It appears I cannot even declare dask storage on a LocalCUDACluster.

from dask.distributed import Client
import dask.distributed
import dask_optuna
import joblib
from dask_cuda import LocalCUDACluster

def lstm_HPO_dask(timeout=100, hpo_type=None,n_jobs=5):

        if __name__ == '__main__':
            cluster = LocalCUDACluster()
            with Client(cluster) as client:

                storage = dask_optuna.DaskStorage(cluster)

                study = optuna.create_study(storage=storage,direction='minimize')

                # Optimize in parallel on your Dask cluster
                with joblib.parallel_backend("dask"):
                    study.optimize(lstm_cv_obj, n_trials=100, n_jobs=n_jobs)

                print(f"best_params = {study.best_params}")
        return None

lstm_HPO_dask(timeout=N_SECONDS,n_jobs=n_jobs)

`

Stack trace below

-> storage = dask_optuna.DaskStorage(cluster) (Pdb) c Traceback (most recent call last): File "/global/homes/j/jwillard/miniconda3/envs/stml/lib/python3.9/site-packages/distributed/protocol/pickle.py", line 49, in dumps result = pickle.dumps(x, **dump_kwargs) TypeError: cannot pickle '_thread.RLock' object

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/global/u2/j/jwillard/stream-temperature-ml/workflows/hpc/to_submit/lstm_hpo_regional_dask.py", line 162, in lstm_HPO_dask(timeout=N_SECONDS,n_jobs=n_jobs) File "/global/u2/j/jwillard/stream-temperature-ml/workflows/hpc/to_submit/lstm_hpo_regional_dask.py", line 147, in lstm_HPO_dask storage = dask_optuna.DaskStorage(cluster) File "/global/homes/j/jwillard/miniconda3/envs/stml/lib/python3.9/site-packages/dask_optuna/storage.py", line 325, in init self.client.run_on_scheduler( File "/global/homes/j/jwillard/miniconda3/envs/stml/lib/python3.9/site-packages/distributed/client.py", line 2406, in run_on_scheduler return self.sync(self._run_on_scheduler, function, *args, kwargs) File "/global/homes/j/jwillard/miniconda3/envs/stml/lib/python3.9/site-packages/distributed/client.py", line 860, in sync return sync( File "/global/homes/j/jwillard/miniconda3/envs/stml/lib/python3.9/site-packages/distributed/utils.py", line 326, in sync raise exc.with_traceback(tb) File "/global/homes/j/jwillard/miniconda3/envs/stml/lib/python3.9/site-packages/distributed/utils.py", line 309, in f result[0] = yield future File "/global/homes/j/jwillard/miniconda3/envs/stml/lib/python3.9/site-packages/tornado/gen.py", line 762, in run value = future.result() File "/global/homes/j/jwillard/miniconda3/envs/stml/lib/python3.9/site-packages/distributed/client.py", line 2368, in _run_on_scheduler kwargs=dumps(kwargs, protocol=4), File "/global/homes/j/jwillard/miniconda3/envs/stml/lib/python3.9/site-packages/distributed/protocol/pickle.py", line 60, in dumps result = cloudpickle.dumps(x, dump_kwargs) File "/global/homes/j/jwillard/miniconda3/envs/stml/lib/python3.9/site-packages/cloudpickle/cloudpickle_fast.py", line 73, in dumps

Environment consists of the following packages _libgcc_mutex 0.1 conda_forge conda-forge _openmp_mutex 4.5 1_gnu conda-forge _py-xgboost-mutex 2.0 cpu_0
abseil-cpp 20210324.2 h9c3ff4c_0 conda-forge affine 2.3.0 py_0 anaconda alembic 1.7.4 pyhd8ed1ab_0 conda-forge alsa-lib 1.2.3 h516909a_0 conda-forge arrow-cpp 5.0.0 py39h247a7eb_8_cpu conda-forge attrs 21.2.0 pyhd8ed1ab_0 conda-forge autopage 0.4.0 pyhd8ed1ab_0 conda-forge aws-c-cal 0.5.11 h95a6274_0 conda-forge aws-c-common 0.6.2 h7f98852_0 conda-forge aws-c-event-stream 0.2.7 h3541f99_13 conda-forge aws-c-io 0.10.5 hfb6a706_0 conda-forge aws-checksums 0.1.11 ha31a3da_7 conda-forge aws-sdk-cpp 1.8.186 hb4091e7_3 conda-forge backcall 0.2.0 pyh9f0ad1d_0 conda-forge backports 1.0 py_2 anaconda backports.functools_lru_cache 1.6.4 pyhd8ed1ab_0 conda-forge blas 1.0 mkl anaconda blosc 1.21.0 h9c3ff4c_0 conda-forge bokeh 2.4.2 py39hf3d152e_0 conda-forge boost-cpp 1.74.0 h312852a_4 conda-forge brotlipy 0.7.0 py39h3811e60_1001 conda-forge bzip2 1.0.8 h7f98852_4 conda-forge c-ares 1.18.1 h7f98852_0 conda-forge ca-certificates 2021.10.8 ha878542_0 conda-forge cached-property 1.5.2 hd8ed1ab_1 conda-forge cached_property 1.5.2 pyha770c72_1 conda-forge cairo 1.16.0 h6cf1ce9_1008 conda-forge certifi 2021.10.8 py39hf3d152e_1 conda-forge cffi 1.14.6 py39h4bc2ebd_1 conda-forge cfitsio 4.0.0 h9a35b8e_0 conda-forge chardet 4.0.0 py39hf3d152e_1 conda-forge charset-normalizer 2.0.0 pyhd8ed1ab_0 conda-forge click 8.0.3 py39hf3d152e_1 conda-forge click-plugins 1.1.1 py_0 anaconda cliff 3.9.0 pyhd8ed1ab_0 conda-forge cligj 0.7.2 pyhd8ed1ab_1 conda-forge cloudpickle 2.0.0 pyhd8ed1ab_0 conda-forge cmaes 0.8.2 pyh44b312d_0 conda-forge cmd2 2.2.0 py39hf3d152e_0 conda-forge colorama 0.4.4 pyh9f0ad1d_0 conda-forge colorlog 6.5.0 py39hf3d152e_0 conda-forge cryptography 35.0.0 py39h95dcef6_1 conda-forge cuda-python 11.6.0 py39h3fd9d12_0 nvidia cudatoolkit 11.3.1 ha36c431_9 nvidia curl 7.80.0 h2574ce0_0 conda-forge cycler 0.10.0 py_2 conda-forge cytoolz 0.11.2 py39h3811e60_1 conda-forge dask 2021.9.1 pyhd8ed1ab_0 conda-forge dask-core 2021.9.1 pyhd8ed1ab_0 conda-forge dask-cuda 21.10.0 pyhd8ed1ab_0 conda-forge dask-jobqueue 0.7.3 pyhd8ed1ab_0 conda-forge dask-mpi 2021.11.0 py39hdc36942_0 conda-forge dask-optuna 0.0.2 pypi_0 pypi dbus 1.13.6 h48d8840_2 conda-forge debugpy 1.5.1 py39he80948d_0 conda-forge decorator 5.1.0 pyhd8ed1ab_0 conda-forge distributed 2021.9.1 py39hf3d152e_0 conda-forge entrypoints 0.3 pyhd8ed1ab_1003 conda-forge expat 2.4.1 h9c3ff4c_0 conda-forge ffmpeg 4.3 hf484d3e_0 pytorch font-ttf-dejavu-sans-mono 2.37 hab24e00_0 conda-forge font-ttf-inconsolata 3.000 h77eed37_0 conda-forge font-ttf-source-code-pro 2.038 h77eed37_0 conda-forge font-ttf-ubuntu 0.83 hab24e00_0 conda-forge fontconfig 2.13.1 hba837de_1005 conda-forge fonts-conda-ecosystem 1 0 conda-forge fonts-conda-forge 1 0 conda-forge freetype 2.10.4 h0708190_1 conda-forge freexl 1.0.6 h7f98852_0 conda-forge fsspec 2022.1.0 pyhd8ed1ab_0 conda-forge geos 3.10.0 h9c3ff4c_0 conda-forge geotiff 1.7.0 hcfb7246_3 conda-forge gettext 0.19.8.1 h73d1719_1008 conda-forge gflags 2.2.2 he1b5a44_1004 conda-forge giflib 5.2.1 h36c2ea0_2 conda-forge glib 2.70.0 h780b84a_1 conda-forge glib-tools 2.70.0 h780b84a_1 conda-forge glog 0.5.0 h48cff8f_0 conda-forge gmp 6.2.1 h58526e2_0 conda-forge gnutls 3.6.13 h85f3911_1 conda-forge greenlet 1.1.2 py39he80948d_0 conda-forge grpc-cpp 1.40.0 h05f19cf_2 conda-forge gst-plugins-base 1.18.5 hf529b03_0 conda-forge gstreamer 1.18.5 h76c114f_0 conda-forge h5py 3.4.0 nompi_py39h7e08c79_101 conda-forge hdf4 4.2.15 h10796ff_3 conda-forge hdf5 1.12.1 nompi_h2750804_101 conda-forge heapdict 1.0.1 py_0 conda-forge icu 68.2 h9c3ff4c_0 conda-forge idna 3.1 pyhd3deb0d_0 conda-forge importlib-metadata 4.8.1 py39hf3d152e_0 conda-forge importlib_resources 5.3.0 pyhd8ed1ab_0 conda-forge intel-openmp 2021.3.0 h06a4308_3350
ipykernel 6.4.2 py39hef51801_0 conda-forge ipython 7.29.0 py39hef51801_1 conda-forge ipython_genutils 0.2.0 py_1 conda-forge jbig 2.1 h7f98852_2003 conda-forge jedi 0.18.0 py39hf3d152e_3 conda-forge jinja2 3.0.3 pyhd8ed1ab_0 conda-forge joblib 0.17.0 py_0 anaconda jpeg 9d h36c2ea0_0 conda-forge json-c 0.15 h98cffda_0 conda-forge jupyter_client 7.0.6 pyhd8ed1ab_0 conda-forge jupyter_core 4.9.1 py39hf3d152e_0 conda-forge kealib 1.4.14 h87e4c3c_3 conda-forge kiwisolver 1.3.2 py39h1a9c180_0 conda-forge krb5 1.19.2 hcc1bbae_2 conda-forge lame 3.100 h7f98852_1001 conda-forge lcms2 2.12 hddcbb42_0 conda-forge ld_impl_linux-64 2.36.1 hea4e1c9_2 conda-forge lerc 3.0 h9c3ff4c_0 conda-forge libblas 3.9.0 11_linux64_mkl conda-forge libbrotlicommon 1.0.9 h7f98852_5 conda-forge libbrotlidec 1.0.9 h7f98852_5 conda-forge libbrotlienc 1.0.9 h7f98852_5 conda-forge libcblas 3.9.0 11_linux64_mkl conda-forge libclang 11.1.0 default_ha53f305_1 conda-forge libcurl 7.80.0 h2574ce0_0 conda-forge libdap4 3.20.6 hd7c4107_2 conda-forge libdeflate 1.8 h7f98852_0 conda-forge libedit 3.1.20191231 he28a2e2_2 conda-forge libev 4.33 h516909a_1 conda-forge libevent 2.1.10 h9b69904_4 conda-forge libffi 3.4.2 h9c3ff4c_4 conda-forge libgcc-ng 11.2.0 h1d223b6_11 conda-forge libgdal 3.3.3 h18e3bf0_2 conda-forge libgfortran-ng 11.2.0 h69a702a_11 conda-forge libgfortran5 11.2.0 h5c6108e_11 conda-forge libglib 2.70.0 h174f98d_1 conda-forge libgomp 11.2.0 h1d223b6_11 conda-forge libiconv 1.16 h516909a_0 conda-forge libkml 1.3.0 h238a007_1014 conda-forge liblapack 3.9.0 11_linux64_mkl conda-forge libllvm11 11.1.0 hf817b99_2 conda-forge libnetcdf 4.8.1 nompi_hb3fd0d9_101 conda-forge libnghttp2 1.43.0 h812cca2_1 conda-forge libnsl 2.0.0 h7f98852_0 conda-forge libogg 1.3.4 h7f98852_1 conda-forge libopus 1.3.1 h7f98852_1 conda-forge libpng 1.6.37 h21135ba_2 conda-forge libpq 13.5 hd57d9b9_1 conda-forge libprotobuf 3.18.1 h780b84a_0 conda-forge librmm 22.02.00 hbecc9b5_0 conda-forge librttopo 1.1.0 h0ad649c_7 conda-forge libsodium 1.0.18 h36c2ea0_1 conda-forge libspatialite 5.0.1 h1d9e4f1_10 conda-forge libssh2 1.10.0 ha56f1ee_2 conda-forge libstdcxx-ng 11.2.0 he4da1e4_11 conda-forge libthrift 0.15.0 he6d91bd_1 conda-forge libtiff 4.3.0 h6f004c6_2 conda-forge libutf8proc 2.6.1 h7f98852_0 conda-forge libuuid 2.32.1 h7f98852_1000 conda-forge libuv 1.42.0 h7f98852_0 conda-forge libvorbis 1.3.7 h9c3ff4c_0 conda-forge libwebp-base 1.2.1 h7f98852_0 conda-forge libxcb 1.13 h7f98852_1003 conda-forge libxgboost 1.5.0 h295c915_1
libxkbcommon 1.0.3 he3ba5ed_0 conda-forge libxml2 2.9.12 h72842e0_0 conda-forge libzip 1.8.0 h4de3113_1 conda-forge libzlib 1.2.11 h36c2ea0_1013 conda-forge llvmlite 0.37.0 py39h1bbdace_1 conda-forge locket 0.2.0 py_2 conda-forge lz4-c 1.9.3 h9c3ff4c_1 conda-forge lzo 2.10 h516909a_1000 conda-forge mako 1.1.5 pyhd8ed1ab_0 conda-forge markupsafe 2.0.1 py39h3811e60_0 conda-forge matplotlib 3.4.3 py39hf3d152e_1 conda-forge matplotlib-base 3.4.3 py39h2fa2bec_1 conda-forge matplotlib-inline 0.1.3 pyhd8ed1ab_0 conda-forge mkl 2021.3.0 h06a4308_520
mkl-service 2.4.0 py39h3811e60_0 conda-forge mkl_fft 1.3.1 py39hd3c417c_0
mkl_random 1.2.2 py39hde0f152_0 conda-forge mock 4.0.3 py39hf3d152e_1 conda-forge mpi 1.0 openmpi conda-forge mpi4py 3.1.3 py39hc649577_0 conda-forge msgpack-python 1.0.3 py39h1a9c180_0 conda-forge mysql-common 8.0.27 ha770c72_0 conda-forge mysql-libs 8.0.27 hfa10184_0 conda-forge ncurses 6.2 h58526e2_4 conda-forge nest-asyncio 1.5.1 pyhd8ed1ab_0 conda-forge nettle 3.6 he412f7d_0 conda-forge nspr 4.30 h9c3ff4c_0 conda-forge nss 3.69 hb5efdd6_1 conda-forge numba 0.54.1 py39h56b8d98_0 conda-forge numexpr 2.7.3 py39hde0f152_0 conda-forge numpy 1.20.3 py39hdbf815f_1 conda-forge olefile 0.46 pyh9f0ad1d_1 conda-forge openh264 2.1.1 h780b84a_0 conda-forge openjpeg 2.4.0 hb52868f_1 conda-forge openmpi 4.1.2 hbfc84c5_0 conda-forge openssl 1.1.1l h7f98852_0 conda-forge optuna 2.1.0 py_0 conda-forge orc 1.7.0 h68e2c4e_0 conda-forge packaging 21.0 pyhd8ed1ab_0 conda-forge pandas 1.4.0 py39hde0f152_0 conda-forge parquet-cpp 1.5.1 2 conda-forge parso 0.8.2 pyhd8ed1ab_0 conda-forge partd 1.2.0 pyhd8ed1ab_0 conda-forge patsy 0.5.2 pyhd8ed1ab_0 conda-forge pbr 5.6.0 pyhd8ed1ab_0 conda-forge pcre 8.45 h9c3ff4c_0 conda-forge pexpect 4.8.0 pyh9f0ad1d_2 conda-forge pickleshare 0.7.5 py_1003 conda-forge pillow 8.3.2 py39ha612740_0 conda-forge pip 21.3.1 pyhd8ed1ab_0 conda-forge pixman 0.40.0 h36c2ea0_0 conda-forge poppler 21.09.0 ha39eefc_3 conda-forge poppler-data 0.4.11 hd8ed1ab_0 conda-forge postgresql 13.5 h2510834_1 conda-forge prettytable 2.2.1 pyhd8ed1ab_0 conda-forge proj 8.1.1 h277dcde_2 conda-forge prompt-toolkit 3.0.22 pyha770c72_0 conda-forge psutil 5.9.0 py39h3811e60_0 conda-forge pthread-stubs 0.4 h36c2ea0_1001 conda-forge ptyprocess 0.7.0 pyhd3deb0d_0 conda-forge py-xgboost 1.5.0 py39h06a4308_1
pyarrow 5.0.0 py39h3ebc44c_8_cpu conda-forge pycparser 2.20 pyh9f0ad1d_2 conda-forge pygments 2.10.0 pyhd8ed1ab_0 conda-forge pynvml 11.4.1 pyhd8ed1ab_0 conda-forge pyopenssl 21.0.0 pyhd8ed1ab_0 conda-forge pyparsing 3.0.3 pyhd8ed1ab_0 conda-forge pyperclip 1.8.2 pyhd8ed1ab_2 conda-forge pyqt 5.12.3 py39hf3d152e_7 conda-forge pyqt-impl 5.12.3 py39h0fcd23e_7 conda-forge pyqt5-sip 4.19.18 py39he80948d_7 conda-forge pyqtchart 5.12 py39h0fcd23e_7 conda-forge pyqtwebengine 5.12.1 py39h0fcd23e_7 conda-forge pysocks 1.7.1 py39hf3d152e_3 conda-forge pytables 3.6.1 py39h2669a42_4 conda-forge python 3.9.7 hb7a2778_3_cpython conda-forge python-dateutil 2.8.2 pyhd8ed1ab_0 conda-forge python_abi 3.9 2_cp39 conda-forge pytorch 1.10.0 py3.9_cuda11.3_cudnn8.2.0_0 pytorch pytorch-mutex 1.0 cuda pytorch pytz 2021.3 pyhd8ed1ab_0 conda-forge pyyaml 6.0 py39h3811e60_3 conda-forge pyzmq 22.3.0 py39h37b5a0c_1 conda-forge qt 5.12.9 hda022c4_4 conda-forge rasterio 1.2.10 py39hb37810a_0 conda-forge re2 2021.09.01 h9c3ff4c_0 conda-forge readline 8.1 h46c0cb4_0 conda-forge requests 2.26.0 pyhd8ed1ab_0 conda-forge rmm 22.02.00 cuda11_py39_ge3e3215_0_has_cma rapidsai s2n 1.0.10 h9b69904_0 conda-forge scikit-learn 1.0.1 py39h7c5d8c9_1 conda-forge scipy 1.7.1 py39hee8e79c_0 conda-forge seaborn 0.11.0 py_0 anaconda setuptools 58.2.0 py39hf3d152e_0 conda-forge shap 0.40.0 py39hde0f152_0 conda-forge six 1.16.0 pyh6c4a22f_0 conda-forge slicer 0.0.7 pyhd8ed1ab_0 conda-forge snappy 1.1.8 he1b5a44_3 conda-forge snuggs 1.4.7 py_0 anaconda sortedcontainers 2.4.0 pyhd8ed1ab_0 conda-forge spdlog 1.9.2 h4bd325d_1 conda-forge sqlalchemy 1.4.26 py39h3811e60_0 conda-forge sqlite 3.36.0 h9cd32fc_2 conda-forge statsmodels 0.13.0 py39hce5d2b2_0 conda-forge stevedore 3.5.0 py39hf3d152e_0 conda-forge tblib 1.7.0 pyhd8ed1ab_0 conda-forge threadpoolctl 2.1.0 pyh5ca1d4c_0 anaconda thrust 1.11.0 ha770c72_1 conda-forge tiledb 2.3.4 he87e0bf_0 conda-forge tk 8.6.11 h27826a3_1 conda-forge toolz 0.11.2 pyhd8ed1ab_0 conda-forge torchaudio 0.10.0 py39_cu113 pytorch torchvision 0.11.1 py39_cu113 pytorch tornado 6.1 py39h3811e60_1 conda-forge tqdm 4.62.3 pyhd8ed1ab_0 conda-forge traitlets 5.1.1 pyhd8ed1ab_0 conda-forge typing_extensions 3.10.0.2 pyha770c72_0 conda-forge tzcode 2021e h7f98852_0 conda-forge tzdata 2021e he74cb21_0 conda-forge urllib3 1.26.7 pyhd8ed1ab_0 conda-forge wcwidth 0.2.5 pyh9f0ad1d_2 conda-forge wheel 0.37.0 pyhd8ed1ab_1 conda-forge xerces-c 3.2.3 h9d8b166_3 conda-forge xgboost 1.5.0 py39h06a4308_1
xorg-kbproto 1.0.7 h7f98852_1002 conda-forge xorg-libice 1.0.10 h7f98852_0 conda-forge xorg-libsm 1.2.3 hd9c2040_1000 conda-forge xorg-libx11 1.7.2 h7f98852_0 conda-forge xorg-libxau 1.0.9 h7f98852_0 conda-forge xorg-libxdmcp 1.1.3 h7f98852_0 conda-forge xorg-libxext 1.3.4 h7f98852_1 conda-forge xorg-libxrender 0.9.10 h7f98852_1003 conda-forge xorg-renderproto 0.11.1 h7f98852_1002 conda-forge xorg-xextproto 7.3.0 h7f98852_1002 conda-forge xorg-xproto 7.0.31 h7f98852_1007 conda-forge xz 5.2.5 h516909a_1 conda-forge yaml 0.2.5 h516909a_0 conda-forge zeromq 4.3.4 h9c3ff4c_1 conda-forge zict 2.0.0 py_0 conda-forge zipp 3.6.0 pyhd8ed1ab_0 conda-forge zlib 1.2.11 h36c2ea0_1013 conda-forge zstd 1.5.0 ha95c52a_0 conda-forge

[jakirkham]

Does the same thing happen when using LocalCluster from distributed?

[jdwillard19]

Yes, exact same error using dask.distributed.LocalCluster()