[Bug] Can't set temp_dir, get permission errors

Search before asking

[X] I searched the issues and found no similar issues.

Ray Component

Ray Tune

What happened + What you expected to happen

When I try to connect a worker node to a ray cluster started on a head node (using the steps provided in this documentation) I get the following error,

PermissionError: [Errno 13] Permission denied: '/tmp/ray'

I assume I am supposed to supply a temporary directory that I actually have permission in to the init function to solve this issue like so,

ray.init(address='128.55.68.181:6379', _redis_password='5241590000000000',_temp_dir='~tmp/ray',local_mode=True)

However this does not fix the issue. Stack trace below

2022-02-20 23:16:03,858 INFO worker.py:852 -- Connecting to existing Ray cluster at address: 128.55.68.181:6379 Traceback (most recent call last): File "", line 1, in File "/global/homes/j/jwillard/miniconda3/envs/stml/lib/python3.9/site-packages/ray/_private/client_mode_hook.py", line 105, in wrapper return func(*args, **kwargs) File "/global/homes/j/jwillard/miniconda3/envs/stml/lib/python3.9/site-packages/ray/worker.py", line 954, in init _global_node = ray.node.Node( File "/global/homes/j/jwillard/miniconda3/envs/stml/lib/python3.9/site-packages/ray/node.py", line 151, in init self._init_temp() File "/global/homes/j/jwillard/miniconda3/envs/stml/lib/python3.9/site-packages/ray/node.py", line 280, in _init_temp try_to_create_directory(self._temp_dir) File "/global/homes/j/jwillard/miniconda3/envs/stml/lib/python3.9/site-packages/ray/_private/utils.py", line 835, in try_to_create_directory os.makedirs(directory_path, exist_ok=True) File "/global/homes/j/jwillard/miniconda3/envs/stml/lib/python3.9/os.py", line 225, in makedirs mkdir(name, mode) PermissionError: [Errno 13] Permission denied: '/tmp/ray'

Versions / Dependencies

I am on SUSE Linux Enterprise Server 15 SP2, and my conda environment looks as follows

_libgcc_mutex 0.1 conda_forge conda-forge _openmp_mutex 4.5 1_gnu conda-forge _py-xgboost-mutex 2.0 cpu_0
abseil-cpp 20210324.2 h9c3ff4c_0 conda-forge affine 2.3.0 py_0 anaconda alembic 1.7.4 pyhd8ed1ab_0 conda-forge alsa-lib 1.2.3 h516909a_0 conda-forge argon2-cffi 21.3.0 pyhd8ed1ab_0 conda-forge argon2-cffi-bindings 21.2.0 py39h3811e60_1 conda-forge arrow-cpp 5.0.0 py39h247a7eb_8_cpu conda-forge attrs 21.2.0 pyhd8ed1ab_0 conda-forge autopage 0.4.0 pyhd8ed1ab_0 conda-forge aws-c-cal 0.5.11 h95a6274_0 conda-forge aws-c-common 0.6.2 h7f98852_0 conda-forge aws-c-event-stream 0.2.7 h3541f99_13 conda-forge aws-c-io 0.10.5 hfb6a706_0 conda-forge aws-checksums 0.1.11 ha31a3da_7 conda-forge aws-sdk-cpp 1.8.186 hb4091e7_3 conda-forge backcall 0.2.0 pyh9f0ad1d_0 conda-forge backports 1.0 py_2 anaconda backports.functools_lru_cache 1.6.4 pyhd8ed1ab_0 conda-forge blas 1.0 mkl anaconda bleach 4.1.0 pyhd8ed1ab_0 conda-forge blosc 1.21.0 h9c3ff4c_0 conda-forge bokeh 2.4.2 py39hf3d152e_0 conda-forge boost-cpp 1.74.0 h312852a_4 conda-forge brotlipy 0.7.0 py39h3811e60_1001 conda-forge bzip2 1.0.8 h7f98852_4 conda-forge c-ares 1.18.1 h7f98852_0 conda-forge ca-certificates 2021.10.8 ha878542_0 conda-forge cached-property 1.5.2 hd8ed1ab_1 conda-forge cached_property 1.5.2 pyha770c72_1 conda-forge cairo 1.16.0 h6cf1ce9_1008 conda-forge certifi 2021.10.8 py39hf3d152e_1 conda-forge cffi 1.14.6 py39h4bc2ebd_1 conda-forge cfitsio 4.0.0 h9a35b8e_0 conda-forge chardet 4.0.0 py39hf3d152e_1 conda-forge charset-normalizer 2.0.0 pyhd8ed1ab_0 conda-forge click 8.0.3 py39hf3d152e_1 conda-forge click-plugins 1.1.1 py_0 anaconda cliff 3.9.0 pyhd8ed1ab_0 conda-forge cligj 0.7.2 pyhd8ed1ab_1 conda-forge cloudpickle 2.0.0 pyhd8ed1ab_0 conda-forge cmaes 0.8.2 pyh44b312d_0 conda-forge cmd2 2.2.0 py39hf3d152e_0 conda-forge colorama 0.4.4 pyh9f0ad1d_0 conda-forge colorlog 6.5.0 py39hf3d152e_0 conda-forge cryptography 35.0.0 py39h95dcef6_1 conda-forge cuda-python 11.6.0 py39h3fd9d12_0 nvidia cudatoolkit 11.3.1 ha36c431_9 nvidia curl 7.80.0 h2574ce0_0 conda-forge cycler 0.10.0 py_2 conda-forge cython 0.29.25 py39hdbfa776_0
cytoolz 0.11.2 py39h3811e60_1 conda-forge dask 2021.9.1 pyhd8ed1ab_0 conda-forge dask-core 2021.9.1 pyhd8ed1ab_0 conda-forge dask-cuda 21.10.0 pyhd8ed1ab_0 conda-forge dask-jobqueue 0.7.3 pyhd8ed1ab_0 conda-forge dask-mpi 2021.11.0 py39hdc36942_0 conda-forge dask-optuna 0.0.2 pypi_0 pypi dbus 1.13.6 h48d8840_2 conda-forge debugpy 1.5.1 py39he80948d_0 conda-forge decorator 5.1.0 pyhd8ed1ab_0 conda-forge defusedxml 0.7.1 pyhd8ed1ab_0 conda-forge deprecated 1.2.13 pypi_0 pypi distributed 2021.9.1 py39hf3d152e_0 conda-forge entrypoints 0.3 pyhd8ed1ab_1003 conda-forge expat 2.4.1 h9c3ff4c_0 conda-forge ffmpeg 4.3 hf484d3e_0 pytorch filelock 3.6.0 pypi_0 pypi flit-core 3.6.0 pyhd8ed1ab_0 conda-forge font-ttf-dejavu-sans-mono 2.37 hab24e00_0 conda-forge font-ttf-inconsolata 3.000 h77eed37_0 conda-forge font-ttf-source-code-pro 2.038 h77eed37_0 conda-forge font-ttf-ubuntu 0.83 hab24e00_0 conda-forge fontconfig 2.13.1 hba837de_1005 conda-forge fonts-conda-ecosystem 1 0 conda-forge fonts-conda-forge 1 0 conda-forge freetype 2.10.4 h0708190_1 conda-forge freexl 1.0.6 h7f98852_0 conda-forge fsspec 2022.1.0 pyhd8ed1ab_0 conda-forge geos 3.10.0 h9c3ff4c_0 conda-forge geotiff 1.7.0 hcfb7246_3 conda-forge gettext 0.19.8.1 h73d1719_1008 conda-forge gflags 2.2.2 he1b5a44_1004 conda-forge giflib 5.2.1 h36c2ea0_2 conda-forge glib 2.70.0 h780b84a_1 conda-forge glib-tools 2.70.0 h780b84a_1 conda-forge glog 0.5.0 h48cff8f_0 conda-forge gmp 6.2.1 h58526e2_0 conda-forge gnutls 3.6.13 h85f3911_1 conda-forge greenlet 1.1.2 py39he80948d_0 conda-forge grpc-cpp 1.40.0 h05f19cf_2 conda-forge grpcio 1.44.0 pypi_0 pypi gst-plugins-base 1.18.5 hf529b03_0 conda-forge gstreamer 1.18.5 h76c114f_0 conda-forge h5py 3.4.0 nompi_py39h7e08c79_101 conda-forge hdf4 4.2.15 h10796ff_3 conda-forge hdf5 1.12.1 nompi_h2750804_101 conda-forge heapdict 1.0.1 py_0 conda-forge icu 68.2 h9c3ff4c_0 conda-forge idna 3.1 pyhd3deb0d_0 conda-forge importlib-metadata 4.8.1 py39hf3d152e_0 conda-forge importlib_resources 5.3.0 pyhd8ed1ab_0 conda-forge intel-openmp 2021.3.0 h06a4308_3350
ipykernel 6.4.2 py39hef51801_0 conda-forge ipython 7.29.0 py39hef51801_1 conda-forge ipython_genutils 0.2.0 py_1 conda-forge jbig 2.1 h7f98852_2003 conda-forge jedi 0.18.0 py39hf3d152e_3 conda-forge jinja2 3.0.3 pyhd8ed1ab_0 conda-forge joblib 0.17.0 py_0 anaconda jpeg 9d h36c2ea0_0 conda-forge json-c 0.15 h98cffda_0 conda-forge jsonschema 4.4.0 pyhd8ed1ab_0 conda-forge jupyter_client 7.0.6 pyhd8ed1ab_0 conda-forge jupyter_contrib_core 0.3.3 py_2 conda-forge jupyter_contrib_nbextensions 0.5.1 pyhd8ed1ab_2 conda-forge jupyter_core 4.9.1 py39hf3d152e_0 conda-forge jupyter_highlight_selected_word 0.2.0 py39hf3d152e_1005 conda-forge jupyter_latex_envs 1.4.6 pyhd8ed1ab_1002 conda-forge jupyter_nbextensions_configurator 0.4.1 py39hf3d152e_2 conda-forge jupyterlab_pygments 0.1.2 pyh9f0ad1d_0 conda-forge kealib 1.4.14 h87e4c3c_3 conda-forge kiwisolver 1.3.2 py39h1a9c180_0 conda-forge krb5 1.19.2 hcc1bbae_2 conda-forge lame 3.100 h7f98852_1001 conda-forge lcms2 2.12 hddcbb42_0 conda-forge ld_impl_linux-64 2.36.1 hea4e1c9_2 conda-forge lerc 3.0 h9c3ff4c_0 conda-forge libblas 3.9.0 11_linux64_mkl conda-forge libbrotlicommon 1.0.9 h7f98852_5 conda-forge libbrotlidec 1.0.9 h7f98852_5 conda-forge libbrotlienc 1.0.9 h7f98852_5 conda-forge libcblas 3.9.0 11_linux64_mkl conda-forge libclang 11.1.0 default_ha53f305_1 conda-forge libcurl 7.80.0 h2574ce0_0 conda-forge libdap4 3.20.6 hd7c4107_2 conda-forge libdeflate 1.8 h7f98852_0 conda-forge libedit 3.1.20191231 he28a2e2_2 conda-forge libev 4.33 h516909a_1 conda-forge libevent 2.1.10 h9b69904_4 conda-forge libffi 3.4.2 h9c3ff4c_4 conda-forge libgcc-ng 11.2.0 h1d223b6_11 conda-forge libgdal 3.3.3 h18e3bf0_2 conda-forge libgfortran-ng 11.2.0 h69a702a_11 conda-forge libgfortran5 11.2.0 h5c6108e_11 conda-forge libglib 2.70.0 h174f98d_1 conda-forge libgomp 11.2.0 h1d223b6_11 conda-forge libhwloc 2.3.0 h5e5b7d1_1 conda-forge libiconv 1.16 h516909a_0 conda-forge libkml 1.3.0 h238a007_1014 conda-forge liblapack 3.9.0 11_linux64_mkl conda-forge libllvm11 11.1.0 hf817b99_2 conda-forge libnetcdf 4.8.1 nompi_hb3fd0d9_101 conda-forge libnghttp2 1.43.0 h812cca2_1 conda-forge libnsl 2.0.0 h7f98852_0 conda-forge libogg 1.3.4 h7f98852_1 conda-forge libopus 1.3.1 h7f98852_1 conda-forge libpng 1.6.37 h21135ba_2 conda-forge libpq 13.5 hd57d9b9_1 conda-forge libprotobuf 3.18.1 h780b84a_0 conda-forge librmm 22.02.00 hbecc9b5_0 conda-forge librttopo 1.1.0 h0ad649c_7 conda-forge libsodium 1.0.18 h36c2ea0_1 conda-forge libspatialite 5.0.1 h1d9e4f1_10 conda-forge libssh2 1.10.0 ha56f1ee_2 conda-forge libstdcxx-ng 11.2.0 he4da1e4_11 conda-forge libthrift 0.15.0 he6d91bd_1 conda-forge libtiff 4.3.0 h6f004c6_2 conda-forge libutf8proc 2.6.1 h7f98852_0 conda-forge libuuid 2.32.1 h7f98852_1000 conda-forge libuv 1.42.0 h7f98852_0 conda-forge libvorbis 1.3.7 h9c3ff4c_0 conda-forge libwebp-base 1.2.1 h7f98852_0 conda-forge libxcb 1.13 h7f98852_1003 conda-forge libxgboost 1.5.0 h295c915_1
libxkbcommon 1.0.3 he3ba5ed_0 conda-forge libxml2 2.9.12 h72842e0_0 conda-forge libxslt 1.1.33 h15afd5d_2 conda-forge libzip 1.8.0 h4de3113_1 conda-forge libzlib 1.2.11 h36c2ea0_1013 conda-forge llvmlite 0.37.0 py39h1bbdace_1 conda-forge locket 0.2.0 py_2 conda-forge lxml 4.8.0 py39h107f48f_0 conda-forge lz4-c 1.9.3 h9c3ff4c_1 conda-forge lzo 2.10 h516909a_1000 conda-forge mako 1.1.5 pyhd8ed1ab_0 conda-forge markupsafe 2.0.1 py39h3811e60_0 conda-forge matplotlib 3.4.3 py39hf3d152e_1 conda-forge matplotlib-base 3.4.3 py39h2fa2bec_1 conda-forge matplotlib-inline 0.1.3 pyhd8ed1ab_0 conda-forge mistune 0.8.4 py39h3811e60_1005 conda-forge mkl 2021.3.0 h06a4308_520
mkl-service 2.4.0 py39h3811e60_0 conda-forge mkl_fft 1.3.1 py39hd3c417c_0
mkl_random 1.2.2 py39hde0f152_0 conda-forge mock 4.0.3 py39hf3d152e_1 conda-forge mpi 1.0 openmpi conda-forge mpi4py 3.1.3 py39hc649577_0 conda-forge msgpack-python 1.0.3 py39h1a9c180_0 conda-forge mysql-common 8.0.27 ha770c72_0 conda-forge mysql-libs 8.0.27 hfa10184_0 conda-forge nbclient 0.5.11 pyhd8ed1ab_0 conda-forge nbconvert 6.4.2 py39hf3d152e_0 conda-forge nbformat 5.1.3 pyhd8ed1ab_0 conda-forge ncurses 6.2 h58526e2_4 conda-forge nest-asyncio 1.5.1 pyhd8ed1ab_0 conda-forge nettle 3.6 he412f7d_0 conda-forge notebook 6.4.8 pyha770c72_0 conda-forge nspr 4.30 h9c3ff4c_0 conda-forge nss 3.69 hb5efdd6_1 conda-forge numba 0.54.1 py39h56b8d98_0 conda-forge numexpr 2.7.3 py39hde0f152_0 conda-forge numpy 1.20.3 py39hdbf815f_1 conda-forge olefile 0.46 pyh9f0ad1d_1 conda-forge openh264 2.1.1 h780b84a_0 conda-forge openjpeg 2.4.0 hb52868f_1 conda-forge openmpi 4.1.0 hbfc84c5_4 conda-forge openssl 1.1.1l h7f98852_0 conda-forge optuna 2.1.0 py_0 conda-forge orc 1.7.0 h68e2c4e_0 conda-forge packaging 21.0 pyhd8ed1ab_0 conda-forge pandas 1.4.0 py39hde0f152_0 conda-forge pandoc 2.17.1.1 ha770c72_0 conda-forge pandocfilters 1.5.0 pyhd8ed1ab_0 conda-forge parquet-cpp 1.5.1 2 conda-forge parso 0.8.2 pyhd8ed1ab_0 conda-forge partd 1.2.0 pyhd8ed1ab_0 conda-forge patsy 0.5.2 pyhd8ed1ab_0 conda-forge pbr 5.6.0 pyhd8ed1ab_0 conda-forge pcre 8.45 h9c3ff4c_0 conda-forge pexpect 4.8.0 pyh9f0ad1d_2 conda-forge pickleshare 0.7.5 py_1003 conda-forge pillow 8.3.2 py39ha612740_0 conda-forge pip 21.3.1 pyhd8ed1ab_0 conda-forge pixman 0.40.0 h36c2ea0_0 conda-forge poppler 21.09.0 ha39eefc_3 conda-forge poppler-data 0.4.11 hd8ed1ab_0 conda-forge postgresql 13.5 h2510834_1 conda-forge prettytable 2.2.1 pyhd8ed1ab_0 conda-forge proj 8.1.1 h277dcde_2 conda-forge prometheus_client 0.13.1 pyhd8ed1ab_0 conda-forge prompt-toolkit 3.0.22 pyha770c72_0 conda-forge protobuf 3.19.4 pypi_0 pypi psutil 5.9.0 py39h3811e60_0 conda-forge pthread-stubs 0.4 h36c2ea0_1001 conda-forge ptyprocess 0.7.0 pyhd3deb0d_0 conda-forge py-xgboost 1.5.0 py39h06a4308_1
pyarrow 5.0.0 py39h3ebc44c_8_cpu conda-forge pycparser 2.20 pyh9f0ad1d_2 conda-forge pygments 2.10.0 pyhd8ed1ab_0 conda-forge pynvml 11.4.1 pyhd8ed1ab_0 conda-forge pyopenssl 21.0.0 pyhd8ed1ab_0 conda-forge pyparsing 3.0.3 pyhd8ed1ab_0 conda-forge pyperclip 1.8.2 pyhd8ed1ab_2 conda-forge pyqt 5.12.3 py39hf3d152e_7 conda-forge pyqt-impl 5.12.3 py39h0fcd23e_7 conda-forge pyqt5-sip 4.19.18 py39he80948d_7 conda-forge pyqtchart 5.12 py39h0fcd23e_7 conda-forge pyqtwebengine 5.12.1 py39h0fcd23e_7 conda-forge pyrsistent 0.18.1 py39h3811e60_0 conda-forge pysocks 1.7.1 py39hf3d152e_3 conda-forge pytables 3.6.1 py39h2669a42_4 conda-forge python 3.9.7 hb7a2778_3_cpython conda-forge python-dateutil 2.8.2 pyhd8ed1ab_0 conda-forge python_abi 3.9 2_cp39 conda-forge pytorch 1.10.0 py3.9_cuda11.3_cudnn8.2.0_0 pytorch pytorch-mutex 1.0 cuda pytorch pytz 2021.3 pyhd8ed1ab_0 conda-forge pyyaml 6.0 py39h3811e60_3 conda-forge pyzmq 22.3.0 py39h37b5a0c_1 conda-forge qt 5.12.9 hda022c4_4 conda-forge rasterio 1.2.10 py39hb37810a_0 conda-forge ray 1.10.0 pypi_0 pypi re2 2021.09.01 h9c3ff4c_0 conda-forge readline 8.1 h46c0cb4_0 conda-forge redis 4.1.4 pypi_0 pypi requests 2.26.0 pyhd8ed1ab_0 conda-forge rmm 22.02.00 cuda11_py39_ge3e3215_0_has_cma rapidsai s2n 1.0.10 h9b69904_0 conda-forge scikit-learn 1.0.1 py39h7c5d8c9_1 conda-forge scipy 1.7.1 py39hee8e79c_0 conda-forge seaborn 0.11.0 py_0 anaconda send2trash 1.8.0 pyhd8ed1ab_0 conda-forge setuptools 58.2.0 py39hf3d152e_0 conda-forge shap 0.40.0 py39hde0f152_0 conda-forge six 1.16.0 pyh6c4a22f_0 conda-forge slicer 0.0.7 pyhd8ed1ab_0 conda-forge snappy 1.1.8 he1b5a44_3 conda-forge snuggs 1.4.7 py_0 anaconda sortedcontainers 2.4.0 pyhd8ed1ab_0 conda-forge spdlog 1.9.2 h4bd325d_1 conda-forge sqlalchemy 1.4.26 py39h3811e60_0 conda-forge sqlite 3.36.0 h9cd32fc_2 conda-forge statsmodels 0.13.0 py39hce5d2b2_0 conda-forge stevedore 3.5.0 py39hf3d152e_0 conda-forge tabulate 0.8.9 pypi_0 pypi tblib 1.7.0 pyhd8ed1ab_0 conda-forge tensorboardx 2.4.1 pypi_0 pypi terminado 0.13.1 py39hf3d152e_0 conda-forge testpath 0.5.0 pyhd8ed1ab_0 conda-forge threadpoolctl 2.1.0 pyh5ca1d4c_0 anaconda thrust 1.11.0 ha770c72_1 conda-forge tiledb 2.3.4 he87e0bf_0 conda-forge tk 8.6.11 h27826a3_1 conda-forge toolz 0.11.2 pyhd8ed1ab_0 conda-forge torchaudio 0.10.0 py39_cu113 pytorch torchvision 0.11.1 py39_cu113 pytorch tornado 6.1 py39h3811e60_1 conda-forge tqdm 4.62.3 pyhd8ed1ab_0 conda-forge traitlets 5.1.1 pyhd8ed1ab_0 conda-forge typing_extensions 3.10.0.2 pyha770c72_0 conda-forge tzcode 2021e h7f98852_0 conda-forge tzdata 2021e he74cb21_0 conda-forge ucx 1.12.0+gd367332 cuda11.2_0 rapidsai ucx-proc 1.0.0 gpu conda-forge ucx-py 0.24.0 py39_gd367332_0 rapidsai urllib3 1.26.7 pyhd8ed1ab_0 conda-forge wcwidth 0.2.5 pyh9f0ad1d_2 conda-forge webencodings 0.5.1 py_1 conda-forge wheel 0.37.0 pyhd8ed1ab_1 conda-forge wrapt 1.13.3 pypi_0 pypi xerces-c 3.2.3 h9d8b166_3 conda-forge xgboost 1.5.0 py39h06a4308_1
xorg-kbproto 1.0.7 h7f98852_1002 conda-forge xorg-libice 1.0.10 h7f98852_0 conda-forge xorg-libsm 1.2.3 hd9c2040_1000 conda-forge xorg-libx11 1.7.2 h7f98852_0 conda-forge xorg-libxau 1.0.9 h7f98852_0 conda-forge xorg-libxdmcp 1.1.3 h7f98852_0 conda-forge xorg-libxext 1.3.4 h7f98852_1 conda-forge xorg-libxrender 0.9.10 h7f98852_1003 conda-forge xorg-renderproto 0.11.1 h7f98852_1002 conda-forge xorg-xextproto 7.3.0 h7f98852_1002 conda-forge xorg-xproto 7.0.31 h7f98852_1007 conda-forge xz 5.2.5 h516909a_1 conda-forge yaml 0.2.5 h516909a_0 conda-forge zeromq 4.3.4 h9c3ff4c_1 conda-forge zict 2.0.0 py_0 conda-forge zipp 3.6.0 pyhd8ed1ab_0 conda-forge zlib 1.2.11 h36c2ea0_1013 conda-forge zstd 1.5.0 ha95c52a_0 conda-forge

Reproduction script

On host node,

ray start --head --port=6379

On worker node, ray start --address='128.55.68.181:6379' --redis-password='5241590000000000' --temp-dir='~tmp/ray'

Anything else

Always

Are you willing to submit a PR?

[ ] Yes I am willing to submit a PR!

ray-project / ray