[Core] ray workers ignore `OMP_NUM_THREADS` env variable

[jmakov]

What happened + What you expected to happen

A library is using all the cores on a node (multithreading). The idea is to distribute the work by using ray.remote(num_cpus=_max_cpus_on_a_node_). Running the code without ray, all cores are used. Running with ray even the driver node uses only 1 core/thread. Do env vars need to be set somewhere else than before worker start command?

This works (uses all cores) but warnings are reported (since we're using an empty string instead of an int). Is there a better way?

ray.init(
            runtime_env={
                "working_dir": workdir,
                "excludes": ["*.txt", "bin", "doc", "examples", "notebooks"],
                "env_vars": {"MKL_NUM_THREADS": "", "OPENBLAS_NUM_THREADS": "", "OMP_NUM_THREADS": ""}
            },
            address='auto',
            log_to_driver=True,
            _redis_password='xxx'
        )

Versions / Dependencies

env.yaml:

name: puma-lab
channels:
  - pyviz
  - intel
  - conda-forge
  - defaults
dependencies:
  - _libgcc_mutex=0.1=conda_forge
  - _openmp_mutex=4.5=2_gnu
  - abseil-cpp=20211102.0=h27087fc_1
  - aiohttp=3.8.1=py38h0a891b7_1
  - aiohttp-cors=0.7.0=py_0
  - aiohttp-retry=2.4.6=pyhd8ed1ab_1
  - aioredis=1.3.1=py_0
  - aiorwlock=1.3.0=pyhd8ed1ab_0
  - aiosignal=1.2.0=pyhd8ed1ab_0
  - alsa-lib=1.2.3.2=h166bdaf_0
  - anyio=3.6.1=py38h578d9bd_0
  - aom=3.3.0=h27087fc_1
  - aplus=0.11.0=py_1
  - appdirs=1.4.4=pyh9f0ad1d_0
  - argon2-cffi=21.3.0=pyhd8ed1ab_0
  - argon2-cffi-bindings=21.2.0=py38h0a891b7_2
  - arrow-cpp=6.0.1=py38hd79a172_19_cpu
  - asgiref=3.5.2=pyhd8ed1ab_0
  - astropy=5.1=py38h71d37f0_0
  - asttokens=2.0.5=pyhd8ed1ab_0
  - async-timeout=4.0.2=pyhd8ed1ab_0
  - asyncssh=2.10.1=pyhd8ed1ab_0
  - atk-1.0=2.36.0=h3371d22_4
  - atpublic=3.0.1=pyhd8ed1ab_0
  - attrs=21.4.0=pyhd8ed1ab_0
  - aws-c-cal=0.5.11=h95a6274_0
  - aws-c-common=0.6.2=h7f98852_0
  - aws-c-event-stream=0.2.7=h3541f99_13
  - aws-c-io=0.10.5=hfb6a706_0
  - aws-checksums=0.1.11=ha31a3da_7
  - aws-sdk-cpp=1.8.186=hb4091e7_3
  - babel=2.10.1=pyhd8ed1ab_0
  - backcall=0.2.0=pyh9f0ad1d_0
  - backports=1.0=py_2
  - backports.functools_lru_cache=1.6.4=pyhd8ed1ab_0
  - backports.zoneinfo=0.2.1=py38h497a2fe_4
  - beautifulsoup4=4.11.1=pyha770c72_0
  - blake3=0.2.1=py38ha8db356_0
  - bleach=5.0.0=pyhd8ed1ab_0
  - blessings=1.7=py38h578d9bd_1006
  - blosc=1.21.1=h83bc5f7_3
  - bokeh=2.4.3=py38h578d9bd_0
  - bqplot=0.12.33=pyhd8ed1ab_0
  - brotli=1.0.9=h166bdaf_7
  - brotli-bin=1.0.9=h166bdaf_7
  - brotlipy=0.7.0=py38h0a891b7_1004
  - brunsli=0.1=h9c3ff4c_0
  - bzip2=1.0.8=hb9a14ef_9
  - c-ares=1.18.1=h7f98852_0
  - c-blosc2=2.1.1=h7a311fb_2
  - ca-certificates=2022.5.18.1=ha878542_0
  - cached-property=1.5.2=hd8ed1ab_1
  - cached_property=1.5.2=pyha770c72_1
  - cachetools=5.0.0=pyhd8ed1ab_0
  - cairo=1.16.0=h6cf1ce9_1008
  - certifi=2022.5.18.1=py38h578d9bd_0
  - cffi=1.14.6=py38ha65f79e_0
  - cfitsio=4.1.0=hd9d235c_0
  - charls=2.3.4=h9c3ff4c_0
  - charset-normalizer=2.0.12=pyhd8ed1ab_0
  - click=8.1.3=py38h578d9bd_0
  - clickhouse-cityhash=1.0.2.3=py38h709712a_3
  - clickhouse-driver=0.2.3=py38h497a2fe_0
  - cloudpickle=2.1.0=pyhd8ed1ab_0
  - colorama=0.4.4=pyh9f0ad1d_0
  - colorcet=3.0.0=pyhd8ed1ab_0
  - colorful=0.5.4=pyhd8ed1ab_0
  - commonmark=0.9.1=py_0
  - configobj=5.0.6=py_0
  - cramjam=2.5.0=py38ha8db356_0
  - cryptography=37.0.2=py38h2b5fc30_0
  - cycler=0.11.0=pyhd8ed1ab_0
  - cytoolz=0.11.2=py38h0a891b7_2
  - dask=2022.5.2=pyhd8ed1ab_0
  - dask-core=2022.5.2=pyhd8ed1ab_0
  - dataclasses=0.8=pyhc8e2a94_3
  - datashader=0.14.0=pyh6c4a22f_0
  - datashape=0.5.4=py_1
  - dbus=1.13.6=h48d8840_2
  - debugpy=1.6.0=py38hfa26641_0
  - decorator=5.1.1=pyhd8ed1ab_0
  - defusedxml=0.7.1=pyhd8ed1ab_0
  - dictdiffer=0.9.0=pyhd8ed1ab_0
  - diskcache=5.4.0=pyhd8ed1ab_0
  - distlib=0.3.4=pyhd8ed1ab_0
  - distributed=2022.5.2=pyhd8ed1ab_0
  - distro=1.6.0=pyhd8ed1ab_0
  - dm-tree=0.1.7=py38h47df419_0
  - dpath=2.0.6=py38h578d9bd_1
  - dpcpp-cpp-rt=2022.1.0=intel_3768
  - dpcpp_cpp_rt=2022.1.0=intel_3768
  - dulwich=0.20.42=py38h0a891b7_0
  - dvc=2.10.2=py38h578d9bd_0
  - dvc-render=0.0.5=pyhd8ed1ab_0
  - dvc-ssh=2.10.2=py38h578d9bd_0
  - dvclive=0.8.2=pyhd8ed1ab_0
  - entrypoints=0.4=pyhd8ed1ab_0
  - executing=0.8.3=pyhd8ed1ab_0
  - expat=2.4.8=h27087fc_0
  - fastapi=0.78.0=pyhd8ed1ab_0
  - fastparquet=0.8.1=py38h71d37f0_0
  - filelock=3.7.1=pyhd8ed1ab_0
  - flatten-dict=0.4.2=pyhd8ed1ab_1
  - flit-core=3.7.1=pyhd8ed1ab_0
  - flufl.lock=7.0=pyhd8ed1ab_0
  - font-ttf-dejavu-sans-mono=2.37=hab24e00_0
  - font-ttf-inconsolata=3.000=h77eed37_0
  - font-ttf-source-code-pro=2.038=h77eed37_0
  - font-ttf-ubuntu=0.83=hab24e00_0
  - fontconfig=2.14.0=h8e229c2_0
  - fonts-conda-ecosystem=1=0
  - fonts-conda-forge=1=0
  - fonttools=4.33.3=py38h0a891b7_0
  - freetype=2.11.0=h70c0345_0
  - fribidi=1.0.10=h36c2ea0_0
  - frozendict=2.3.2=py38h0a891b7_0
  - frozenlist=1.3.0=py38h0a891b7_1
  - fsspec=2022.5.0=pyhd8ed1ab_0
  - ftfy=6.1.1=pyhd8ed1ab_0
  - funcy=1.17=pyhd8ed1ab_0
  - future=0.18.2=py38h578d9bd_5
  - gdk-pixbuf=2.42.6=h04a7f16_0
  - gettext=0.19.8.1=h0b5b191_1005
  - gflags=2.2.2=he1b5a44_1004
  - giflib=5.2.1=h36c2ea0_2
  - gitdb=4.0.9=pyhd8ed1ab_0
  - gitpython=3.1.27=pyhd8ed1ab_0
  - glib=2.68.4=h9c3ff4c_0
  - glib-tools=2.68.4=h9c3ff4c_0
  - glog=0.6.0=h6f12383_0
  - google-api-core=2.8.1=pyhd8ed1ab_0
  - google-auth=2.6.6=pyh6c4a22f_0
  - googleapis-common-protos=1.56.2=py38h578d9bd_0
  - gpustat=0.6.0=pyhd8ed1ab_1
  - grandalf=0.6=py_0
  - graphite2=1.3.13=h58526e2_1001
  - graphviz=2.49.1=h85b4f2f_0
  - grpc-cpp=1.46.3=hc275302_0
  - grpcio=1.43.0=py38hdd6454d_0
  - gst-plugins-base=1.18.5=hf529b03_0
  - gstreamer=1.18.5=h76c114f_0
  - gtk2=2.24.33=h539f30e_1
  - gts=0.7.6=h64030ff_2
  - gym=0.21.0=py38he5a9106_2
  - h11=0.13.0=pyhd8ed1ab_1
  - h5py=3.6.0=nompi_py38hfbb2109_100
  - harfbuzz=3.0.0=h83ec7ef_1
  - hdf5=1.12.1=nompi_h2386368_104
  - heapdict=1.0.1=py_0
  - hiredis=2.0.0=py38h0a891b7_2
  - holoviews=1.14.9=py_0
  - hvplot=0.8.0=py_0
  - icc_rt=2022.1.0=intel_3768
  - icu=68.2=h9c3ff4c_0
  - idna=3.3=pyhd8ed1ab_0
  - imagecodecs=2022.2.22=py38h43af60f_5
  - imageio=2.13.1=pyhd8ed1ab_0
  - importlib-metadata=4.11.4=py38h578d9bd_0
  - importlib_metadata=4.11.4=hd8ed1ab_0
  - importlib_resources=5.7.1=pyhd8ed1ab_1
  - intel-cmplr-lib-rt=2022.1.0=intel_3768
  - intel-cmplr-lic-rt=2022.1.0=intel_3768
  - intel-opencl-rt=2022.1.0=intel_3768
  - intel-openmp=2022.1.0=intel_3768
  - intelpython=2022.1.0=0
  - ipydatawidgets=4.3.1.post1=pyhc268e32_0
  - ipykernel=6.13.0=py38h7f3c49e_0
  - ipyleaflet=0.16.0=pyhd8ed1ab_4
  - ipympl=0.9.1=pyhd8ed1ab_0
  - ipython=8.4.0=py38h578d9bd_0
  - ipython_genutils=0.2.0=py_1
  - ipyvolume=0.6.0a8=pyhd8ed1ab_0
  - ipyvue=1.7.0=pyhd8ed1ab_0
  - ipyvuetify=1.8.2=pyhd8ed1ab_0
  - ipywebrtc=0.6.0=pyhd8ed1ab_0
  - ipywidgets=7.7.0=pyhd8ed1ab_0
  - jedi=0.18.1=py38h578d9bd_1
  - jinja2=3.1.2=pyhd8ed1ab_0
  - jpeg=9e=h166bdaf_1
  - json5=0.9.5=pyh9f0ad1d_0
  - jsonschema=4.5.1=pyhd8ed1ab_0
  - jupyter-server-mathjax=0.2.5=pyhc268e32_0
  - jupyter_client=7.3.1=pyhd8ed1ab_0
  - jupyter_contrib_core=0.3.3=py_2
  - jupyter_contrib_nbextensions=0.5.1=pyhd8ed1ab_2
  - jupyter_core=4.10.0=py38h578d9bd_0
  - jupyter_highlight_selected_word=0.2.0=py38h578d9bd_1005
  - jupyter_latex_envs=1.4.6=pyhd8ed1ab_1002
  - jupyter_nbextensions_configurator=0.4.1=pyhd8ed1ab_2
  - jupyter_server=1.17.0=pyhd8ed1ab_0
  - jupyterlab=3.4.2=pyhd8ed1ab_0
  - jupyterlab-git=0.37.1=pyhd8ed1ab_0
  - jupyterlab_execute_time=2.1.0=pyhd8ed1ab_0
  - jupyterlab_pygments=0.2.2=pyhd8ed1ab_0
  - jupyterlab_server=2.14.0=pyhd8ed1ab_0
  - jupyterlab_widgets=1.1.0=pyhd8ed1ab_0
  - jxrlib=1.1=h7f98852_2
  - keyutils=1.6.1=h166bdaf_0
  - kiwisolver=1.4.2=py38h43d8883_1
  - krb5=1.19.3=h3790be6_0
  - kubernetes=1.23.6=ha770c72_0
  - kubernetes-client=1.23.6=h77c71de_0
  - kubernetes-node=1.23.6=h77c71de_0
  - kubernetes-server=1.23.6=h77c71de_0
  - lcms2=2.12=hddcbb42_0
  - lerc=3.0=h9c3ff4c_0
  - libaec=1.0.6=h9c3ff4c_0
  - libavif=0.10.1=h166bdaf_0
  - libblas=3.9.0=14_linux64_openblas
  - libbrotlicommon=1.0.9=h166bdaf_7
  - libbrotlidec=1.0.9=h166bdaf_7
  - libbrotlienc=1.0.9=h166bdaf_7
  - libcblas=3.9.0=14_linux64_openblas
  - libclang=11.1.0=default_ha53f305_1
  - libcurl=7.83.1=h7bff187_0
  - libdeflate=1.10=h7f98852_0
  - libedit=3.1.20191231=he28a2e2_2
  - libev=4.33=h516909a_1
  - libevent=2.1.10=h9b69904_4
  - libffi=3.3=14
  - libgcc-ng=12.1.0=h8d9b700_16
  - libgd=2.3.3=h6ad9fb6_0
  - libgfortran-ng=12.1.0=h69a702a_16
  - libgfortran5=12.1.0=hdcd56e2_16
  - libgit2=1.4.3=h6529ace_0
  - libglib=2.68.4=h3e27bee_0
  - libgomp=12.1.0=h8d9b700_16
  - libiconv=1.16=h516909a_0
  - liblapack=3.9.0=14_linux64_openblas
  - libllvm10=10.0.1=he513fc3_3
  - libllvm11=11.1.0=hf817b99_3
  - libnghttp2=1.47.0=h727a467_0
  - libogg=1.3.4=h7f98852_1
  - libopenblas=0.3.20=pthreads_h78a6416_0
  - libopus=1.3.1=h7f98852_1
  - libpng=1.6.37=h21135ba_2
  - libpq=13.5=hd57d9b9_1
  - libprotobuf=3.20.1=h6239696_0
  - librsvg=2.52.2=hc3c00ef_0
  - libsodium=1.0.18=h36c2ea0_1
  - libssh2=1.10.0=ha56f1ee_2
  - libstdcxx-ng=12.1.0=ha89aaad_16
  - libta-lib=0.4.0=h516909a_0
  - libthrift=0.16.0=h519c5ea_1
  - libtiff=4.3.0=h0fcbabc_4
  - libtool=2.4.6=h9c3ff4c_1008
  - libunwind=1.6.2=h9c3ff4c_0
  - libutf8proc=2.7.0=h7f98852_0
  - libuuid=2.32.1=h7f98852_1000
  - libuv=1.43.0=h7f98852_0
  - libvorbis=1.3.7=h9c3ff4c_0
  - libwebp=1.2.2=h3452ae3_0
  - libwebp-base=1.2.2=h7f98852_1
  - libxcb=1.13=h7f98852_1004
  - libxkbcommon=1.0.3=he3ba5ed_0
  - libxml2=2.9.12=h72842e0_0
  - libxslt=1.1.33=h15afd5d_2
  - libzlib=1.2.12=h166bdaf_0
  - libzopfli=1.0.3=h9c3ff4c_0
  - llvmlite=0.38.1=py38h38d86a4_0
  - locket=1.0.0=pyhd8ed1ab_0
  - lxml=4.8.0=py38h0a891b7_2
  - lz4=4.0.0=py38h1bf946c_2
  - lz4-c=1.9.3=h9c3ff4c_1
  - mailchecker=4.1.17=pyhd8ed1ab_0
  - markdown=3.3.7=pyhd8ed1ab_0
  - markupsafe=2.1.1=py38h0a891b7_1
  - matplotlib=3.5.1=py38h578d9bd_0
  - matplotlib-base=3.5.1=py38ha18d171_1
  - matplotlib-inline=0.1.3=pyhd8ed1ab_0
  - mistune=0.8.4=py38h497a2fe_1005
  - mkl=2022.1.0=intel_223
  - mkl-service=2.4.0=py38h7987b76_11
  - mkl_fft=1.3.1=py38h542d01b_16
  - mkl_random=1.2.2=py38hdd7f59e_16
  - mkl_umath=0.1.1=py38h3242e30_26
  - msgpack-python=1.0.3=py38h43d8883_1
  - multidict=6.0.2=py38h0a891b7_1
  - multipledispatch=0.6.0=py_0
  - munkres=1.1.4=pyh9f0ad1d_0
  - mysql-common=8.0.29=haf5c9bc_1
  - mysql-libs=8.0.29=h28c427c_1
  - nanotime=0.5.2=py_0
  - nb_conda_kernels=2.3.1=py38h578d9bd_1
  - nbclassic=0.3.7=pyhd8ed1ab_0
  - nbclient=0.6.4=pyhd8ed1ab_1
  - nbconvert=6.5.0=pyhd8ed1ab_0
  - nbconvert-core=6.5.0=pyhd8ed1ab_0
  - nbconvert-pandoc=6.5.0=pyhd8ed1ab_0
  - nbdime=3.1.1=pyhd8ed1ab_0
  - nbformat=5.4.0=pyhd8ed1ab_0
  - ncurses=6.3=h7f8727e_2
  - nest-asyncio=1.5.5=pyhd8ed1ab_0
  - networkx=2.8.2=pyhd8ed1ab_0
  - nodejs=14.18.3=h92b4a50_1
  - notebook=6.4.11=pyha770c72_0
  - notebook-shim=0.1.0=pyhd8ed1ab_0
  - nspr=4.32=h9c3ff4c_1
  - nss=3.77=h2350873_0
  - numba=0.55.1=py38hefe1858_1
  - numpy=1.21.4=py38ha3f613e_16
  - numpy-base=1.21.4=py38h1b56c09_16
  - nvidia-ml=7.352.0=py_0
  - olefile=0.46=pyh9f0ad1d_1
  - opencensus=0.9.0=pyhd8ed1ab_0
  - opencensus-context=0.1.2=py38h578d9bd_5
  - openjpeg=2.4.0=hb52868f_1
  - openssl=1.1.1o=h166bdaf_0
  - orc=1.7.4=h6c59b99_1
  - packaging=21.3=pyhd8ed1ab_0
  - pandas=1.4.2=py38h47df419_2
  - pandoc=2.18=ha770c72_0
  - pandocfilters=1.5.0=pyhd8ed1ab_0
  - panel=0.13.1=py_0
  - pango=1.48.10=h54213e6_2
  - param=1.12.1=pyh6c4a22f_0
  - parquet-cpp=1.5.1=2
  - parso=0.8.3=pyhd8ed1ab_0
  - partd=1.2.0=pyhd8ed1ab_0
  - pathlib2=2.3.7.post1=py38h578d9bd_1
  - pathspec=0.9.0=pyhd8ed1ab_0
  - patsy=0.5.2=pyhd8ed1ab_0
  - pcre=8.45=h9c3ff4c_0
  - pcre2=10.37=h032f7d1_0
  - pexpect=4.8.0=pyh9f0ad1d_2
  - phonenumbers=8.12.49=pyhd8ed1ab_0
  - pickleshare=0.7.5=py_1003
  - pillow=8.3.1=py38h8e6f84c_0
  - pip=21.2.4=py38h06a4308_0
  - pixman=0.40.0=h36c2ea0_0
  - platformdirs=2.5.1=pyhd8ed1ab_0
  - ply=3.11=py_1
  - pooch=1.4.0=pyhd8ed1ab_0
  - progressbar2=4.0.0=pyhd8ed1ab_0
  - prometheus_client=0.13.1=pyhd8ed1ab_0
  - prompt-toolkit=3.0.29=pyha770c72_0
  - protobuf=3.20.1=py38hfa26641_0
  - psutil=5.9.1=py38h0a891b7_0
  - pthread-stubs=0.4=h36c2ea0_1001
  - ptyprocess=0.7.0=pyhd3deb0d_0
  - pure_eval=0.2.2=pyhd8ed1ab_0
  - py-spy=0.3.11=h060cca7_1
  - pyarrow=6.0.1=py38h9f6a473_19_cpu
  - pyasn1=0.4.8=py_0
  - pyasn1-modules=0.2.7=py_0
  - pycparser=2.21=pyhd8ed1ab_0
  - pyct=0.4.6=py_0
  - pyct-core=0.4.6=py_0
  - pydantic=1.9.1=py38h0a891b7_0
  - pydot=1.4.2=py38h578d9bd_1
  - pyerfa=2.0.0.1=py38h71d37f0_2
  - pygit2=1.9.0=py38h0a891b7_1
  - pygments=2.12.0=pyhd8ed1ab_0
  - pygtrie=2.4.2=pyhd3deb0d_0
  - pykalman=0.9.5=py_1
  - pyopenssl=22.0.0=pyhd8ed1ab_0
  - pyparsing=2.4.7=pyhd8ed1ab_1
  - pyqt=5.12.3=py38h578d9bd_8
  - pyqt-impl=5.12.3=py38h0ffb2e6_8
  - pyqt5-sip=4.19.18=py38h709712a_8
  - pyqtchart=5.12=py38h7400c14_8
  - pyqtwebengine=5.12.1=py38h7400c14_8
  - pyrsistent=0.18.1=py38h0a891b7_1
  - pysocks=1.7.1=py38h578d9bd_5
  - python=3.8.12=h70b2ee3_6
  - python-benedict=0.25.1=pyhd8ed1ab_0
  - python-dateutil=2.8.2=pyhd8ed1ab_0
  - python-fastjsonschema=2.15.3=pyhd8ed1ab_0
  - python-fsutil=0.6.1=pyhd8ed1ab_0
  - python-gssapi=1.7.3=py38hf949b76_1
  - python-slugify=6.1.2=pyhd8ed1ab_0
  - python-tzdata=2022.1=pyhd8ed1ab_0
  - python-utils=3.3.3=pyhd8ed1ab_0
  - python_abi=3.8=2_cp38
  - pythreejs=2.3.0=pyhd8ed1ab_0
  - pytz=2022.1=pyhd8ed1ab_0
  - pytz-deprecation-shim=0.1.0.post0=py38h578d9bd_1
  - pyu2f=0.1.5=pyhd8ed1ab_0
  - pyviz_comms=2.2.0=py_0
  - pywavelets=1.3.0=py38h71d37f0_1
  - pyyaml=6.0=py38h0a891b7_4
  - pyzmq=23.0.0=py38hfc09fa9_0
  - qt=5.12.9=hda022c4_4
  - ray-all=1.12.1=py38h578d9bd_0
  - ray-core=1.12.1=py38hb9c8dfe_0
  - ray-dashboard=1.12.1=py38ha54862b_0
  - ray-data=1.12.1=py38h578d9bd_0
  - ray-default=1.12.1=py38h578d9bd_0
  - ray-k8s=1.12.1=py38h578d9bd_0
  - ray-rllib=1.12.1=py38h578d9bd_0
  - ray-serve=1.12.1=py38h578d9bd_0
  - ray-tune=1.12.1=py38h578d9bd_0
  - re2=2022.04.01=h27087fc_0
  - readline=8.1.2=h7f8727e_1
  - requests=2.27.1=pyhd8ed1ab_0
  - rich=12.4.4=pyhd8ed1ab_0
  - rsa=4.8=pyhd8ed1ab_0
  - ruamel.yaml=0.17.21=py38h0a891b7_1
  - ruamel.yaml.clib=0.2.6=py38h0a891b7_1
  - s2n=1.0.10=h9b69904_0
  - scikit-image=0.19.2=py38h43a58ef_0
  - scipy=1.8.1=py38h1ee437e_0
  - scmrepo=0.0.19=pyhd8ed1ab_0
  - send2trash=1.8.0=pyhd8ed1ab_0
  - setproctitle=1.2.2=py38h0a891b7_2
  - setuptools=58.0.4=py38h06a4308_0
  - shortuuid=1.0.8=py38h578d9bd_0
  - shtab=1.5.4=pyhd8ed1ab_0
  - six=1.16.0=pyh6c4a22f_0
  - smart_open=6.0.0=pyhd8ed1ab_0
  - smmap=3.0.5=pyh44b312d_0
  - snappy=1.1.9=hbd366e4_1
  - sniffio=1.2.0=py38h578d9bd_3
  - sortedcontainers=2.4.0=pyhd8ed1ab_0
  - soupsieve=2.3.1=pyhd8ed1ab_0
  - sqlite=3.38.0=hc218d9a_0
  - sshfs=2022.3.1=py38h578d9bd_1
  - stack_data=0.2.0=pyhd8ed1ab_0
  - starlette=0.19.1=pyhd8ed1ab_0
  - statsmodels=0.13.2=py38h6c62de6_0
  - ta-lib=0.4.19=py38h6c62de6_3
  - tabulate=0.8.9=pyhd8ed1ab_0
  - tbb=2021.6.0=intel_835
  - tbb4py=2021.6.0=py38_intel_835
  - tblib=1.7.0=pyhd8ed1ab_0
  - tensorboardx=2.5=pyhd8ed1ab_0
  - terminado=0.15.0=py38h578d9bd_0
  - text-unidecode=1.3=py_0
  - tifffile=2022.5.4=pyhd8ed1ab_0
  - tinycss2=1.1.1=pyhd8ed1ab_0
  - tk=8.6.10=h8e2d9d6_3
  - toml=0.10.2=pyhd8ed1ab_0
  - toolz=0.11.2=pyhd8ed1ab_0
  - tornado=6.1=py38h0a891b7_3
  - tqdm=4.64.0=pyhd8ed1ab_0
  - traitlets=5.2.2.post1=pyhd8ed1ab_0
  - traittypes=0.2.1=pyh9f0ad1d_2
  - typing-extensions=4.2.0=hd8ed1ab_1
  - typing_extensions=4.2.0=pyha770c72_1
  - tzdata=2022a=h191b570_0
  - tzlocal=4.2=py38h578d9bd_0
  - unicodedata2=14.0.0=py38h0a891b7_1
  - unidecode=1.3.4=pyhd8ed1ab_0
  - urllib3=1.26.9=pyhd8ed1ab_0
  - uvicorn=0.16.0=py38h578d9bd_0
  - vaex=4.9.0=pyhd8ed1ab_0
  - vaex-astro=0.9.1=pyhd8ed1ab_0
  - vaex-core=4.9.1=py38hed40c54_0
  - vaex-hdf5=0.12.1=pyhd8ed1ab_0
  - vaex-jupyter=0.7.0=pyhd8ed1ab_0
  - vaex-ml=0.17.0=pyhd8ed1ab_0
  - vaex-server=0.8.1=pyhd8ed1ab_0
  - vaex-viz=0.5.1=pyhd8ed1ab_0
  - virtualenv=20.14.1=py38h578d9bd_0
  - voluptuous=0.13.1=pyhd8ed1ab_0
  - wcwidth=0.2.5=pyh9f0ad1d_2
  - webencodings=0.5.1=py_1
  - websocket-client=1.3.2=pyhd8ed1ab_0
  - wheel=0.37.1=pyhd3eb1b0_0
  - widgetsnbextension=3.6.0=py38h578d9bd_0
  - xarray=2022.3.0=pyhd8ed1ab_0
  - xmltodict=0.13.0=pyhd8ed1ab_0
  - xorg-kbproto=1.0.7=h7f98852_1002
  - xorg-libice=1.0.10=h7f98852_0
  - xorg-libsm=1.2.3=hd9c2040_1000
  - xorg-libx11=1.7.2=h7f98852_0
  - xorg-libxau=1.0.9=h7f98852_0
  - xorg-libxdmcp=1.1.3=h7f98852_0
  - xorg-libxext=1.3.4=h7f98852_1
  - xorg-libxrender=0.9.10=h7f98852_1003
  - xorg-renderproto=0.11.1=h7f98852_1002
  - xorg-xextproto=7.3.0=h7f98852_1002
  - xorg-xproto=7.0.31=h7f98852_1007
  - xyzservices=2022.4.0=pyhd8ed1ab_0
  - xz=5.2.5=h74280d8_2
  - yaml=0.2.5=h7f98852_2
  - yarl=1.7.2=py38h0a891b7_2
  - zc.lockfile=2.0=pyhd8ed1ab_1
  - zeromq=4.3.4=h9c3ff4c_1
  - zfp=0.5.5=h9c3ff4c_8
  - zict=2.2.0=pyhd8ed1ab_0
  - zipp=3.8.0=pyhd8ed1ab_0
  - zlib=1.2.12=h166bdaf_0
  - zlib-ng=2.0.6=h166bdaf_0
  - zstandard=0.17.0=py38h0a891b7_1
  - zstd=1.5.2=h8a70e8d_1
  - pip:
    - lckr-jupyterlab-variableinspector==3.0.9

Reproduction script

ray up --no-config-cache examples/ray_cluster.yaml  -y

ray_cluster.yaml:

cluster_name: default
provider:
  type: local
  head_ip: 192.168.0.101
  worker_ips:
    - 192.168.0.110
    - 192.168.0.106
    - 192.168.0.107
    - 192.168.0.108
auth:
  ssh_user: toaster
upscaling_speed: 1.0
idle_timeout_minutes: 5

# needed so we can run `setup commands`
file_mounts: {
      "/mnt/workspace/ray/mount_driver_node": "~/workspace/repos/puma-lab",
}
cluster_synced_files: []
file_mounts_sync_continuously: False
rsync_exclude:
  - "**/.git"
  - "**/.git/**"
rsync_filter:
  - ".gitignore"
initialization_commands: []
setup_commands:
  - mamba env create -q -n puma-lab -f /mnt/workspace/ray/mount_driver_node/env.yaml || mamba env update -q -n puma-lab -f /mnt/workspace/ray/mount_driver_node/env.yaml
head_setup_commands: []
worker_setup_commands: []
head_start_ray_commands:
  - conda activate puma-lab && ray stop
  - conda activate puma-lab && ulimit -c unlimited && export RAY_DISABLE_MEMORY_MONITOR=1; MKL_NUM_THREADS=$(nproc); OPENBLAS_NUM_THREADS=$(nproc); OMP_NUM_THREADS=$(nproc) && ray start --head --port=6379 --autoscaling-config=~/ray_bootstrap_config.yaml --system-config='{"automatic_object_spilling_enabled":true,"max_io_workers":8,"min_spilling_size":104857600,"object_spilling_config":"{\"type\":\"filesystem\",\"params\":{\"directory_path\":\"/mnt/workspace/ray/object_spilling\"}}"}'
worker_start_ray_commands:
  - conda activate puma-lab && ray stop
  - conda activate puma-lab && ulimit -c unlimited && export RAY_DISABLE_MEMORY_MONITOR=1; MKL_NUM_THREADS=$(nproc); OPENBLAS_NUM_THREADS=$(nproc); OMP_NUM_THREADS=$(nproc) && ray start --address=$RAY_HEAD_IP:6379

Issue Severity

Medium

[stale[bot]]

Hi, I'm a bot from the Ray team :)

To help human contributors to focus on more relevant issues, I will automatically add the stale label to issues that have had no activity for more than 4 months.

If there is no further activity in the 14 days, the issue will be closed!

• If you'd like to keep the issue open, just leave any comment, and the stale label will be removed!
• If you'd like to get more attention to the issue, please tag one of Ray's contributors.

You can always ask for help on our discussion forum or Ray's public slack channel.

[stale[bot]]

Hi again! The issue will be closed because there has been no more activity in the 14 days since the last message.

Please feel free to reopen or open a new issue if you'd still like it to be addressed.

Again, you can always ask for help on our discussion forum or Ray's public slack channel.

Thanks again for opening the issue!