search

ray-project / ray

Ray is a unified framework for scaling AI and Python applications. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.
https://ray.io
Apache License 2.0
31.98k stars 5.44k forks source link

[RLllib] Warning when building standard PPOConfig #36424

Open janrope opened 1 year ago

janrope commented 1 year ago

What happened + What you expected to happen

What happened Whenever I create a PPOConfig I get the following warning:

WARNING algorithm_config.py:2546 -- Setting exploration_config={} because you set _enable_rl_module_api=True. When RLModule API are enabled, exploration_config can not be set. If you want to implement custom exploration behaviour, please modify the forward_exploration method of the RLModule at hand. On configs that have a default exploration config, this must be done with config.exploration_config={}.

I assume this is caused by this code snippet in PPOConfig in ppo.py that was changed recently by https://github.com/ray-project/ray/pull/32808:

self.exploration_config = {
    # The Exploration class to use. In the simplest case, this is the name
    # (str) of any class present in the `rllib.utils.exploration` package.
    # You can also provide the python class directly or the full location
    # of your class (e.g. "ray.rllib.utils.exploration.epsilon_greedy.
    # EpsilonGreedy").
    "type": "StochasticSampling",
    # Add constructor kwargs here (if any).
}

# enable the rl module api by default
self.rl_module(_enable_rl_module_api=True)
self.training(_enable_learner_api=True)

What I expected I expected the standard PPOConfig to build without any warning and/or exception.

Versions / Dependencies

I'm building from the following base image from docker hub: rayproject/ray-ml:e957f1-py310-gpu

Full list of dependencies ``` # Name Version Build Channel _libgcc_mutex 0.1 main _openmp_mutex 5.1 1_gnu absl-py 1.4.0 pypi_0 pypi accelerate 0.17.1 pypi_0 pypi adal 1.2.7 pypi_0 pypi aim 3.16.1 pypi_0 pypi aim-ui 3.16.1 pypi_0 pypi aimrecords 0.0.7 pypi_0 pypi aimrocks 0.3.1 pypi_0 pypi aiobotocore 2.4.2 pypi_0 pypi aiofiles 22.1.0 pypi_0 pypi aiohttp 3.8.4 pypi_0 pypi aiohttp-cors 0.7.0 pypi_0 pypi aioitertools 0.11.0 pypi_0 pypi aiorwlock 1.3.0 pypi_0 pypi aiosignal 1.3.1 pypi_0 pypi aiosqlite 0.19.0 pypi_0 pypi ale-py 0.8.1 pypi_0 pypi alembic 1.11.1 pypi_0 pypi anyio 3.7.0 pypi_0 pypi applicationinsights 0.11.10 pypi_0 pypi argcomplete 1.12.3 pypi_0 pypi argon2-cffi 21.3.0 pypi_0 pypi argon2-cffi-bindings 21.2.0 pypi_0 pypi array-record 0.2.0 pypi_0 pypi arrow 1.2.3 pypi_0 pypi asttokens 2.2.1 pypi_0 pypi astunparse 1.6.3 pypi_0 pypi async-timeout 4.0.2 pypi_0 pypi attrs 23.1.0 pypi_0 pypi autocfg 0.0.8 pypi_0 pypi autograd 1.5 pypi_0 pypi autopage 0.5.1 pypi_0 pypi autorom 0.6.1 pypi_0 pypi autorom-accept-rom-license 0.6.1 pypi_0 pypi ax-platform 0.3.1 pypi_0 pypi azure-cli-core 2.40.0 pypi_0 pypi azure-cli-telemetry 1.0.8 pypi_0 pypi azure-common 1.1.28 pypi_0 pypi azure-core 1.27.0 pypi_0 pypi azure-identity 1.10.0 pypi_0 pypi azure-mgmt-compute 23.1.0 pypi_0 pypi azure-mgmt-core 1.4.0 pypi_0 pypi azure-mgmt-network 19.0.0 pypi_0 pypi azure-mgmt-resource 20.0.0 pypi_0 pypi babel 2.12.1 pypi_0 pypi backcall 0.2.0 pypi_0 pypi backoff 2.2.1 pypi_0 pypi base58 2.0.1 pypi_0 pypi bayesian-optimization 1.2.0 pypi_0 pypi bcrypt 4.0.1 pypi_0 pypi beautifulsoup4 4.12.2 pypi_0 pypi bleach 6.0.0 pypi_0 pypi blessed 1.20.0 pypi_0 pypi blinker 1.6.2 pypi_0 pypi boltons 23.0.0 py310h06a4308_0 boto3 1.26.82 pypi_0 pypi botocore 1.27.59 pypi_0 pypi botorch 0.8.3 pypi_0 pypi brotlipy 0.7.0 py310h7f8727e_1002 build 0.10.0 pypi_0 pypi bzip2 1.0.8 h7b6447c_0 ca-certificates 2023.05.30 h06a4308_0 cached-property 1.5.2 pypi_0 pypi cachetools 5.3.1 pypi_0 pypi catboost 1.2 pypi_0 pypi certifi 2023.5.7 py310h06a4308_0 cffi 1.15.1 py310h74dc2b5_0 charset-normalizer 2.0.4 pyhd3eb1b0_0 chess 1.7.0 pypi_0 pypi chex 0.1.7 pypi_0 pypi click 8.1.3 pypi_0 pypi cliff 4.3.0 pypi_0 pypi cloudpickle 2.2.1 pypi_0 pypi cma 2.7.0 pypi_0 pypi cmaes 0.9.1 pypi_0 pypi cmd2 2.4.3 pypi_0 pypi coloredlogs 15.0.1 pypi_0 pypi colorful 0.5.5 pypi_0 pypi colorlog 6.7.0 pypi_0 pypi comet-ml 3.31.9 pypi_0 pypi comm 0.1.3 pypi_0 pypi commonmark 0.9.1 pypi_0 pypi conda 23.5.0 py310h06a4308_0 conda-content-trust 0.1.3 py310h06a4308_0 conda-package-handling 1.9.0 py310h5eee18b_1 configobj 5.0.8 pypi_0 pypi configspace 0.4.18 pypi_0 pypi contourpy 1.0.7 pypi_0 pypi coolname 2.2.0 pypi_0 pypi cryptography 38.0.1 pypi_0 pypi cycler 0.11.0 pypi_0 pypi cython 0.29.32 pypi_0 pypi databricks-cli 0.17.7 pypi_0 pypi datasets 2.0.0 pypi_0 pypi debugpy 1.6.7 pypi_0 pypi decorator 5.1.1 pypi_0 pypi deepspeed 0.8.3 pypi_0 pypi defusedxml 0.7.1 pypi_0 pypi deprecated 1.2.14 pypi_0 pypi dill 0.3.6 pypi_0 pypi distlib 0.3.6 pypi_0 pypi dm-tree 0.1.8 pypi_0 pypi docker 6.1.3 pypi_0 pypi docker-pycreds 0.4.0 pypi_0 pypi dopamine-rl 4.0.5 pypi_0 pypi dragonfly-opt 0.1.6 pypi_0 pypi dulwich 0.21.5 pypi_0 pypi entrypoints 0.4 pypi_0 pypi etils 1.3.0 pypi_0 pypi everett 3.2.0 pypi_0 pypi exceptiongroup 1.1.1 pypi_0 pypi executing 1.2.0 pypi_0 pypi fairscale 0.4.6 pypi_0 pypi farama-notifications 0.0.4 pypi_0 pypi fastapi 0.96.1 pypi_0 pypi fasteners 0.18 pypi_0 pypi fastjsonschema 2.17.1 pypi_0 pypi filelock 3.12.1 pypi_0 pypi flaml 1.1.1 pypi_0 pypi flask 2.3.2 pypi_0 pypi flatbuffers 2.0.7 pypi_0 pypi flax 0.6.10 pypi_0 pypi flit-core 3.6.0 pyhd3eb1b0_0 fonttools 4.39.4 pypi_0 pypi fqdn 1.5.1 pypi_0 pypi freezegun 1.1.0 pypi_0 pypi frozenlist 1.3.3 pypi_0 pypi fsspec 2023.1.0 pypi_0 pypi future 0.18.3 pypi_0 pypi gast 0.4.0 pypi_0 pypi gin-config 0.5.0 pypi_0 pypi gitdb 4.0.10 pypi_0 pypi gitpython 3.1.31 pypi_0 pypi glfw 2.5.9 pypi_0 pypi gluoncv 0.10.5.post0 pypi_0 pypi google-api-core 2.11.0 pypi_0 pypi google-api-python-client 1.7.8 pypi_0 pypi google-auth 2.19.1 pypi_0 pypi google-auth-httplib2 0.1.0 pypi_0 pypi google-auth-oauthlib 0.4.6 pypi_0 pypi google-oauth 1.0.1 pypi_0 pypi google-pasta 0.2.0 pypi_0 pypi googleapis-common-protos 1.59.0 pypi_0 pypi gpustat 1.1 pypi_0 pypi gputil 1.4.0 pypi_0 pypi gpy 1.10.0 pypi_0 pypi gpytorch 1.9.1 pypi_0 pypi greenlet 2.0.2 pypi_0 pypi grpcio 1.54.2 pypi_0 pypi gunicorn 20.1.0 pypi_0 pypi gym 0.26.2 pypi_0 pypi gym-notices 0.0.8 pypi_0 pypi gymnasium 0.28.1 pypi_0 pypi gymnasium-notices 0.0.1 pypi_0 pypi h11 0.14.0 pypi_0 pypi h5py 3.7.0 pypi_0 pypi hebo 0.3.2 pypi_0 pypi higher 0.2.1 pypi_0 pypi hjson 3.1.0 pypi_0 pypi hpbandster 0.7.4 pypi_0 pypi httplib2 0.22.0 pypi_0 pypi huggingface-hub 0.15.1 pypi_0 pypi humanfriendly 10.0 pypi_0 pypi hyperopt 0.2.5 pypi_0 pypi idna 3.4 py310h06a4308_0 imageio 2.31.0 pypi_0 pypi imageio-ffmpeg 0.4.5 pypi_0 pypi importlib-metadata 6.0.1 pypi_0 pypi importlib-resources 5.12.0 pypi_0 pypi importlib_metadata 4.11.3 hd3eb1b0_0 iniconfig 2.0.0 pypi_0 pypi ipykernel 6.23.1 pypi_0 pypi ipython 8.14.0 pypi_0 pypi ipython-genutils 0.2.0 pypi_0 pypi ipywidgets 8.0.6 pypi_0 pypi isodate 0.6.1 pypi_0 pypi isoduration 20.11.0 pypi_0 pypi itsdangerous 2.1.2 pypi_0 pypi jax 0.4.12 pypi_0 pypi jax-jumpy 1.0.0 pypi_0 pypi jaxlib 0.4.12 pypi_0 pypi jedi 0.18.2 pypi_0 pypi jinja2 3.1.2 pypi_0 pypi jmespath 1.0.1 pypi_0 pypi joblib 1.2.0 pypi_0 pypi json5 0.9.14 pypi_0 pypi jsonpatch 1.32 pyhd3eb1b0_0 jsonpointer 2.1 pyhd3eb1b0_0 jsonschema 4.17.3 pypi_0 pypi jupyter-client 8.2.0 pypi_0 pypi jupyter-core 5.3.0 pypi_0 pypi jupyter-events 0.6.3 pypi_0 pypi jupyter-server 2.6.0 pypi_0 pypi jupyter-server-fileid 0.9.0 pypi_0 pypi jupyter-server-terminals 0.4.4 pypi_0 pypi jupyter-server-ydoc 0.6.1 pypi_0 pypi jupyter-ydoc 0.2.4 pypi_0 pypi jupyterlab 3.6.1 pypi_0 pypi jupyterlab-pygments 0.2.2 pypi_0 pypi jupyterlab-server 2.22.1 pypi_0 pypi jupyterlab-widgets 3.0.7 pypi_0 pypi kaggle-environments 1.7.11 pypi_0 pypi keras 2.11.0 pypi_0 pypi kiwisolver 1.4.4 pypi_0 pypi knack 0.10.1 pypi_0 pypi kubernetes 26.1.0 pypi_0 pypi lazy-loader 0.2 pypi_0 pypi ld_impl_linux-64 2.38 h1181459_1 libclang 16.0.0 pypi_0 pypi libffi 3.3 he6710b0_2 libgcc-ng 11.2.0 h1234567_1 libgomp 11.2.0 h1234567_1 libstdcxx-ng 11.2.0 h1234567_1 libuuid 1.41.5 h5eee18b_0 lightgbm 3.3.5 pypi_0 pypi lightgbm-ray 0.1.8 pypi_0 pypi lightning-bolts 0.4.0 pypi_0 pypi linear-operator 0.3.0 pypi_0 pypi llvmlite 0.40.1rc1 pypi_0 pypi lz4 4.3.2 pypi_0 pypi mako 1.2.4 pypi_0 pypi markdown 3.4.3 pypi_0 pypi markdown-it-py 2.2.0 pypi_0 pypi markupsafe 2.1.3 pypi_0 pypi matplotlib 3.7.1 pypi_0 pypi matplotlib-inline 0.1.6 pypi_0 pypi mdurl 0.1.2 pypi_0 pypi minigrid 2.1.1 pypi_0 pypi mistune 2.0.5 pypi_0 pypi ml-dtypes 0.2.0 pypi_0 pypi mlagents-envs 0.28.0 pypi_0 pypi mlflow 2.2.2 pypi_0 pypi modin 0.18.1 pypi_0 pypi monotonic 1.6 pypi_0 pypi mosaicml 0.12.1 pypi_0 pypi mpmath 1.3.0 pypi_0 pypi msal 1.18.0b1 pypi_0 pypi msal-extensions 1.0.0 pypi_0 pypi msgpack 1.0.5 pypi_0 pypi msgpack-numpy 0.4.8 pypi_0 pypi msrest 0.7.1 pypi_0 pypi msrestazure 0.6.4 pypi_0 pypi mujoco 2.2.0 pypi_0 pypi mujoco-py 2.1.2.14 pypi_0 pypi multidict 6.0.4 pypi_0 pypi multipledispatch 0.6.0 pypi_0 pypi multiprocess 0.70.14 pypi_0 pypi mxnet 1.8.0.post0 pypi_0 pypi nbclassic 1.0.0 pypi_0 pypi nbclient 0.8.0 pypi_0 pypi nbconvert 7.4.0 pypi_0 pypi nbformat 5.9.0 pypi_0 pypi ncurses 6.4 h6a678d5_0 nest-asyncio 1.5.6 pypi_0 pypi netifaces 0.11.0 pypi_0 pypi networkx 3.1 pypi_0 pypi nevergrad 0.4.3.post7 pypi_0 pypi ninja 1.11.1 pypi_0 pypi notebook 6.5.4 pypi_0 pypi notebook-shim 0.2.3 pypi_0 pypi numba 0.57.0 pypi_0 pypi numpy 1.24.3 pypi_0 pypi nvidia-cublas-cu11 11.10.3.66 pypi_0 pypi nvidia-cuda-nvrtc-cu11 11.7.99 pypi_0 pypi nvidia-cuda-runtime-cu11 11.7.99 pypi_0 pypi nvidia-cudnn-cu11 8.5.0.96 pypi_0 pypi nvidia-ml-py 11.525.112 pypi_0 pypi oauthlib 3.2.2 pypi_0 pypi onnx 1.12.0 pypi_0 pypi onnxruntime 1.14.1 pypi_0 pypi open-spiel 1.2 pypi_0 pypi opencensus 0.11.2 pypi_0 pypi opencensus-context 0.1.3 pypi_0 pypi opencv-python 4.7.0.72 pypi_0 pypi openssl 1.1.1t h7f8727e_0 opentelemetry-api 1.1.0 pypi_0 pypi opentelemetry-exporter-otlp 1.1.0 pypi_0 pypi opentelemetry-exporter-otlp-proto-common 1.18.0 pypi_0 pypi opentelemetry-exporter-otlp-proto-grpc 1.1.0 pypi_0 pypi opentelemetry-exporter-otlp-proto-http 1.18.0 pypi_0 pypi opentelemetry-proto 1.1.0 pypi_0 pypi opentelemetry-sdk 1.1.0 pypi_0 pypi opentelemetry-semantic-conventions 0.20b0 pypi_0 pypi opt-einsum 3.3.0 pypi_0 pypi optax 0.1.5 pypi_0 pypi optuna 2.10.0 pypi_0 pypi orbax-checkpoint 0.2.5 pypi_0 pypi overrides 7.3.1 pypi_0 pypi packaging 23.1 pypi_0 pypi pandas 1.5.3 pypi_0 pypi pandocfilters 1.5.0 pypi_0 pypi paramiko 2.12.0 pypi_0 pypi paramz 0.9.5 pypi_0 pypi parso 0.8.3 pypi_0 pypi pathtools 0.1.2 pypi_0 pypi patsy 0.5.3 pypi_0 pypi pbr 5.11.1 pypi_0 pypi pettingzoo 1.22.1 pypi_0 pypi pexpect 4.8.0 pypi_0 pypi pickleshare 0.7.5 pypi_0 pypi pillow 9.5.0 pypi_0 pypi pip 23.1.2 pypi_0 pypi pip-tools 6.13.0 pypi_0 pypi pkginfo 1.9.6 pypi_0 pypi platformdirs 3.5.3 pypi_0 pypi plotly 5.15.0 pypi_0 pypi pluggy 1.0.0 py310h06a4308_1 portalocker 2.7.0 pypi_0 pypi prettytable 3.7.0 pypi_0 pypi prometheus-client 0.13.1 pypi_0 pypi promise 2.3 pypi_0 pypi prompt-toolkit 3.0.38 pypi_0 pypi protobuf 3.19.6 pypi_0 pypi psutil 5.9.5 pypi_0 pypi ptyprocess 0.7.0 pypi_0 pypi pure-eval 0.2.2 pypi_0 pypi py-cpuinfo 9.0.0 pypi_0 pypi py-spy 0.3.14 pypi_0 pypi py3nvml 0.2.7 pypi_0 pypi pyaml 23.5.9 pypi_0 pypi pyarrow 11.0.0 pypi_0 pypi pyasn1 0.5.0 pypi_0 pypi pyasn1-modules 0.3.0 pypi_0 pypi pycosat 0.6.4 py310h5eee18b_0 pycparser 2.21 pyhd3eb1b0_0 pydantic 1.10.9 pypi_0 pypi pydeprecate 0.3.2 pypi_0 pypi pygame 2.1.2 pypi_0 pypi pyglet 1.5.15 pypi_0 pypi pygments 2.15.1 pypi_0 pypi pyjwt 2.7.0 pypi_0 pypi pymoo 0.5.0 pypi_0 pypi pymunk 6.2.1 pypi_0 pypi pynacl 1.5.0 pypi_0 pypi pyopengl 3.1.7 pypi_0 pypi pyopenssl 22.1.0 pypi_0 pypi pyparsing 3.0.9 pypi_0 pypi pyperclip 1.8.2 pypi_0 pypi pypng 0.20220715.0 pypi_0 pypi pyproject-hooks 1.0.0 pypi_0 pypi pyro-api 0.1.2 pypi_0 pypi pyro-ppl 1.8.4 pypi_0 pypi pyro4 4.82 pypi_0 pypi pyrsistent 0.19.3 pypi_0 pypi pysocks 1.7.1 py310h06a4308_0 pytest 7.3.2 pypi_0 pypi pytest-remotedata 0.3.2 pypi_0 pypi python 3.10.8 haa1d7c7_0 python-dateutil 2.8.2 pypi_0 pypi python-graphviz 0.8.4 pypi_0 pypi python-json-logger 2.0.7 pypi_0 pypi pytorch-lightning 1.6.5 pypi_0 pypi pytorch-ranger 0.1.1 pypi_0 pypi pytz 2022.7.1 pypi_0 pypi pywavelets 1.4.1 pypi_0 pypi pyyaml 6.0 pypi_0 pypi pyzmq 25.1.0 pypi_0 pypi querystring-parser 1.2.4 pypi_0 pypi ray 3.0.0.dev0 pypi_0 pypi ray-lightning 0.3.0 pypi_0 pypi readline 8.2 h5eee18b_0 recsim 0.2.4 pypi_0 pypi redis 3.5.3 pypi_0 pypi regex 2023.6.3 pypi_0 pypi requests 2.31.0 pypi_0 pypi requests-oauthlib 1.3.1 pypi_0 pypi requests-toolbelt 1.0.0 pypi_0 pypi responses 0.18.0 pypi_0 pypi restrictedpython 6.0 pypi_0 pypi rfc3339-validator 0.1.4 pypi_0 pypi rfc3986-validator 0.1.1 pypi_0 pypi rich 12.0.1 pypi_0 pypi rsa 4.9 pypi_0 pypi ruamel.yaml 0.17.21 py310h5eee18b_0 ruamel.yaml.clib 0.2.6 py310h5eee18b_1 rware 1.0.3 dev_0 s3fs 2023.1.0 pypi_0 pypi s3transfer 0.6.1 pypi_0 pypi scikit-image 0.21.0 pypi_0 pypi scikit-learn 1.2.2 pypi_0 pypi scikit-optimize 0.9.0 pypi_0 pypi scipy 1.10.1 pypi_0 pypi segment-analytics-python 2.2.2 pypi_0 pypi semantic-version 2.10.0 pypi_0 pypi send2trash 1.8.2 pypi_0 pypi sentencepiece 0.1.96 pypi_0 pypi sentry-sdk 1.25.1 pypi_0 pypi serpent 1.41 pypi_0 pypi setproctitle 1.3.2 pypi_0 pypi setuptools 67.8.0 py310h06a4308_0 shap 0.41.0 pypi_0 pypi shortuuid 1.0.1 pypi_0 pypi sigopt 7.5.0 pypi_0 pypi six 1.13.0 pypi_0 pypi slicer 0.0.7 pypi_0 pypi smart-open 6.3.0 pypi_0 pypi smmap 5.0.0 pypi_0 pypi sniffio 1.3.0 pypi_0 pypi soupsieve 2.4.1 pypi_0 pypi sqlalchemy 1.4.48 pypi_0 pypi sqlite 3.40.1 h5082296_0 sqlparse 0.4.4 pypi_0 pypi stack-data 0.6.2 pypi_0 pypi starlette 0.27.0 pypi_0 pypi statsmodels 0.14.0 pypi_0 pypi stevedore 5.1.0 pypi_0 pypi supersuit 3.7.0 pypi_0 pypi sympy 1.12 pypi_0 pypi tabulate 0.9.0 pypi_0 pypi tblib 1.7.0 pypi_0 pypi tenacity 8.2.2 pypi_0 pypi tensorboard 2.11.2 pypi_0 pypi tensorboard-data-server 0.6.1 pypi_0 pypi tensorboard-plugin-wit 1.8.1 pypi_0 pypi tensorboardx 2.4.1 pypi_0 pypi tensorflow 2.11.0 pypi_0 pypi tensorflow-datasets 4.9.0 pypi_0 pypi tensorflow-estimator 2.11.0 pypi_0 pypi tensorflow-io-gcs-filesystem 0.32.0 pypi_0 pypi tensorflow-metadata 1.13.0 pypi_0 pypi tensorflow-probability 0.19.0 pypi_0 pypi tensorstore 0.1.37 pypi_0 pypi termcolor 2.3.0 pypi_0 pypi terminado 0.17.1 pypi_0 pypi tf-slim 1.1.0 pypi_0 pypi tf2onnx 1.13.0 pypi_0 pypi threadpoolctl 3.1.0 pypi_0 pypi tifffile 2023.4.12 pypi_0 pypi timm 0.4.5 pypi_0 pypi tinycss2 1.2.1 pypi_0 pypi tinyscaler 1.2.6 pypi_0 pypi tk 8.6.12 h1ccaba5_0 tokenizers 0.12.1 pypi_0 pypi toml 0.10.2 pypi_0 pypi tomli 2.0.1 pypi_0 pypi toolz 0.12.0 py310h06a4308_0 torch 1.13.0+cu116 pypi_0 pypi torch-cluster 1.6.0+pt113cu116 pypi_0 pypi torch-geometric 2.1.0 pypi_0 pypi torch-optimizer 0.3.0 pypi_0 pypi torch-scatter 2.1.0+pt113cu116 pypi_0 pypi torch-sparse 0.6.15+pt113cu116 pypi_0 pypi torch-spline-conv 1.2.1+pt113cu116 pypi_0 pypi torchmetrics 0.9.3 pypi_0 pypi torchvision 0.14.0+cu116 pypi_0 pypi tornado 6.3.2 pypi_0 pypi tqdm 4.65.0 py310h2f386ee_0 traitlets 5.9.0 pypi_0 pypi transformers 4.19.1 pypi_0 pypi tune-sklearn 0.4.4 pypi_0 pypi typeguard 2.13.3 pypi_0 pypi typer 0.9.0 pypi_0 pypi typing-extensions 4.6.3 pypi_0 pypi tzdata 2023.3 pypi_0 pypi uri-template 1.2.0 pypi_0 pypi uritemplate 3.0.1 pypi_0 pypi urllib3 1.26.16 py310h06a4308_0 uvicorn 0.22.0 pypi_0 pypi virtualenv 20.21.0 pypi_0 pypi wandb 0.13.4 pypi_0 pypi wcwidth 0.2.6 pypi_0 pypi webcolors 1.13 pypi_0 pypi webencodings 0.5.1 pypi_0 pypi websocket-client 1.5.3 pypi_0 pypi werkzeug 2.3.6 pypi_0 pypi wheel 0.37.1 pyhd3eb1b0_0 widgetsnbextension 4.0.7 pypi_0 pypi wrapt 1.15.0 pypi_0 pypi wurlitzer 3.0.3 pypi_0 pypi xgboost 1.3.3 pypi_0 pypi xgboost-ray 0.1.15 pypi_0 pypi xmltodict 0.13.0 pypi_0 pypi xxhash 3.2.0 pypi_0 pypi xz 5.2.10 h5eee18b_1 y-py 0.5.9 pypi_0 pypi yacs 0.1.8 pypi_0 pypi yarl 1.9.2 pypi_0 pypi ypy-websocket 0.8.2 pypi_0 pypi zipp 3.11.0 py310h06a4308_0 zlib 1.2.13 h5eee18b_0 zoopt 0.4.1 pypi_0 pypi ```

Reproduction script

from ray.rllib.algorithms.ppo import PPOConfig

config = PPOConfig()

Issue Severity

Low: It annoys or frustrates me.

lyzyn commented 8 months ago

I have also encountered this problem. Have you resolved it? This troubles me a lot.

gresavage commented 8 months ago

This is an artifact from being in an intermediary state between the old APIs and new - rest assured there is no actual bug/issue with the code and it will run just fine.

The way I see it, you have three options in the interim until the old APIs are completely deprecated:

a) set log_level="ERROR" in the configuration of your algorithm b) Create a custom PPOConfig and override the PPOConfig.rl_module method to set self.exploration_config={} before calling super() c) Disable the new APIs by setting _enable_learner_api=False and _enable_rl_module_api=False in your configuration

Hope this helps