Problem with ray 2.0.0

[loooic]

Hi,

I am very interested with this work. I am looking for solution to couple Eplus with RL for hvac control optimization. I tried to run this repo but i did not managed to install ray[RLlib]==2.0.0. It sounds to be to old.

If I install a newer version of ray, it doesn't work and there is a recommendation to work with gymnasium instead of gym.

Is there any update of this work ?

Thanks a lot.

[antoine-galataud]

Hi @loooic, thank you for taking the time to report this issue!

I guess you tried with latest Ray release (2.3.0) which replaced OpenAI gym by gymnasium.

Can you try with Ray 2.2.0? If you use tensorflow it will ask for tensorflow-probability too (which is a bug fixed later in Ray).

Here is a sample requirements.txt (I've tried a training with it and it runs ok):

ray[rllib]==2.2.0
gym==0.23.1
numpy==1.23.5
tensorflow==2.9.0
tensorflow-probability==0.14.1

Thanks!

[antoine-galataud]

@loooic I actually took some time to upgrade to Ray 2.3.0 and Gymnasium. Please check #2 and let me know if something goes wrong.

[loooic]

@antoine-galataud thanks a lot for your quick reply. Now I managed to install all the requirements that you indicate in #2. The script starts to run but unfortunately I still have the following error :

if obs["htg_spt"] > 0 and obs["clg_spt"] > 0: (PPO pid=21424) KeyError: 'htg_spt' (PPO pid=21424) (PPO pid=21424) The above exception was the direct cause of the following exception: (PPO pid=21424) (PPO pid=21424) ray::PPO.init() (pid=21424, ip=127.0.0.1, repr=PPO) (PPO pid=21424) File "C:\Users\dell\anaconda3\envs\RLlib_env2\lib\site-packages\ray\rllib\utils\pre_checks\env.py", line 82, in check_env (PPO pid=21424) check_gym_environments(env) (PPO pid=21424) File "C:\Users\dell\anaconda3\envs\RLlib_env2\lib\site-packages\ray\rllib\utils\pre_checks\env.py", line 183, in check_gym_environments (PPO pid=21424) raise ValueError( (PPO pid=21424) ValueError: Your gymnasium.Env's reset() method raised an Exception! (PPO pid=21424) (PPO pid=21424) The above error has been found in your environment! We've added a module for checking your custom environments. It may cause your experiment to fail if your environment is not set up correctly. You can disable this behavior via calling config.environment(disable_env_checking=True). You can run the environment checking module standalone by calling ray.rllib.utils.check_env([your env]).

I don't know if you have an idea of the problem.

Anyway thanks a lot for your help !

[antoine-galataud]

@loooic have you tested with both the upgraded dependencies and the new environment code? I've simplified and strengthened flow control to avoid issues with E+ running in the background. If it still occurs, can you provide the whole stack trace? What's your environment like (OS, python version, ...)?

[loooic]

@antoine-galataud yes, I runned the new script with the upgraded dependency.

Here is my environment on windows 10 pro (RLlib_env2) C:\Users\dell>conda list python packages in environment at C:\Users\dell\anaconda3\envs\RLlib_env2:

Name Version Build Channel ipython 8.11.0 pyh08f2357_0 conda-forge python 3.10.9 h966fe2a_0 python-dateutil 2.8.2 pyhd8ed1ab_0 conda-forge python_abi 3.10 2_cp310 conda-forge

(RLlib_env2) C:\Users\dell>pip list Package Version

---

absl-py 1.4.0 aiosignal 1.3.1 asttokens 2.2.1 astunparse 1.6.3 attrs 22.2.0 backcall 0.2.0 backports.functools-lru-cache 1.6.4 cachetools 5.3.0 certifi 2022.12.7 charset-normalizer 3.0.1 click 8.1.3 cloudpickle 2.2.1 colorama 0.4.6 debugpy 1.5.1 decorator 5.1.1 distlib 0.3.6 dm-tree 0.1.8 executing 1.2.0 filelock 3.9.0 flatbuffers 1.12 frozenlist 1.3.3 gast 0.4.0 google-auth 2.16.1 google-auth-oauthlib 0.4.6 google-pasta 0.2.0 grpcio 1.51.3 Gymnasium 0.26.3 gymnasium-notices 0.0.1 h5py 3.8.0 idna 3.4 imageio 2.26.0 importlib-metadata 6.0.0 ipykernel 6.15.0 ipython 8.11.0 jedi 0.18.2 jsonschema 4.17.3 jupyter_client 8.0.3 jupyter_core 5.2.0 keras 2.9.0 Keras-Preprocessing 1.1.2 lazy_loader 0.1 libclang 15.0.6.1 lz4 4.3.2 Markdown 3.4.1 markdown-it-py 2.2.0 MarkupSafe 2.1.2 matplotlib-inline 0.1.6 mdurl 0.1.2 msgpack 1.0.4 nest-asyncio 1.5.6 networkx 3.0 numpy 1.23.5 oauthlib 3.2.2 opt-einsum 3.3.0 packaging 23.0 pandas 1.5.3 parso 0.8.3 pickleshare 0.7.5 Pillow 9.4.0 pip 22.3.1 platformdirs 3.0.0 prompt-toolkit 3.0.38 protobuf 3.19.6 psutil 5.9.0 pure-eval 0.2.2 pyasn1 0.4.8 pyasn1-modules 0.2.8 Pygments 2.14.0 pyrsistent 0.19.3 python-dateutil 2.8.2 pytz 2022.7.1 PyWavelets 1.4.1 pywin32 305.1 PyYAML 6.0 pyzmq 23.2.0 ray 2.3.0 requests 2.28.2 requests-oauthlib 1.3.1 rich 13.3.1 rsa 4.9 scikit-image 0.20.0 scipy 1.10.1 setuptools 65.6.3 six 1.16.0 stack-data 0.6.2 tabulate 0.9.0 tensorboard 2.9.1 tensorboard-data-server 0.6.1 tensorboard-plugin-wit 1.8.1 tensorboardX 2.6 tensorflow 2.9.0 tensorflow-estimator 2.9.0 tensorflow-io-gcs-filesystem 0.31.0 tensorflow-probability 0.14.1 termcolor 2.2.0 tifffile 2023.2.28 tornado 6.2 traitlets 5.9.0 typer 0.7.0 typing_extensions 4.5.0 urllib3 1.26.14 virtualenv 20.20.0 wcwidth 0.2.6 Werkzeug 2.2.3 wheel 0.38.4 wincertstore 0.2 wrapt 1.15.0 zipp 3.15.0

Here is a text file that contain a copy of the terminal output during run time : out_put2.txt

Here is the error file that is produced during run time : error.txt

Maybe I made something wrong with the environment ?

[antoine-galataud]

I'm not able to reproduce your problem so I'm proposing a workaround by intuition. Please checkout the latest branch commit and retest!

[loooic]

@antoine-galataud I ran the latest branch commit, it is still running 200 min, so I think your last commit resolved the last problem for me.

Still running, so lets see but anyway, thanks a lot.

[loooic]

@antoine-galataud Still running after 845 min, is it normal ?

Thanks

[antoine-galataud]

Do you see a Ray Tune output like this:

+-------------------------------+----------+---------------------+--------+------------------+--------+----------+----------------------+----------------------+--------------------+
| Trial name                    | status   | loc                 |   iter |   total time (s) |     ts |   reward |   episode_reward_max |   episode_reward_min |   episode_len_mean |
|-------------------------------+----------+---------------------+--------+------------------+--------+----------+----------------------+----------------------+--------------------|
| PPO_EnergyPlusEnv_9f072_00000 | RUNNING  | 192.168.1.20:579218 |    118 |          336.243 | 120832 | -55148.4 |             -55078.3 |             -55218.6 |              35039 |
+-------------------------------+----------+---------------------+--------+------------------+--------+----------+----------------------+----------------------+--------------------+

?

It's supposed to stop after 1e6 timesteps (default cli argument).

[loooic]

Hi @antoine-galataud,

Would like to thank you for your support.

I finally install the environment on linux and ran it and it is now working.

Thanks a lot.

[antoine-galataud]

@loooic that's great news. And that's likely a better bet for your future experiments ;)