search

ray-project / ray

Ray is a unified framework for scaling AI and Python applications. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.
https://ray.io
Apache License 2.0
32.78k stars 5.56k forks source link

Registering & Using OpenAI Gym environments from the RoboHive API in Ray RLlib & Tune #37231

Open ChristosPeridis opened 1 year ago

ChristosPeridis commented 1 year ago

Hello dear Ray Team,

I hope you are doing well. I have been using Ray API ( and more specifically RLlib & Tune) for conducting Reinforcement Learning experiments with the various RL algorithms RLlib offers throughout my research journey. I used Ray to test the performance of DRL SOTA algorithms (like Rainbow DQN, PPO, DD-PPO, SAC, etc.) on the CT-graph benchmark. The CT-graph benchmark is a custom OpenAI Gym (now been upgraded to Gymnasium) environment that offers multiple RL tasks and MDP & POMDP settings. Since the CT-graph is a custom environment I have created a "env_creator()" function, where I configure and make the environment using "gym.makke()" method, and then I return the newly build environment. Next I register the specific configuration build of the CT-graph to the Ray API using the "register_env()" method, where I pass the desired name to use for the registration and then then a reference to the "env_creator" method. You can see the exact implementation in the following notebook . Following these steps I was able to register any configuration of the custom environment CT-graph to Ray API, and then use its implemented SOTA RL algorithms along with Tune to conduct experiments and test the algorithms performance.

In my new project I have to work with multiple tasks in Robotics scenarios. For these setting I have found the RoboHive API which is build on the MuJoco simulator, uses mujoco-py and mujoco python libraries and supports multi-task scenarios natively. All its environments are developed as OpenAI Gym environments so that they are easily used in RL settings. Since the environments offered by the RoboHHive API are not from the classic standard OpenAI Gym environments, but custom ones, I had followed the same steps and technique, I followed for registering and using the CT-graph benchmark, to register these custom environments to the Ray API. Despite following exactly the same steps and technique, for example for registering and using the RoboHive "FrankaPickPlaceFixed-v0" environment (which has been registered to Gym by the RoboHive API), I received the following error message when attempting to run one experiment with Tune using SAC:

2023-07-09 22:30:20,527 ERROR ray_trial_executor.py:102 -- An exception occurred when trying to stop the Ray actor:Traceback (most recent call last): File "/home/cocp5/anaconda3/envs/rhRL03/lib/python3.8/site-packages/ray/tune/ray_trial_executor.py", line 93, in post_stop_cleanup ray.get(future, timeout=0) File "/home/cocp5/anaconda3/envs/rhRL03/lib/python3.8/site-packages/ray/_private/client_mode_hook.py", line 105, in wrapper return func(*args, **kwargs) File "/home/cocp5/anaconda3/envs/rhRL03/lib/python3.8/site-packages/ray/worker.py", line 1811, in get raise value ray.exceptions.RayActorError: The actor died because of an error raised in its creation task, ray::SACTrainer.init() (pid=108390, ip=172.19.132.220, repr=SACTrainer) File "/home/cocp5/anaconda3/envs/rhRL03/lib/python3.8/site-packages/ray/rllib/agents/trainer.py", line 1035, in _init raise NotImplementedError NotImplementedError

During handling of the above exception, another exception occurred:

ray::SACTrainer.init() (pid=108390, ip=172.19.132.220, repr=SACTrainer) File "/home/cocp5/anaconda3/envs/rhRL03/lib/python3.8/site-packages/gym/envs/registration.py", line 132, in spec return self.env_specs[id] KeyError: 'FrankaPickPlaceFixed-v0'

During handling of the above exception, another exception occurred:

ray::SACTrainer.init() (pid=108390, ip=172.19.132.220, repr=SACTrainer) File "/home/cocp5/anaconda3/envs/rhRL03/lib/python3.8/site-packages/ray/rllib/env/utils.py", line 54, in gym_env_creator return gym.make(env_descriptor, env_context) File "/home/cocp5/anaconda3/envs/rhRL03/lib/python3.8/site-packages/gym/envs/registration.py", line 156, in make return registry.make(id, kwargs) File "/home/cocp5/anaconda3/envs/rhRL03/lib/python3.8/site-packages/gym/envs/registration.py", line 100, in make spec = self.spec(path) File "/home/cocp5/anaconda3/envs/rhRL03/lib/python3.8/site-packages/gym/envs/registration.py", line 142, in spec raise error.UnregisteredEnv('No registered env with id: {}'.format(id)) gym.error.UnregisteredEnv: No registered env with id: FrankaPickPlaceFixed-v0

During handling of the above exception, another exception occurred:

ray::SACTrainer.init() (pid=108390, ip=172.19.132.220, repr=SACTrainer) File "/home/cocp5/anaconda3/envs/rhRL03/lib/python3.8/site-packages/ray/rllib/agents/sac/sac.py", line 192, in init super().init(*args, **kwargs) File "/home/cocp5/anaconda3/envs/rhRL03/lib/python3.8/site-packages/ray/rllib/agents/trainer.py", line 830, in init super().init( File "/home/cocp5/anaconda3/envs/rhRL03/lib/python3.8/site-packages/ray/tune/trainable.py", line 149, in init self.setup(copy.deepcopy(self.config)) File "/home/cocp5/anaconda3/envs/rhRL03/lib/python3.8/site-packages/ray/rllib/agents/trainer.py", line 911, in setup self.workers = WorkerSet( File "/home/cocp5/anaconda3/envs/rhRL03/lib/python3.8/site-packages/ray/rllib/evaluation/worker_set.py", line 162, in init self._local_worker = self._make_worker( File "/home/cocp5/anaconda3/envs/rhRL03/lib/python3.8/site-packages/ray/rllib/evaluation/worker_set.py", line 567, in _make_worker worker = cls( File "/home/cocp5/anaconda3/envs/rhRL03/lib/python3.8/site-packages/ray/rllib/evaluation/rollout_worker.py", line 493, in init self.env = env_creator(copy.deepcopy(self.env_context)) File "/home/cocp5/anaconda3/envs/rhRL03/lib/python3.8/site-packages/ray/rllib/env/utils.py", line 56, in gym_env_creator raise EnvError(ERR_MSG_INVALID_ENV_DESCRIPTOR.format(env_descriptor)) ray.rllib.utils.error.EnvError: The env string you provided ('FrankaPickPlaceFixed-v0') is: a) Not a supported/installed environment. b) Not a tune-registered environment creator. c) Not a valid env class string.

Try one of the following: a) For Atari support: pip install gym[atari] autorom[accept-rom-license]. For VizDoom support: Install VizDoom (https://github.com/mwydmuch/ViZDoom/blob/master/doc/Building.md) and pip install vizdoomgym. For PyBullet support: pip install pybullet. b) To register your custom env, do from ray import tune; tune.register('[name]', lambda cfg: [return env obj from here using cfg]). Then in your config, do config['env'] = [name]. c) Make sure you provide a fully qualified classpath, e.g.: ray.rllib.examples.env.repeat_after_me_env.RepeatAfterMeEnv

My "env_creator()" method is the following:

def env_creator(env_config={}): import robohive env = gym.make('FrankaPickPlaceFixed-v0') env.reset() return env

I have been using Ray version 1.12.1 for testing the code.

What is it causing the issue and how could I fix it?

Thank you very much in advance for your valuable help!!!

I am always at your disposal for any further queries regarding my code and my environment setup.

Kind regards,

Christos Peridis

lazyaa commented 1 year ago

I had the same problem. Did you solve it?