Ray is a unified framework for scaling AI and Python applications. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.
I hope you are doing well. I have been using Ray API ( and more specifically RLlib & Tune) for conducting Reinforcement Learning experiments with the various RL algorithms RLlib offers throughout my research journey. I used Ray to test the performance of DRL SOTA algorithms (like Rainbow DQN, PPO, DD-PPO, SAC, etc.) on the CT-graph benchmark. The CT-graph benchmark is a custom OpenAI Gym (now been upgraded to Gymnasium) environment that offers multiple RL tasks and MDP & POMDP settings. Since the CT-graph is a custom environment I have created a "env_creator()" function, where I configure and make the environment using "gym.makke()" method, and then I return the newly build environment. Next I register the specific configuration build of the CT-graph to the Ray API using the "register_env()" method, where I pass the desired name to use for the registration and then then a reference to the "env_creator" method. You can see the exact implementation in the following notebook . Following these steps I was able to register any configuration of the custom environment CT-graph to Ray API, and then use its implemented SOTA RL algorithms along with Tune to conduct experiments and test the algorithms performance.
In my new project I have to work with multiple tasks in Robotics scenarios. For these setting I have found the RoboHive API which is build on the MuJoco simulator, uses mujoco-py and mujoco python libraries and supports multi-task scenarios natively. All its environments are developed as OpenAI Gym environments so that they are easily used in RL settings. Since the environments offered by the RoboHHive API are not from the classic standard OpenAI Gym environments, but custom ones, I had followed the same steps and technique, I followed for registering and using the CT-graph benchmark, to register these custom environments to the Ray API. Despite following exactly the same steps and technique, for example for registering and using the RoboHive "FrankaPickPlaceFixed-v0" environment (which has been registered to Gym by the RoboHive API), I received the following error message when attempting to run one experiment with Tune using SAC:
2023-07-09 22:30:20,527 ERROR ray_trial_executor.py:102 -- An exception occurred when trying to stop the Ray actor:Traceback (most recent call last):
File "/home/cocp5/anaconda3/envs/rhRL03/lib/python3.8/site-packages/ray/tune/ray_trial_executor.py", line 93, in post_stop_cleanup
ray.get(future, timeout=0)
File "/home/cocp5/anaconda3/envs/rhRL03/lib/python3.8/site-packages/ray/_private/client_mode_hook.py", line 105, in wrapper
return func(*args, **kwargs)
File "/home/cocp5/anaconda3/envs/rhRL03/lib/python3.8/site-packages/ray/worker.py", line 1811, in get
raise value
ray.exceptions.RayActorError: The actor died because of an error raised in its creation task, ray::SACTrainer.init() (pid=108390, ip=172.19.132.220, repr=SACTrainer)
File "/home/cocp5/anaconda3/envs/rhRL03/lib/python3.8/site-packages/ray/rllib/agents/trainer.py", line 1035, in _init
raise NotImplementedError
NotImplementedError
During handling of the above exception, another exception occurred:
ray::SACTrainer.init() (pid=108390, ip=172.19.132.220, repr=SACTrainer)
File "/home/cocp5/anaconda3/envs/rhRL03/lib/python3.8/site-packages/gym/envs/registration.py", line 132, in spec
return self.env_specs[id]
KeyError: 'FrankaPickPlaceFixed-v0'
During handling of the above exception, another exception occurred:
ray::SACTrainer.init() (pid=108390, ip=172.19.132.220, repr=SACTrainer)
File "/home/cocp5/anaconda3/envs/rhRL03/lib/python3.8/site-packages/ray/rllib/env/utils.py", line 54, in gym_env_creator
return gym.make(env_descriptor, env_context)
File "/home/cocp5/anaconda3/envs/rhRL03/lib/python3.8/site-packages/gym/envs/registration.py", line 156, in make
return registry.make(id, kwargs)
File "/home/cocp5/anaconda3/envs/rhRL03/lib/python3.8/site-packages/gym/envs/registration.py", line 100, in make
spec = self.spec(path)
File "/home/cocp5/anaconda3/envs/rhRL03/lib/python3.8/site-packages/gym/envs/registration.py", line 142, in spec
raise error.UnregisteredEnv('No registered env with id: {}'.format(id))
gym.error.UnregisteredEnv: No registered env with id: FrankaPickPlaceFixed-v0
During handling of the above exception, another exception occurred:
ray::SACTrainer.init() (pid=108390, ip=172.19.132.220, repr=SACTrainer)
File "/home/cocp5/anaconda3/envs/rhRL03/lib/python3.8/site-packages/ray/rllib/agents/sac/sac.py", line 192, in init
super().init(*args, **kwargs)
File "/home/cocp5/anaconda3/envs/rhRL03/lib/python3.8/site-packages/ray/rllib/agents/trainer.py", line 830, in init
super().init(
File "/home/cocp5/anaconda3/envs/rhRL03/lib/python3.8/site-packages/ray/tune/trainable.py", line 149, in init
self.setup(copy.deepcopy(self.config))
File "/home/cocp5/anaconda3/envs/rhRL03/lib/python3.8/site-packages/ray/rllib/agents/trainer.py", line 911, in setup
self.workers = WorkerSet(
File "/home/cocp5/anaconda3/envs/rhRL03/lib/python3.8/site-packages/ray/rllib/evaluation/worker_set.py", line 162, in init
self._local_worker = self._make_worker(
File "/home/cocp5/anaconda3/envs/rhRL03/lib/python3.8/site-packages/ray/rllib/evaluation/worker_set.py", line 567, in _make_worker
worker = cls(
File "/home/cocp5/anaconda3/envs/rhRL03/lib/python3.8/site-packages/ray/rllib/evaluation/rollout_worker.py", line 493, in init
self.env = env_creator(copy.deepcopy(self.env_context))
File "/home/cocp5/anaconda3/envs/rhRL03/lib/python3.8/site-packages/ray/rllib/env/utils.py", line 56, in gym_env_creator
raise EnvError(ERR_MSG_INVALID_ENV_DESCRIPTOR.format(env_descriptor))
ray.rllib.utils.error.EnvError: The env string you provided ('FrankaPickPlaceFixed-v0') is:
a) Not a supported/installed environment.
b) Not a tune-registered environment creator.
c) Not a valid env class string.
Try one of the following:
a) For Atari support: pip install gym[atari] autorom[accept-rom-license].
For VizDoom support: Install VizDoom
(https://github.com/mwydmuch/ViZDoom/blob/master/doc/Building.md) and
pip install vizdoomgym.
For PyBullet support: pip install pybullet.
b) To register your custom env, do from ray import tune; tune.register('[name]', lambda cfg: [return env obj from here using cfg]).
Then in your config, do config['env'] = [name].
c) Make sure you provide a fully qualified classpath, e.g.:
ray.rllib.examples.env.repeat_after_me_env.RepeatAfterMeEnv
Hello dear Ray Team,
I hope you are doing well. I have been using Ray API ( and more specifically RLlib & Tune) for conducting Reinforcement Learning experiments with the various RL algorithms RLlib offers throughout my research journey. I used Ray to test the performance of DRL SOTA algorithms (like Rainbow DQN, PPO, DD-PPO, SAC, etc.) on the CT-graph benchmark. The CT-graph benchmark is a custom OpenAI Gym (now been upgraded to Gymnasium) environment that offers multiple RL tasks and MDP & POMDP settings. Since the CT-graph is a custom environment I have created a "env_creator()" function, where I configure and make the environment using "gym.makke()" method, and then I return the newly build environment. Next I register the specific configuration build of the CT-graph to the Ray API using the "register_env()" method, where I pass the desired name to use for the registration and then then a reference to the "env_creator" method. You can see the exact implementation in the following notebook . Following these steps I was able to register any configuration of the custom environment CT-graph to Ray API, and then use its implemented SOTA RL algorithms along with Tune to conduct experiments and test the algorithms performance.
In my new project I have to work with multiple tasks in Robotics scenarios. For these setting I have found the RoboHive API which is build on the MuJoco simulator, uses mujoco-py and mujoco python libraries and supports multi-task scenarios natively. All its environments are developed as OpenAI Gym environments so that they are easily used in RL settings. Since the environments offered by the RoboHHive API are not from the classic standard OpenAI Gym environments, but custom ones, I had followed the same steps and technique, I followed for registering and using the CT-graph benchmark, to register these custom environments to the Ray API. Despite following exactly the same steps and technique, for example for registering and using the RoboHive "FrankaPickPlaceFixed-v0" environment (which has been registered to Gym by the RoboHive API), I received the following error message when attempting to run one experiment with Tune using SAC:
2023-07-09 22:30:20,527 ERROR ray_trial_executor.py:102 -- An exception occurred when trying to stop the Ray actor:Traceback (most recent call last): File "/home/cocp5/anaconda3/envs/rhRL03/lib/python3.8/site-packages/ray/tune/ray_trial_executor.py", line 93, in post_stop_cleanup ray.get(future, timeout=0) File "/home/cocp5/anaconda3/envs/rhRL03/lib/python3.8/site-packages/ray/_private/client_mode_hook.py", line 105, in wrapper return func(*args, **kwargs) File "/home/cocp5/anaconda3/envs/rhRL03/lib/python3.8/site-packages/ray/worker.py", line 1811, in get raise value ray.exceptions.RayActorError: The actor died because of an error raised in its creation task, ray::SACTrainer.init() (pid=108390, ip=172.19.132.220, repr=SACTrainer) File "/home/cocp5/anaconda3/envs/rhRL03/lib/python3.8/site-packages/ray/rllib/agents/trainer.py", line 1035, in _init raise NotImplementedError NotImplementedError
During handling of the above exception, another exception occurred:
ray::SACTrainer.init() (pid=108390, ip=172.19.132.220, repr=SACTrainer) File "/home/cocp5/anaconda3/envs/rhRL03/lib/python3.8/site-packages/gym/envs/registration.py", line 132, in spec return self.env_specs[id] KeyError: 'FrankaPickPlaceFixed-v0'
During handling of the above exception, another exception occurred:
ray::SACTrainer.init() (pid=108390, ip=172.19.132.220, repr=SACTrainer) File "/home/cocp5/anaconda3/envs/rhRL03/lib/python3.8/site-packages/ray/rllib/env/utils.py", line 54, in gym_env_creator return gym.make(env_descriptor, env_context) File "/home/cocp5/anaconda3/envs/rhRL03/lib/python3.8/site-packages/gym/envs/registration.py", line 156, in make return registry.make(id, kwargs) File "/home/cocp5/anaconda3/envs/rhRL03/lib/python3.8/site-packages/gym/envs/registration.py", line 100, in make spec = self.spec(path) File "/home/cocp5/anaconda3/envs/rhRL03/lib/python3.8/site-packages/gym/envs/registration.py", line 142, in spec raise error.UnregisteredEnv('No registered env with id: {}'.format(id)) gym.error.UnregisteredEnv: No registered env with id: FrankaPickPlaceFixed-v0
During handling of the above exception, another exception occurred:
ray::SACTrainer.init() (pid=108390, ip=172.19.132.220, repr=SACTrainer) File "/home/cocp5/anaconda3/envs/rhRL03/lib/python3.8/site-packages/ray/rllib/agents/sac/sac.py", line 192, in init super().init(*args, **kwargs) File "/home/cocp5/anaconda3/envs/rhRL03/lib/python3.8/site-packages/ray/rllib/agents/trainer.py", line 830, in init super().init( File "/home/cocp5/anaconda3/envs/rhRL03/lib/python3.8/site-packages/ray/tune/trainable.py", line 149, in init self.setup(copy.deepcopy(self.config)) File "/home/cocp5/anaconda3/envs/rhRL03/lib/python3.8/site-packages/ray/rllib/agents/trainer.py", line 911, in setup self.workers = WorkerSet( File "/home/cocp5/anaconda3/envs/rhRL03/lib/python3.8/site-packages/ray/rllib/evaluation/worker_set.py", line 162, in init self._local_worker = self._make_worker( File "/home/cocp5/anaconda3/envs/rhRL03/lib/python3.8/site-packages/ray/rllib/evaluation/worker_set.py", line 567, in _make_worker worker = cls( File "/home/cocp5/anaconda3/envs/rhRL03/lib/python3.8/site-packages/ray/rllib/evaluation/rollout_worker.py", line 493, in init self.env = env_creator(copy.deepcopy(self.env_context)) File "/home/cocp5/anaconda3/envs/rhRL03/lib/python3.8/site-packages/ray/rllib/env/utils.py", line 56, in gym_env_creator raise EnvError(ERR_MSG_INVALID_ENV_DESCRIPTOR.format(env_descriptor)) ray.rllib.utils.error.EnvError: The env string you provided ('FrankaPickPlaceFixed-v0') is: a) Not a supported/installed environment. b) Not a tune-registered environment creator. c) Not a valid env class string.
Try one of the following: a) For Atari support:
pip install gym[atari] autorom[accept-rom-license]
. For VizDoom support: Install VizDoom (https://github.com/mwydmuch/ViZDoom/blob/master/doc/Building.md) andpip install vizdoomgym
. For PyBullet support:pip install pybullet
. b) To register your custom env, dofrom ray import tune; tune.register('[name]', lambda cfg: [return env obj from here using cfg])
. Then in your config, doconfig['env'] = [name]
. c) Make sure you provide a fully qualified classpath, e.g.:ray.rllib.examples.env.repeat_after_me_env.RepeatAfterMeEnv
My "env_creator()" method is the following:
def env_creator(env_config={}): import robohive env = gym.make('FrankaPickPlaceFixed-v0') env.reset() return env
I have been using Ray version 1.12.1 for testing the code.
What is it causing the issue and how could I fix it?
Thank you very much in advance for your valuable help!!!
I am always at your disposal for any further queries regarding my code and my environment setup.
Kind regards,
Christos Peridis