Custom RL Environment Issue with GymNE

[pkrobinette]

When making a problem like this:

problem = GymNE(
    env = env,
    network="Linear(obs_length, act_length)",
    observation_normalization=True,
    num_episodes=20, # each solution evaluated on 20 episodes
    num_actors="max",
)

where env is a gym environment made with gym.make("CartPole-v1"), I get the following error.

I modified the GymNE file as shown below, and it fixed the problem. There might be a better way to do it, but this works.

[pkrobinette]

That actually did not work :) I updated the def _make_env() function instead as follows, and that worked.

[NaturalGradient]

Hi @pkrobinette . Thanks for raising this issue! I'll talk to @engintoklu to see if the fix you presented here should be the general default behavior. As far as I can tell, the assumption is 'if env is not callable, then it's already an environment instance, so return it'. My only concern is that maybe this obfuscates some other error through the introduction of the try ... except. Maybe we can check the type of env instead to resolve this.

[engintoklu]

Hello @pkrobinette,

Thank you for your feedback, and for using EvoTorch!

Looking at your modification, I think I understand the same thing with @NaturalGradient: your aim is to see whether or not the environment is callable (handled via try in your implementation). If it is not callable (the except case), then the environment itself is returned.

While I understand the underlying idea, I would like to point out that the error you receive by the unmodified version of EvoTorch is intentional. We have designed the API of GymNE in such a way that the argument env expects not a gym.Env instance, but one of these:

• Environment id as string
• A function which makes and returns the environment
• A class which inherits from gym.Env

The motivation of this interface is to make sure that GymNE knows how to re-create the target environment. This allows the parallelization procedures of GymNE to go smoothly, without having to worry about the picklability of the environment (indeed, the environment could have an underlying simulator which is not picklable/clonable). In more details, when GymNE is transferred to a remote actor for parallelized evaluation (with num_actors=N, N being "max" or an integer bigger than 1), the environment is re-created there.

I think that you wish to eventually work with a custom environment. May I suggest the following:

def make_custom_env() -> gym.Env:
    # User's preferred way of instantiating the custom environment goes here
    # my_new_env = ...
    my_new_env = gym.make("CartPole-v1")
    return my_new_env

problem = GymNE(
    env=make_custom_env,
    ...
)

Would you like to try it like this? I hope it will work for your case?

Thanks & happy coding!

[Higgcz]

Let's close for now. Feel free to reopen if you have further questions.

[valhallen13]

Hello @engintoklu ,

I seem to be having the same issue as pkrobinette. The fix you provided does not work in my situation as I get this error:

  File "/home/xxx/Desktop/yyy/venv/lib/python3.8/site-packages/evotorch/core.py", line 170, in evaluate_batch_piece
    return piece_index, self.evaluate_batch(batch_piece)
  File "/home/xxx/Desktop/yyy/venv/lib/python3.8/site-packages/evotorch/core.py", line 154, in evaluate_batch
    self._problem.evaluate(solution_batch)
  File "/home/xxx/Desktop/yyy/venv/lib/python3.8/site-packages/evotorch/core.py", line 2546, in evaluate
    self._start_preparations()
  File "/home/xxx/Desktop/yyy/venv/lib/python3.8/site-packages/evotorch/core.py", line 2466, in _start_preparations
    self._prepare()
  File "/home/xxx/Desktop/yyy/venv/lib/python3.8/site-packages/evotorch/neuroevolution/gymne.py", line 355, in _prepare
    self._get_env()
  File "/home/xxx/Desktop/yyy/venv/lib/python3.8/site-packages/evotorch/neuroevolution/gymne.py", line 325, in _get_env
    self._env = self._instantiate_new_env()
  File "/home/xxx/Desktop/yyy/venv/lib/python3.8/site-packages/evotorch/neuroevolution/gymne.py", line 318, in _instantiate_new_env
    env = _make_env(self._env_maker, **env_config)
  File "/home/xxx/Desktop/yyy/venv/lib/python3.8/site-packages/evotorch/neuroevolution/gymne.py", line 61, in _make_env
    return env(**kwargs)
  File "/home/xxx/Desktop/yyy/main.py", line 14, in make_custom_env
    my_new_env = gym.make("CustomEnvironment-v0", render_mode="human")
  File "/home/xxx/Desktop/yyy/venv/lib/python3.8/site-packages/gymnasium/envs/registration.py", line 741, in make
    env_spec = _find_spec(id)
  File "/home/xxx/Desktop/yyy/venv/lib/python3.8/site-packages/gymnasium/envs/registration.py", line 527, in _find_spec
    _check_version_exists(ns, name, version)
  File "/home/xxx/Desktop/yyy/venv/lib/python3.8/site-packages/gymnasium/envs/registration.py", line 393, in _check_version_exists
    _check_name_exists(ns, name)
  File "/home/xxx/Desktop/yyy/venv/lib/python3.8/site-packages/gymnasium/envs/registration.py", line 370, in _check_name_exists
    raise error.NameNotFound(
gymnasium.error.NameNotFound: Environment `CustomEnvironment` doesn't exist. Did you mean: `GymV26Environment`?

pkrobinette's fix seems to work for me however. Any idea how I could fix this without having to modify the _make_env function?

[engintoklu]

Hello valhallen13,

Thank you for trying EvoTorch!

From the error message, it looks like gymnasium cannot find your custom environment in its default registry, most probably because your custom environment needs to be imported from its module. Perhaps you could try referring to your environment with the module name prefix, like shown below?

problem = GymNE(
    env="name_of_your_module:CustomEnvironment-v0",
    ...
)

If that doesn't work, then you could try writing a helper function and providing that function as env instead of the environment ID:

import gymnasium as gym

...

def make_custom_env() -> gym.Env:
    # Local imports for making sure that your custom environment is registered
    import name_of_your_module

    my_new_env = gym.make("CustomEnvironment-v0")
    return my_new_env

problem = GymNE(
    env=make_custom_env,
    ...
)

I hope this helps! Feel free to let us know whether or not these suggestions worked.

[valhallen13]

Very interesting, it seems to work when I import the environment locally as you mentioned, but not when the environment is imported at the top of the file.

Thank you for the help! Your solution worked.