Dreamer error: frame_skip unexpected

[janetwise]

Env: Ray 2.2.0, gym: 0.21.0 Ran the sample dreamer code below listed on Ray doc site: from ray.rllib.algorithms.dreamer import DreamerConfig config = DreamerConfig().training(gamma=0.9, lr=0.01)
config = config.resources(num_gpus=1)
config = config.rollouts(num_rollout_workers=0)
config = config.framework("torch") print(config.to_dict())

Build a Algorithm object from the config and run 1 training iteration.

algo = config.build(env="CartPole-v1")  
algo.train()  

Got the following error: miniconda3/envs/ray/lib/python3.8/site-packages/gym/envs/registration.py", line 90, in make env = cls(**_kwargs) TypeError: init() got an unexpected keyword argument 'frame_skip'.

I tried different gym version 0.22.0 but the error persisted. Any help is appreciated.

[stillonearth]

You can patch this by removing frame_skip key here: https://github.com/ray-project/ray/blob/master/rllib/env/utils.py#L144

        else:
            print("---")
            print(env_context)
            del env_context["frame_skip"]
            print("---")
            env = gym.make(env_descriptor, **env_context)
        # If we are dealing with an old gym-env API, use the prov

But then an actual error pops up:

  File "/home/__/.local/lib/python3.10/site-packages/torch/nn/modules/conv.py", line 463, in forward
    return self._conv_forward(input, self.weight, self.bias)
  File "/home/__/.local/lib/python3.10/site-packages/torch/nn/modules/conv.py", line 459, in _conv_forward
    return F.conv2d(input, weight, bias, self.stride,
RuntimeError: Given groups=1, weight of size [32, 3, 4, 4], expected input[1, 1, 33, 27] to have 3 channels, but got 1 channels instead

I'm still interested running a listed example.

[stale[bot]]

Hi, I'm a bot from the Ray team :)

To help human contributors to focus on more relevant issues, I will automatically add the stale label to issues that have had no activity for more than 4 months.

If there is no further activity in the 14 days, the issue will be closed!

• If you'd like to keep the issue open, just leave any comment, and the stale label will be removed!
• If you'd like to get more attention to the issue, please tag one of Ray's contributors.

You can always ask for help on our discussion forum or Ray's public slack channel.

[mobeets]

I'm having the same issue. I think the problem might be this:

From the docs page, "Dreamer is an image-only model-based RL method". (emphasis mine)

So my guess is we are all getting this error when trying to use Dreamer with an environment that does not provide images.

[stillonearth]

Right, Dreamer imagines a trajectory under the hood. I was trying to evaluate this versus my home-baked implementation on an environment that supports rendering. If it's still relevant I can try fixing this.

[mobeets]

By fixing do you mean modifying to work without image observations, or do you mean something else?

[stillonearth]

After checking I can confirm that https://github.com/ray-project/ray/issues/32508#issuecomment-1439088056 is no longer relevant, as long as you are using recent (i've checked on 96a6aea3d0ac550e3e9967390a12c37232300c78) version of Gymnasium with v5 of environments that support frame_skip.

Dreamer expects pixel observations of an environment, so potentially PixelObservationWrapper is required, and then "pixels" instead of "obs" should be fed to environment. here.

There's also DreamerV3 on the main, but no documentation on site.

[stillonearth]

This code will change observation space to screen pixels

import gymnasium as gym
import numpy as np

from ray.rllib.algorithms.dreamer.dreamer import DreamerConfig
from supersuit.generic_wrappers import resize_v1

class NormalizedImageEnv(gym.ObservationWrapper):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.observation_space = self.observation_space['pixels']
        self.observation_space = gym.spaces.Box(
            -1.0,
            1.0,
            shape=self.observation_space.shape,
            dtype=np.float32,
        )

    def observation(self, observation):
        observation = observation['pixels']
        obs = (observation.astype(np.float32) / 128.0) - 1.0
        return obs

def create_env(*args, **kwargs):
    orig_env = gym.make("HalfCheetah-v5", render_mode='rgb_array', **kwargs)
    pixel_env = gym.wrappers.PixelObservationWrapper(orig_env)
    return resize_v1(NormalizedImageEnv(pixel_env), x_size=64, y_size=64)

gym.register("cheetah-pixels", entry_point=create_env)
config = DreamerConfig().training(gamma=0.9, lr=0.01).resources(num_gpus=0).rollouts(num_rollout_workers=0)  
config.disable_env_checking = True
algo = config.build(env="cheetah-pixels")

But dreamer will still fail executing nn model.