UnityToGymWrapper doesn't work for Visual observation

[yoheigon]

Describe the bug We are trying to use visual observation for training. Our env is base on this sample https://github.com/Unity-Technologies/ml-agents/blob/master/docs/Learning-Environment-Create-New.md It worked for local training with unity editor but the env doesn't return appropriate observation via UnityToGymWrapper.

To Reproduce Steps to reproduce the behavior:

1. Add camera sensor on unity editor

   

2. Build a binary file

3. Create a gym env

   
   from gym_unity.envs import UnityToGymWrapper
   from mlagents_envs.environment import UnityEnvironment
   from mlagents_envs.side_channel.environment_parameters_channel import EnvironmentParametersChannel

channel = EnvironmentParametersChannel() channel.set_float_parameter("simulation_mode", 1.0) unity_env = UnityEnvironment('RollerBall.x86_64', no_graphics=True, side_channels=[channel], worker_id=15, additional_args=['-logFile', 'unity.log']) env = UnityToGymWrapper(unity_env, uint8_visual=True)

4. Get a first observation

from PIL import Image import matplotlib.pyplot as plt import numpy as np %matplotlib inline state = env.reset() plt.imshow(state) plt.show()

<img width="686" alt="Screen Shot 2021-01-08 at 5 25 05 PM" src="https://user-images.githubusercontent.com/25813762/103998432-5923b000-51df-11eb-8d0c-2524bf40cff0.png">

**Console logs / stack traces**
Please wrap in [triple backticks (```)](https://help.github.com/en/articles/creating-and-highlighting-code-blocks) to make it easier to read.

**Screenshots**
If applicable, add screenshots to help explain your problem.

**Environment (please complete the following information):**
- Unity Version: [e.g. Unity 2020.1f1]
- OS + version: [e.g. Windows 10]
- _ML-Agents version_: mlagents-envs==0.18.0, gym_unity==0.18.0
- _Torch version_: (you can run `pip3 show torch` to get this)
- _Environment_: (which example environment you used to reproduce the error)

**NOTE:** We are unable to help reproduce bugs with custom environments.  Please attempt to reproduce your issue with one of the example environments, or provide a minimal patch to one of the environments needed to reproduce the issue.

[Ihsees]

Hello yoheigon,

if I remember correctly visual observations are only supported when a (game-)window is rendered.

By using "no_graphics = true" you are explicitly telling unity NOT to render a game-window.

Can you try with "no_graphics=false" and see if that fixes your issue?

[andrewcoh]

Hi @yoheigon

@Ihsees is correct in pointing out that no_graphics=True will prevent visual observations from being sent correctly. Please set this flag to False.

[yoheigon]

Thank you for the info. Seems we need e.g. Xvfb for no_graphics=False. Are there any options that we can get visual observation as e.g. an array without rendering to a game-window? We would like to run on docker images and can't set a window system.

[andrewcoh]

You can use xvfb with ml-agents. See this forum post and github issue

[yoheigon]

Thank you for the info

[github-actions[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.