Build not showing agent

NicolasKuske commented 3 months ago

Dear all,

when I run your obsta.exe build (windows), I can play the game and see everything. However, when the build window opens after I instantiated it in a UnityEnvironment it remains blank for all times except for the one frame when I do an img.show call. Is that normal and is there any way to see the agent in real time doing its thing?

Best wishes

Nicolas

MarcoMeter commented 3 months ago

Hi @NicolasKuske

can you share the code that you use to reproduce this issue?

NicolasKuske commented 3 months ago

Hi Marco,

the code is the very beginning from the basic_usage notebook you provide.

from obstacle_tower_env import ObstacleTowerEnv
%matplotlib inline
from matplotlib import pyplot as plt

env = ObstacleTowerEnv(retro=False, realtime_mode=True)

Outputs: INFO:mlagents_envs.environment:Connected to Unity environment with package version 1.2.0-preview and communication version 1.0.0 INFO:mlagents_envs.environment:Connected new brain: ObstacleTowerAgent-v4.1?team=0 INFO:gym_unity:1 agents within environment.

And opens the pop up window.

The conda environment dependencies are:

asttokens=2.0.5=pyhd3eb1b0_0
backcall=0.2.0=pyhd3eb1b0_0
ca-certificates=2024.3.11=haa95532_0
colorama=0.4.6=py38haa95532_0
comm=0.2.1=py38haa95532_0
debugpy=1.6.7=py38hd77b12b_0
decorator=5.1.1=pyhd3eb1b0_0
executing=0.8.3=pyhd3eb1b0_0
importlib_metadata=7.0.1=hd3eb1b0_0
ipykernel=6.28.0=py38haa95532_0
ipython=8.12.2=py38haa95532_0
jedi=0.18.1=py38haa95532_1
jupyter_client=8.6.0=py38haa95532_0
jupyter_core=5.5.0=py38haa95532_0
libffi=3.4.4=hd77b12b_1
libsodium=1.0.18=h62dcd97_0
matplotlib-inline=0.1.6=py38haa95532_0
nest-asyncio=1.6.0=py38haa95532_0
openssl=3.0.13=h2bbff1b_1
packaging=23.2=py38haa95532_0
parso=0.8.3=pyhd3eb1b0_0
pickleshare=0.7.5=pyhd3eb1b0_1003
pip=24.0=py38haa95532_0
platformdirs=3.10.0=py38haa95532_0
prompt-toolkit=3.0.43=py38haa95532_0
psutil=5.9.0=py38h2bbff1b_0
pure_eval=0.2.2=pyhd3eb1b0_0
pygments=2.15.1=py38haa95532_1
python=3.8.19=h1aa4202_0
python-dateutil=2.9.0post0=py38haa95532_0
pywin32=305=py38h2bbff1b_0
pyzmq=25.1.2=py38hd77b12b_0
setuptools=69.5.1=py38haa95532_0
six=1.16.0=pyhd3eb1b0_1
sqlite=3.45.3=h2bbff1b_0
stack_data=0.2.0=pyhd3eb1b0_0
tornado=6.3.3=py38h2bbff1b_0
traitlets=5.7.1=py38haa95532_0
typing_extensions=4.11.0=py38haa95532_0
vc=14.2=h21ff451_1
vs2015_runtime=14.27.29016=h5e58377_2
wcwidth=0.2.5=pyhd3eb1b0_0
wheel=0.43.0=py38haa95532_0
zeromq=4.3.5=hd77b12b_0
pip:
- absl-py==2.1.0
- anyio==3.7.1
- argon2-cffi==23.1.0
- argon2-cffi-bindings==21.2.0
- astunparse==1.6.3
- attrs==23.2.0
- beautifulsoup4==4.12.3
- bleach==6.1.0
- cachetools==5.3.3
- cattrs==1.5.0
- certifi==2024.2.2
- cffi==1.16.0
- charset-normalizer==3.3.2
- cloudpickle==3.0.0
- contourpy==1.1.1
- cycler==0.12.1
- defusedxml==0.7.1
- exceptiongroup==1.2.1
- fastjsonschema==2.19.1
- filelock==3.14.0
- flatbuffers==2.0.7
- fonttools==4.51.0
- gast==0.4.0
- google-auth==2.29.0
- google-auth-oauthlib==1.0.0
- google-pasta==0.2.0
- grpcio==1.63.0
- gym==0.26.2
- gym-notices==0.0.8
- h5py==3.11.0
- idna==3.7
- importlib-metadata==7.1.0
- importlib-resources==6.4.0
- jinja2==3.1.4
- jsonschema==4.22.0
- jsonschema-specifications==2023.12.1
- jupyter-server==1.24.0
- jupyterlab-pygments==0.3.0
- keras==2.7.0
- keras-preprocessing==1.1.2
- kiwisolver==1.4.5
- libclang==18.1.1
- markdown==3.6
- markupsafe==2.1.5
- matplotlib==3.5.0
- mistune==3.0.2
- mlagents==0.18.1
- mlagents-envs==0.18.1
- nbclient==0.10.0
- nbconvert==7.16.4
- nbformat==5.10.4
- numpy==1.18.5
- oauthlib==3.2.2
- opt-einsum==3.3.0
- pandocfilters==1.5.1
- pettingzoo==1.15.0
- pillow==10.3.0
- pkgutil-resolve-name==1.3.10
- prometheus-client==0.20.0
- protobuf==3.19.6
- pyasn1==0.6.0
- pyasn1-modules==0.4.0
- pycparser==2.22
- pyparsing==3.1.2
- pypiwin32==223
- pywinpty==2.0.13
- pyyaml==6.0.1
- referencing==0.35.1
- requests==2.31.0
- requests-oauthlib==2.0.0
- rpds-py==0.18.1
- rsa==4.9
- send2trash==1.8.3
- setuptools-scm==8.1.0
- sniffio==1.3.1
- soupsieve==2.5
- tensorboard==2.14.0
- tensorboard-data-server==0.7.2
- tensorflow==2.7.4
- tensorflow-estimator==2.7.0
- tensorflow-io-gcs-filesystem==0.31.0
- termcolor==2.4.0
- terminado==0.18.1
- tinycss2==1.3.0
- tomli==2.0.1
- urllib3==2.2.1
- webencodings==0.5.1
- websocket-client==1.8.0
- werkzeug==3.0.3
- wrapt==1.16.0
- zipp==3.18.1

NicolasKuske commented 3 months ago

Also env.observation_space independent of retro=False or retro=True always returns Tuple(Box(0.0, 1.0, (84, 84, 3), float32), Discrete(5), Box(0.0, 3.4028235e+38, (1,), float32), Discrete(9999))

Furthermore the cloned repository is only 2.5 MB size. It seems as if the call

env = ObstacleTowerEnv(retro=False, realtime_mode=True)

never initializes the download of the tower.exe build.

I downloaded the file manually and it is still in my downloads folder. My guess is that the cloned repository and the downloaded build never interact...

MarcoMeter commented 3 months ago

The best way to init the environment is to specify the file path:

env = ObstacleTowerEnv(environment_filename="path to ...")

I wouldn't rely on the automatic download as you might need to change permission on the executable on Linux systems.

As long as the returned visual observations of the reset and step functions are correct, then there is nothing to worry about. Don't care about the actual window, which is opened by the Unity build itself.

If the instance of the build and python don't connect, you should receive a timeout on the python side.

NicolasKuske commented 3 months ago

thank you, although I believe in general it should be possible to have the window display what the agent is currently seeing and doing. I mean that seems to be the reason for the realtime_mode option.

Unity-Technologies / obstacle-tower-env

Build not showing agent #129