Tracking cameras?

[pedronahum]

Hi,

Each time that I visualize the hooper-hop or walker-walk domains-tasks, for example, I notice that the agents easily escape the area covered by the camera. Although this is not really an issue if you want to learn from state features, it can be a serious issue if you want to learn from pixels. Is this expected?

Somehow gym circumvents this issue by defining a track camera inside the body. But what I see in the dm control xmls is consistent with the technical document, where you highlight the use of camera 0th. And was wondering if you made other adjustments to the xml to ensure you keep track of the "agent".

Thanks,

[yuvaltassa]

I suspect you somehow might be rendering from the static floating camera?

All the cameras e.g. the hopper's have the mode="trackcom" flag set, which means they track the centre-of-mass of their parent body's subtree. (Yes, the camera is in the world body, but if there is only one tree in the scene (here, the hopper) then the worldbody's CoM and the hopper subtree CoM are identical.)

[pedronahum]

Hi @yuvaltassa,

Thanks for getting back to me.

I am wrapping the environment in the pixels wrapper [pixels.Wrapper(env, pixels_only=False)]. Then I extract from each observation the pixels (ie, time_step.observation["pixels"]). Finally, to visualize the domain / task, I just build a gif using movie-py's ImageSequenceClip using all the extracted pixels.

Do I need to specify to the pixels wrapper some additional "render_kwargs" to obtain the default or desired behavior?

Thanks,

[yotam]

Hi @pedronahum yes something like this should work:

from dm_control import suite
from dm_control.suite.wrappers import pixels

env = suite.load('hopper', 'hop')

wrapped_env = pixels.Wrapper(env, render_kwargs={'camera_id': 'cam0'})

[pedronahum]

Hi @yotam

Many thanks for the tip!

I did get an error when I named the camera_id "env0":

dm_control.mujoco.wrapper.core.Error: Object of type b'camera' with name 'env0' does not exist.

But a small change to "0" made the trick!

pixels.Wrapper(env, pixels_only=False, render_kwargs={'camera_id': 0})

Maybe would be a good idea to set the default value of render_kwargs to {'camera_id': 0} to avoid more random questions like mine :o)

Again, greatly appreciate your help!

[yotam]

Oh, sorry I should have written 'cam0' instead of 'env0'. I'll update my snippet.

To specify a camera, you can either use an integer corresponding to the order of the cameras in the XML (starting from 0), or a string corresponding to the name. By default, we render from MuJoCo's default camera (not explicitly defined in the XML), which has an index of -`.

[pedronahum]

Hi @yotam

No worries. You, nevertheless, point me in the right direction.

I realized a little bit later that you have a nice example showing how to do this (mocap_demo.py)[https://github.com/deepmind/dm_control/blob/master/dm_control/suite/demos/mocap_demo.py]

Again, thanks for your help and for the nice software that is keeping me busy during the evenings :o)