Training On Camera Data

[nmharmon8]

Hey,

While using the vision env, if I call getImage it returns an array of zeros. After reading through the flightmare documentation, it seems this is because the vision env dose not actually setup Unity?

setUnity(unity_render_);
connectUnity();
updateUnity(frame_id);

Dose this seem correct? If so then is there a way to train a server with no display? My GPU box is a remote box and I can't forward the Unity GUI. Is there a headless mode to Unity that I am missing?

[yun-long]

make sure that the unity render is on and RGB camera is yes in the config file

config.yaml

[nmharmon8]

I believe I have those values set correctly.

rgb_camera:
  on: yes 
  t_BC: [0.0, 0.0, 0.3] # translational vector of the camera with repect to the body frame
  r_BC: [0.0, 0.0, -90] # rotational angle (roll, pitch, yaw) of the camera in degree.
  channels: 3
  width: 320 
  height: 240
  fov: 70.0
  enable_depth: yes 
  enable_segmentation: no
  enable_opticalflow: no

unity:
  scene_id: 0 # 0 warehouse, 1 street, 2 forest, 3 wasteland
  render: yes 

In the vision_env.cpp the render filed is read into unity_render_

// environment
  if (cfg["unity"]) {
    unity_render_ = cfg["unity"]["render"].as<bool>();
    scene_id_ = cfg["unity"]["scene_id"].as<SceneID>();
  }

unity_render is only used in the setUnity function. But setUnity is never called.

[nmharmon8]

What I really am asking is do you need to have RPG_Flightmare.x86_64 running and connect to it with the unity bridge in order to get camera data?

That is what I think has to happen based off reading the flightmare docs. I don't have physical access to my GPU box, so I would need to run RPG_Flightmare.x86_64 on a remote server with no display. According to the flighmare docs sound like it might be possible, but it currently has issues when running headless.

[nmharmon8]

I got RPG_Flightmare.x86_64 running on the remote sever using Xvfb. Still not getting images. I believe it is because nothing calls the updateUnity() function.

Seems like the stepUnity() not yet implemented.

Just to make sure I am not missing something. There is no working example of a image based environments for RL using flightmare. The only working example is using explicit object location information?

[nmharmon8]

Got Images working by conneting the unity bridge and then calling:

unity_bridge_ptr_->getRender(frame_id);
unity_bridge_ptr_->handleOutput(frame_id);

In the step function. This make it so when I call getImage() on the env it returns an actual image. However this is incredibly slow (1 fps). It could be the virtualization of x11 with Xvfb. Or am I just missing the point of how the simulator is supposed to work?

[IanYangChina]

@nmharmon8 @yun-long Hi! I'm doing image RL as well, met the same problem. Any update on this issue? Is possible to render image observations on headless mode?

[nmharmon8]

I'm afraid I don't have any updates as of yet.

[IanYangChina]

Trying to follow up again... So I tried to run the standalone RPG_Flightmare.x86_64 with -batchmode. This was supposed to make it headless. I did get images after calling updateUnity(). However, the app still pops up a window with black screen (on Linux). Is there a way to remove the screen or I just have to have that black screen there?

[UltronAI]

Hi @yun-long . Thanks for your great work on the drone simulator. I would like to know if there are any updates on this issue? RL training, especially image-based RL, is super time- and computation-consuming, so it will be limited if it is only available on desktop computers. Parallel and headless training on remote servers would be greater. Do you have a plan to support this mode? I would appreciate any updates on this issue! Thanks!