Relation between build/viewer and the observations output from sim.step()

[anoopsonar30]

So I've been running simulations based off of the Jupyter notebook example habitat-sim-demo.ipynb. I'm essentially compiling the separate RGB frames that you get from observations['color_sensor'] each time step and constructing a video. It's a little unclear how I could run simulations using build/viewer instead since the viewer renders at a higher resolution and leads to better videos.

[mathfac]

Hi @anoopsonar30, To change resolution of your video you can use simulator sensor settings inside Jupyter notebook. You can rebuild the simulator with head attached and run same code from Jupyter notebook in command line and watch rendered navigation trajectory.

[erikwijmans]

Increasing the resolution of the sensor (or adding a second, higher resolution, color sensor) is the way to go. We don't support replaying navigation trajectories in the view currently.

To clarify, you do not need to build in headed mode to increase the resolution of images inside the jupyter notebook.

[anoopsonar30]

Thank you for your reply.

@erikwijmans I did notice the option to adjust the resolution. When I meant better videos, I was referring to being able to replay navigation trajectories in the build/viewer since they look much cleaner than the RGB videos without all the stitching artifacts. It's unfortunate that the feature is not yet supported. I'm curious, what's the potential use case for the build/viewer right now? I really like it because of the low latency however I'm not sure how to use it productively in running and recording simulations.

@mathfac what did you mean by "watch rendered navigation trajectory" ?

[anoopsonar30]

I also noticed that I can now test physics however does this only work inside build/viewer or also in headless mode?

[erikwijmans]

Also physics works in headless mode. We are still very much building out how to control it in headless mode tho, so if you have a use case, we are happy to work with you on getting that to work smoothly.

What do you mean by "When I meant better videos, I was referring to being able to replay navigation trajectories in the build/viewer since they look much cleaner than the RGB videos without all the stitching artifacts." Habitat-api has a function take a list of images and makes a video (images_to_video) and those end up looking great IMO (https://www.youtube.com/watch?v=wNDcvomBRt8)

[anoopsonar30]

Ah that would be wonderful. I'm trying to simulate a quadrotor. Although the lower level dynamics can all be abstracted out, the agent just needs to be able to fly over obstacles. I remember this wasn't possible a few months ago because the robot had to be in contact with the floor due to some sort of navmesh requirement and couldn't detect collisions accurately if I forced a "move_up" command. In simulation this would just be like a small sphere that I can control with higher level commands (like move forward, turn left, turn right, etc) that can fly over say a table or bed in the environment.

My statement was just biased due to the one specific environment skokloster-castle that looks considerably prettier than others haha. My bad.

If you look at the image above you'll notice the slight gap in the mesh. This would be fine if it was solely a rendering issue however when the agent travels over that, it gets detected as a collision although there really is no collision. I'm trying to simulate robots on long time horizons and this has been a bit of a problem. And it's not an isolated incident and can be found in multiple environments (This one in particular is Portal.glb)

Also I noticed that in the youtube video the robot seems to be travelling quite a long distance quite smoothly. Would you be willing to open source that? Is that just one of the examples from habitat-api?

[erikwijmans]

Ah that would be wonderful. I'm trying to simulate a quadrotor. Although the lower level dynamics can all be abstracted out, the agent just needs to be able to fly over obstacles. I remember this wasn't possible a few months ago because the robot had to be in contact with the floor due to some sort of navmesh requirement and couldn't detect collisions accurately if I forced a "move_up" command. In simulation this would just be like a small sphere that I can control with higher level commands (like move forward, turn left, turn right, etc) that can fly over say a table or bed in the environment.

CC @aclegg3 Can we make a quadrotor?

[erikwijmans]

Also I noticed that in the youtube video the robot seems to be travelling quite a long distance quite smoothly. Would you be willing to open source that? Is that just one of the examples from habitat-api?

It is just the ppo-trainer's video mode (https://github.com/facebookresearch/habitat-api/blob/master/habitat_baselines/rl/ppo/ppo_trainer.py#L513-L530) with some extra goodies (text names and the radar thing for the goal) and I made everything higher resolution. It's been on my todo list to add those habitat-api, but I haven't gotten to it.

[aclegg3]

We can absolutely approximate a quadrotor with a collision sphere and force based control. You could enable physics, load your object, and then apply forces to it to counter gravity and for control (you could compute forces applied at 4 corner offsets in the local frame for a more accurate model). You could then manually sync the agent and/or camera state with the object's position to acquire observations.

My TODO list includes sphere primitives for collision which in this case would likely be necessary for training speed (mesh/mesh collision will be quite slow when close to walls). This might make a good physics example/tutorial task. 💭

[erikwijmans]

You could then manually sync the agent and/or camera state with the object's position to acquire observations.

The agent's body is just a scene node, so we could probably control that directly?

[aclegg3]

The agent's body is just a scene node, so we could probably control that directly?

Yes, that is what I would do.

[anoopsonar30]

Great, Thanks! @aclegg3 How far down your to-do list is this particular task? I completely understand that you might have other priorities. In that case I could get started on this myself with some directions from you on what particular classes in the code base deal with the spherical colliders etc. I've been trying to get my simulations to work since quite some time but Gibson doesn't scale for long time horizons because of large training times (GibsonV2 still is somewhat unstable and I'm not sure if the new dataset is ready) and it would be clutch if I could get Habitat to work now that it has Physics support.

[anoopsonar30]

@erikwijmans what are your thoughts on the mesh gap collision issue?

You could reproduce it by using build/viewer with Portal.glb and then once you're on top/near the mesh gap you'll see that the distance to the closest obstacle being printed in the terminal becomes zero.

[anoopsonar30]

Here's another example from Circleville where the gap on the floor is all black. The same applies for the walls on the left side. Is there a way to fix this? I've been getting really bad training performance due to this. If I remember correctly Gibson uses some sort of neural network to fill in these gaps, is there something similar in Habitat?

[erikwijmans]

what are your thoughts on the mesh gap collision issue?

Is what it is, the Gibson dataset is far from perfect, it has holes, that is the downside of working with real reconstructions. There isn't a way I know of to fix this. MP3D is definitely better from a navigation mesh stand-point btw, but still not perfect. Replica is very very good in this regard, but it is still quite small.

The NN used in Gibson fills holes in just the RGB observations returned to the agent, not in the mesh.

[anoopsonar30]

Hmm. Interesting. Well if @aclegg3 and I can get the drone simulation to work that should still be fine for now since I don't have to worry about the floor in that case.

[mathfac]

You could reproduce it by using build/viewer with Portal.glb and then once you're on top/near the mesh gap you'll see that the distance to the closest obstacle being printed in the terminal becomes zero.

@anoopsonar30, what is your concern: a visual appearance of the hole or how it's affecting collisions in Habitat? If the latter, can you describe what would be a right behavior from your perspective?

@anoopsonar30, are you requirements strict regarding sphere shape or a collision bounding box will be also suitable for the experiments?

[anoopsonar30]

@mathfac It's the effect on the collisions that's the main concern. The ideal behavior would be if we could fix the holes in the meshes but since that may not be easy to do in the short term, another quick fix could be - if the agent walks over a hole on the floor then it does not get detected as a collision and the hole is essentially just like a flat surface. The holes on the walls don't matter as much since the walls get detected as obstacles on the agent's depth sensor. However this is not true for the holes on the floor and the agent walks straight into those. Or another fix would be to disable collisions with the floor altogether if that's possible. Although this hole problem wouldn't be relevant in the context of a quadrotor since those are always a certain height above the floor.

@mathfac No it does not have to be a sphere. Sorry if it seemed that way. I just used sphere as an example. It could pretty much just be a box collider. The important criterion is that it should be able to fly over obstacles and still detect collisions accurately which wasn't possible before the physics engine was introduced due to some navmesh restriction.

I need to submit these simulation results for a paper haha would be grateful if we could work this out together soon @erikwijmans @aclegg3

[aclegg3]

@anoopsonar30 I've created a branch quadrotor_prototype with bounding box collision, discrete contact checking, and an example script to get you started.

Just install with --bullet to enable Bullet physics and run python examples/quadrotor_prototype.py --control_mode=DYNAMIC (or --control_mode=KINEMATIC) to run the script and generate some video (assuming you are on a Unix system and you'll need ffmepg installed, otherwise you will need to comment some os.system() commands).

You can replace the cheezit box placeholder object with your own object mesh by editing the "render mesh" path in data/objects/cheezit.phys_properties.json or adding your own <name>.phys_properties.json and referencing it in data/default.phys_scene_config.json. Otherwise you will need to download our example object dataset for the default mesh.

[anoopsonar30]

Thanks a lot! You're a legend :) How did you choose the initial translation (the np.array that you hardcoded)? and the position of the 3rd person camera? Is there a way to generalize those arrays to all environments? @aclegg3

124     static_3rd_person_camera_position = np.array([-0.569043, 2.04804, 12.6156])

134     # place the object in the air
135     sim.set_translation(np.array([-0.569043, 2.04804, 13.6156]), quadrotor_id)

[anoopsonar30]

Moreover after adding -

102     agent_cfg.action_space = {
103     "move_forward": habitat_sim.agent.ActionSpec(
104         "move_forward", habitat_sim.agent.ActuationSpec(amount=0.25)
105         ),
106     "turn_left": habitat_sim.agent.ActionSpec(
107         "turn_left", habitat_sim.agent.ActuationSpec(amount=30.0)
108         ),
109     } 

These actions have no effect. That is the agent doesn't move/rotate at all.

177             sim.step("move_forward")
178             sim.step("turn_left")

I also tried sim.get_agent(quadrotor_id).act("move_forward") But it does not affect the quadrotor at all.

This is for the kinematic case which is the one I need. It is also a little unclear how the agent state and the object state are different

[erikwijmans]

Thanks a lot! You're a legend :) How did you choose the initial translation (the np.array that you hardcoded)? and the position of the 3rd person camera? Is there a way to generalize those arrays to all environments?

You should be able to take the pointnav spawn locations in the Gibson PointNav dataset (https://github.com/facebookresearch/habitat-api#task-datasets) and just add some offset in +y to move if off the ground.

[aclegg3]

How did you choose the initial translation (the np.array that you hardcoded)? and the position of the 3rd person camera? Is there a way to generalize those arrays to all environments?

... Is there a way to generalize those arrays to all environments?

You should be able to take the pointnav spawn locations in the Gibson PointNav dataset (https://github.com/facebookresearch/habitat-api#task-datasets) and just add some offset in +y to move if off the ground.

You may want to use this approach in concert with the sim.contact_test(quadrotor_id) function to implement rejection sampling for a spawn point with no collisions. Something like:

initial_height = 1.0
possible_initial_point = sim.pathfinder.get_random_navigable_point() + np.array([0, initial_height, 0])
sim.set_translation(possible_initial_point, quadrotor_id)
while(sim.contact_test(quadrotor_id)):
   possible_initial_point = sim.pathfinder.get_random_navigable_point() + np.array([0, initial_height, 0])
   sim.set_translation(possible_initial_point, quadrotor_id)

[anoopsonar30]

I also tried sim.get_agent(quadrotor_id).act("move_forward") But it does not affect the quadrotor at all.

Thanks! @erikwijmans @aclegg3 Is there a way I might be able to use the sim.step or the above command? I guess I could find the agent's orientation and do some quaternion manipulation to rotate and move the agent forward but I thought that is what the above commands were abstracting anyway.

[aclegg3]

Is there a way I might be able to use the sim.step or the above command? I guess I could find the agent's orientation and do some quaternion manipulation to rotate and move the agent forward but I thought that is what the above commands were abstracting anyway.

You are correct, that is the purpose of the agent action commands. However, the agent is not currently connected to the PhysicsManager objects. The script in this branch demonstrates how to move the object directly and sync the agent/camera for observations. However, for discrete kinematic control, a simple override of the step filter function to use the scene collision functions instead of the navmesh may suite you better. I'll see about getting something together to bridge the gap for you.

[aclegg3]

@anoopsonar30, I've updated the branch to include a modified step filter for the agent actions which utilizes the contact_test function. Try running python examples/quadrotor_prototype_2.py. Actions are still discrete for now, but you can modify the act/observe loop resolution by adjusting the amount parameter in agent_cfg.action_space. This formulation should be trainable in Habitat-api with minimal modification.

[anoopsonar30]

File "examples/quadrotor_prototype_2.py", line 136, in <module>
sim.agents[0].controls.move_filter_fn = sim._scene_collision_step_filter
AttributeError: 'Simulator' object has no attribute '_scene_collision_step_filter'

Ran into this error @aclegg3 even after pulling from the branch. I'm not sure what's going wrong since Simulator.py clearly has that function. Also I'm using the simulator stand-alone at the moment without using Habitat-api.

Oh lol some stackoverflowing led to the answer - Just close the terminals and start them again. It's working fine now. Not sure what the problem was.

[aclegg3]

This has to do with where the bindings are built when you install vs. where your environment is looking for them. I'm glad you found a solution.

[aclegg3]

Hey @anoopsonar30, I wanted to check in here to see how things are going. Any updates, questions, or feedback?

[anoopsonar30]

Thanks for checking in @aclegg3. I've been able to get the simulations up and running with no crashes. I wish we had a solution for the mesh gaps but it's good enough for now. I did have a few questions -

1) Since I've been running long time horizon simulations it's important that to train well, the robot be initialized at an "appropriate" translation and rotation such that it doesn't collide too early while the ideal controller is still being learned. Would there be a way to initialize the robot taking the semantics of the environment into consideration? I've noticed that with sample_navigable_point the robot gets initialized very close to walls/obstacles sometimes and hence crashes in the first few timesteps in many simulations since it's in a position where it's impossible to navigate or turn fast enough. Would there be a way to initialize the robot, say for example, in the center of a room or something like that so that it learns for longer without colliding too early?

This is especially a problem since we're dealing with indoor environments which are quite cramped compared to other environments where drones might operate. I also noticed that in your example code for habitat-api, eg. shortest_path_follower_example.py the agent has the ability to stop and rotate for multiple timesteps without translating which makes the navigation problem easier. In my simulation, the quadrotor always translates by a certain fixed amount each timestep.

2) This is slightly more trivial and although I feel my modified code should work I haven't been able to achieve the desired effect. The goal is to disable the lookAt feature on the camera. Essentially I want the camera window to stay static and only point to the initial position of the quadrotor and not "look at" it as it moves.

To do this, I saved the initial rotation of the camera where it is looking at the quadrotor and then alternate the camera between first person for quadrotor and 3rd person with this fixed saved rotation. However for some reason it doesn't seem to work. Instead the camera point's in what seems like roughly the opposite direction.

136     initRotation = ut.quat_to_magnum
137                 ut.quat_look_at(
138                     sim.get_translation(quadrotor_id), static_3rd_person_camera_position
139                 )
140             )

164             sim.get_agent(0).scene_node.translation = np.array(
165                 static_3rd_person_camera_position
166             )   
167             sim.get_agent(0).scene_node.rotation = initRotation
168             
169             agent_observations = sim.get_sensor_observations()
170             save_observation(agent_observations, frame, envNum, "3rdperson_")
171 
172             # reset the agent for the next step
173             sim.get_agent(0).scene_node.translation = sim.get_translation(quadrotor_id)
174             sim.get_agent(0).scene_node.rotation = sim.get_rotation(quadrotor_id)

3) Are you eventually planning to fill in the holes with something similar to the NNs in Gibson? We don't have to fill the holes in the mesh itself. I think it'd be great if we could fill in the holes in atleast the depth data (which is the only one I use at the moment for planning). If not, I guess I could use some sort of interpolation technique to fill in the holes and reduce the noise in the depth data. Do you recommend any specific techniques? I'm a little unsure about how to ensure that only the true mesh gaps get interpolated and not the obstacles themselves. The RGB data is purely just for visualization purposes and is not being used by the quadrotor for the simulations I'm working on.

[erikwijmans]

Are you eventually planning to fill in the holes with something similar to the NNs in Gibson?

The hole filling used in Gibson is only on the observations, not the geometry/mesh itself, thus that really isn't desirable for our cases (the Gibson folks did the hole filling NN for a very different reason than reconstruction noise). We are interested in fixing the geometry errors in the meshes, but automatic hole filling at the geometry level is an open area of research and I am unaware of anything that "just works".

[bryanbocao]

Here's another example from Circleville where the gap on the floor is all black. The same applies for the walls on the left side. Is there a way to fix this? I've been getting really bad training performance due to this. If I remember correctly Gibson uses some sort of neural network to fill in these gaps, is there something similar in Habitat?

How did you get so realistic image? Do you use a large resolution? I used 224x224 but I can feel some "unrealistic" just by human eyes.

[erikwijmans]

I just let the agent learn to deal with them. The agent doesn't seem to care about them and things seem to transfer fine to reality https://abhiskk.github.io/sim2real

[anoopsonar30]

How did you get so realistic image? Do you use a large resolution?

Yeah I'm probably using around 1920 x 1080 here or something similar. It's really cool that they support such high resolutions.