Object coordinate system in info_semantic.json file

[andreicnica]

Do the object coordinate system have any correspondence with the habitat agent position? What is the transformation between the coordinate system from the file and the habitat simulator?

We have tried #29 but the coordinates of the object in the habitat sim do not match the "oriented_bbox": "abb": "center" at all.

Thanks!

[erikwijmans]

29 deals with the more complicated transform of getting from habitat-sim's camera specification the Replica SDK's camera specification. The transform between coordinates in habitat-sim and the Replica Dataset is significantly more simple! Habitat just rotates the meshes in replica by the quaternion that rotates -Z to -Y, i.e.

from habitat_sim.utils import quat_from_two_vectors, quat_rotate_vector

rotation_habitat_replica = quat_from_two_vectors(np.array([0, 0, -1]), np.array([0, -1, 0]))
rotation_replica_habitat = rotation_habitat_replica.inverse()

agent_position_habitat = agent.state.position
agent_position_replica = quat_rotate_vector(rotation_replica_habitat, pt_habitat)

[alexandrugavril]

We tried that, yet the for example in the frl_apartment_1 in the Replica Dataset, the agent in front of the refrigerator has more or less the following position [4.43875,-1.3151,0.00484991] which converted to the replica coordinate is [4.43875, -0.00484991, -1.3151] by rotating -Z to -Y. The position in the info_semantic.json has the center in [-0.74199306964874268,-4.4517850875854492,-0.67235374450683594] which is way off. Is there something we are missing? I expect to have a difference since we don't expect to get to the center of the object, but this is a significant one.

[erikwijmans]

which converted to the replica coordinate is [4.43875, -0.00484991, -1.3151] by rotating -Z to -Y

It's the other way around. You go from habitat to replica by rotating -Y to -Z. See the example I posted above.

[mihirp1998]

@alexandrugavril @erikwijmans

did any one of u solve this?.. i am also stuck on the same thing as @alexandrugavril for the last 6 hours. can't find a sensible transformation matrix from replica to habitat origin

Please help!

[erikwijmans]

See above my post above, the transformation matrix from replica to habitat is defined by the unit quaternion that rotates -Z to -Y, you can get that in habitat with

from habitat_sim.utils import quat_from_two_vectors, quat_rotate_vector

# I write my transforms/rotations as, for instance, rotation_<to>_<from>
rotation_habitat_replica = quat_from_two_vectors(np.array([0, 0, -1]), np.array([0, -1, 0])) 

This is the exact transform we apply to the replica meshes in habitat.

[erikwijmans]

You can see how the loading for bounding boxes is done here: https://github.com/facebookresearch/habitat-sim/pull/503

[shamitlal]

Hi @erikwijmans , This transformation will only affect the y and z coordinate values right? I am facing issue similar to what @alexandrugavril has mentioned. If you see his comment, his x coordinate is also not matching with the x coordinate of the object in replica. I don't think this transformation will fix that.

[erikwijmans]

Are you also accounting for the transform of the OBB itself? The bounding boxes in the info_semantic.json are OBBs, not AABBs and need to be treated as such (they have their own coordinate frame). Please refer to this piece of code: https://github.com/facebookresearch/habitat-sim/pull/503/files#diff-8b7305e0ebc1cfa9d0abe2ecda0b6ea0R86-R103

[shamitlal]

Yes I was using the bounding boxes given in json file. Do you know how can I get AABB bounding boxes? When I do sim.semantic_scene.objects[i].aabb, I get (0,0,0) as center and size of bounding box for all objects 'i'. Is there some other way to extract aabb boxes? Thanks

[erikwijmans]

I merged that PR, you should now be able to sim.semantic_scene.objects[I].obb.to_aabb()

[shunya-kato]

@mihirp1998 @shamitlal Did anyone of you solve this problem? We have the same problem. Please help!

[VladimirYugay]

@erikwijmans

Having a similar problem, but for the egomotion. Assuming it is in the Habitat coordinate system, how to move it to the Replica coordinate system?

Based on your comments tried the following, but this doesn't help.

# Transformation to move a point from replica to habitat
rotation_habitat_replica = quat_from_two_vectors(np.array([0, 0, -1]), np.array([0, -1, 0])) 

# Transformation to move a point from habitat to replica
rotation_replica_habitat = rotation_habitat_replica.inverse()

R_habitat = agent_state.rotation
t_habitat = agent_state.position

R_replica = rotation_replica_habitat * R_habitat * rotation_habitat_replica
t_replica = rotation_replica_habitat * t_habitat