How do I understand the npy file of the floorplan layout?

[wenda-gu]

Hi, thanks for sharing this amazing dataset! I try to get the floorplan from the npy file, and I have tried NumPy to load the file in Python. However, I find it difficult to understand the NumPy array I got from the npy file. How should I interpret the content of the floorplan npy files? Thank you.

Here's the code I use to load the file:

import numpy as np
npyData = np.load("example/fp2/FloorPlan2-layout.npy")
print(npyData.shape)
print(npyData)

Here's the output of the shape: (115, 2) and the array:

[[-0.75 -0.75]
 [-0.5  -0.75]
 [-0.25 -0.75]
 [ 0.   -0.75]
 [ 0.25 -0.75]
 [ 0.5  -0.75]
 [ 0.75 -0.75]
 [ 1.   -0.75]
 [-0.75 -0.5 ]
 [ 0.75 -0.5 ]
 [ 1.   -0.5 ]
 [-0.75 -0.25]
 [ 0.75 -0.25]
 [ 1.   -0.25]
 [-0.75  0.  ]
 [ 0.75  0.  ]
 [ 1.    0.  ]
 [ 1.25  0.  ]
 [-0.75  0.25]
 [ 0.75  0.25]
 [ 1.    0.25]
 [ 1.25  0.25]
 [-0.75  0.5 ]
 [ 0.75  0.5 ]
 [ 1.    0.5 ]
 [ 1.25  0.5 ]
 [-0.75  0.75]
 [ 0.75  0.75]
 [ 1.    0.75]
 [ 1.25  0.75]
 [-0.75  1.  ]
 [ 0.75  1.  ]
 [ 1.    1.  ]
 [ 1.25  1.  ]
 [-0.75  1.25]
 [ 0.75  1.25]
 [ 1.    1.25]
 [ 1.25  1.25]
 [-1.25  1.5 ]
 [-1.    1.5 ]
 [-0.75  1.5 ]
 [ 0.75  1.5 ]
 [ 1.    1.5 ]
 [-1.25  1.75]
 [-1.    1.75]
 [-0.75  1.75]
 [-0.5   1.75]
 [ 0.5   1.75]
 [ 0.75  1.75]
 [-1.5   2.  ]
 [-1.25  2.  ]
 [-1.    2.  ]
 [-0.75  2.  ]
 [-0.5   2.  ]
 [-0.25  2.  ]
 [ 0.    2.  ]
 [ 0.25  2.  ]
 [ 0.5   2.  ]
 [ 0.75  2.  ]
 [-1.75  2.25]
 [-1.5   2.25]
 [-1.25  2.25]
 [-1.    2.25]
 [-0.75  2.25]
 [-0.5   2.25]
 [-0.25  2.25]
 [ 0.    2.25]
 [ 0.25  2.25]
 [ 0.5   2.25]
 [ 0.75  2.25]
 [-2.5   2.5 ]
 [-2.25  2.5 ]
 [-2.    2.5 ]
 [-1.75  2.5 ]
 [-1.5   2.5 ]
 [-1.25  2.5 ]
 [-1.    2.5 ]
 [-0.75  2.5 ]
 [-0.5   2.5 ]
 [-0.25  2.5 ]
 [ 0.    2.5 ]
 [ 0.25  2.5 ]
 [ 0.5   2.5 ]
 [ 0.75  2.5 ]
 [-2.5   2.75]
 [-2.25  2.75]
 [-2.    2.75]
 [-1.75  2.75]
 [-1.5   2.75]
 [-1.25  2.75]
 [-1.    2.75]
 [-0.75  2.75]
 [-0.5   2.75]
 [-0.25  2.75]
 [ 0.    2.75]
 [ 0.25  2.75]
 [ 0.5   2.75]
 [ 0.75  2.75]
 [-2.5   3.  ]
 [-2.25  3.  ]
 [-2.    3.  ]
 [-1.75  3.  ]
 [-1.5   3.  ]
 [-1.25  3.  ]
 [-1.    3.  ]
 [-0.75  3.  ]
 [-0.5   3.  ]
 [-0.25  3.  ]
 [ 0.    3.  ]
 [ 0.25  3.  ]
 [ 0.5   3.  ]
 [ 0.75  3.  ]
 [-1.    3.25]
 [-0.75  3.25]
 [-0.5   3.25]]

[MohitShridhar]

@wenda-gu, this is the layout of the room represented as a set of 2D grid points. Each of these pairs are x, z points that correspond to navigable locations in the scene. With the THOR api you can call a TeleportFull action like this:

action = {'action': 'TeleportFull',
               'x': point[0],
               'y': agent_height,
               'z': point[1]}
env.step(action)

where agent_height = env.last_event.metadata['agent']['position']['y']

[yuchenlin]

@MohitShridhar thanks for your prompt help! We were wondering how we should be able to get the exact positions of the (non-randomized) objects in a floor plan such as the sofa, the counter, sink, etc. We know that randomized objects such as potatoes can be found in */traj_data.json files, while we don't know how to get the info of those fixed objects. Thank you very much in advance!

[MohitShridhar]

@yuchenlin checkout FloorPlanX-openable.json files here. Each receptacle object in the scene (sink, sofa, etc.) has a corresponding navigable location (x, z, rotation in deg, camera horizon in deg). Navigate to these locations to interact with objects in/on the receptacle, or the receptacle itself.

[yuchenlin]

Hi @MohitShridhar , thanks for the heads-up. I found that the FloorPlanX-openable.json only contains the objects that can be opened (such as a drawer, cabinet, sink, microwave, fridge), while other non-randomized, non-openable objects such as "CoffeeMachine" and "Toaster" are not included.

Specifically, in FloorPlan2-objects.json, we can see Toaster and CoffeeMachine, but not in the FloorPlan2-openable.json. From the demo, we can find that they are located near the microwave, though.

Would it be possible that we can also get the info about them? Thanks again!

[MohitShridhar]

@yuchenlin, you might have to check env.last_event.metadata['objects'] (see this) to find the ['position'] of these objects, and then find the closest navigable point in FloorPlanX-layout.npy.

Using this metadata during training is fine. But make sure you don't use it for test scenes and leaderboard submissions.

[yuchenlin]

Thank you very much for the tips. We will take a closer look. And yes, we will not use it for any submissions. Actually, we are more interested in data analysis and want to use the ALFRED dataset for other tasks. Thanks again! @MohitShridhar

[MohitShridhar]

Cool, good luck!

[wenda-gu]

Hi @MohitShridhar , thanks for your suggestion! We are looking at precompute_layout_locations.py under layouts folder, and we are trying to get the environment (line 114-115)

env = game_util.create_env(build_path=constants.BUILD_PATH,  quality='Low')

However, we cannot find such a function called create_env in game_util.py or elsewhere. Is it the same as

env = env.thor_env.ThorEnv(build_path=constants.BUILD_PATH,  quality='Low')

Hope you can help. Thank you very much!

[MohitShridhar]

@wenda-gu thanks for pointing this out! Just pushed a fix to master.

You should be able to cd $ALFRED_ROOT/gen and python layout/precompute_layout_locations.py.

[wenda-gu]

Thanks to your helpful replies, we have successfully extracted metadata info from precompute_layout_locations.py. However, we want to know, on the high level, the receptacle relation between objects in order to understand the spatial relation of objects, or, on the low level, how metadata relates to traj_data.json and how and when did the randomization process happen. As far as what we have found, in line 534 in the file gen/scripts/generate_trajectories.py,

object_poses = [{'objectName': obj['name'].split('(Clone)')[0],
                              'position': obj['position'],
                              'rotation': obj['rotation']}
                              for obj in env.last_event.metadata['objects'] if obj['pickupable']]

the info for randomization comes from env.last_event.metadata['objects']. Is that how the randomization works? If not, does it comes from line 551 env.restore_scene(object_poses, object_toggles, dirty_and_empty), and how should we interpret the block of code from line 545 to line 552? Thank you!

print("Performing reset via thor_env API")
env.reset(sampled_scene)
print("Performing restore via thor_env API")
env.restore_scene(object_poses, object_toggles, dirty_and_empty)
event = env.step(dict(constants.data_dict['scene']['init_action']))

[MohitShridhar]

@wenda-gu, by default THOR randomly distributes objects inside a scene. But for reproducibility and deterministic evaluations, we took all the states and poses of objects during data generation and saved them in the JSON file. The restore_scene function deterministically reloads scenes with these saved states and poses. So in summary, object randomization was done only during data generation.

env.reset loads the room and env.restore_scene loads all the objects with saved states and poses.

[MohitShridhar]

Closing due to inactivity. Feel free to continue using this thread.