allenai / Holodeck

CVPR 2024: Language Guided Generation of 3D Embodied AI Environments.
https://yueyang1996.github.io/holodeck
Apache License 2.0
304 stars 25 forks source link

Question about adjusting the spatial relationship of objects #46

Closed zoeyliu1999 closed 1 week ago

zoeyliu1999 commented 2 weeks ago

Dear authors, thank you for your excellent work!

I am trying to render a specified layout of furniture but am having trouble with HOLODECK understanding the spatial relationships. I am confused about how to correctly input this information or modify it directly in the file.

For example, my query is: "a rectangular bedroom with tiled flooring and large windows on one side. Place the bed against the wall opposite the windows, the wardrobe to the right of the bed, and the desk with a chair near the windows." However, the generated layouts always place the bed incorrectly, as shown in the attached pictures.

How can I adjust this? Should I refine my query, or directly modify the positions in the .json file? Which method would be more effective? Any suggestions or guidance would be greatly appreciated. Thank you!

a_rectangular_bedroom_exp1

a_rectangular_bedroom_exp2

zoeyliu1999 commented 2 weeks ago

Hey there, I wanted to add some new information. When I regenerated the scene using a query with a specific orientation, such as "large window on the east wall" and "place the bed against the west wall, opposite the window," HOLODECK performed better. The result, as shown in the attached image, was different from what I initially expected but a significant improvement over the previous query.

a_rectangular_bedroom,_with_ti

Can I directly edit the room's size or object positions in the .json file? The .json file looks like this:

"objects": [
        {
            "assetId": "Bed_9",
            "id": "bed-0 (bedroom)",
            "kinematic": true,
            "position": {
                "x": 1.2107633399963378,
                "y": 0.6772908568382263,
                "z": 1.5
            },
            "rotation": {
                "x": 0,
                "y": 90,
                "z": 0
            },
            "material": null,
            "roomId": "bedroom",
            "vertices": [
                [
                    246.65266799926758,
                    41.439504623413086
                ],
                [
                    246.65266799926758,
                    258.5604953765869
                ],
                [
                    -4.5,
                    258.5604953765869
                ],
                [
                    -4.5,
                    41.439504623413086
                ]
            ],
            "object_name": "bed-0",
            "layer": "Procedural0"
        },

Is it effective to edit the position details in the above "Bed_9" entry? Additionally, I couldn't find the parameters to control the room size. Any suggestions or guidance would be greatly appreciated.

zoeyliu1999 commented 2 weeks ago

I apologize for mistakenly closing the issue. It is still open.

YueYANG1996 commented 1 week ago

We did not consider the fine-grained spatial relation between objects and walls (only consider the non-collision and within the boundary for walls).

However, the DFS constraints solver can return multiple solutions, and you can choose from them.

You could also refer to this function which will rerun the DFS solver and give you a different layout using the same set of objects.

zoeyliu1999 commented 1 week ago

We did not consider the fine-grained spatial relation between objects and walls (only consider the non-collision and within the boundary for walls).

However, the DFS constraints solver can return multiple solutions, and you can choose from them.

You could also refer to this function which will rerun the DFS solver and give you a different layout using the same set of objects.

Thank you for your kind reply!