search

allenai / Holodeck

CVPR 2024: Language Guided Generation of 3D Embodied AI Environments.
https://yueyang1996.github.io/holodeck
Apache License 2.0
304 stars 25 forks source link

Unable to Connect to Unity After Scene Generation #33

Closed DDDDorwin closed 2 months ago

DDDDorwin commented 3 months ago

Hello! First, a big thank you for your great work.

Issue Description But I am having trouble connecting to Unity after generating a scene with a Python script. The scene generation is completed successfully, producing a JSON file and a PNG image. However, when I attempt to connect to Unity using the provided command, I encounter several issues.

Steps to Reproduce Cloned the repository and created a new branch. Ran the scene generation script which executed successfully, generating the expected JSON and PNG files. Attempted to connect to Unity with the following command:

python3 connect_to_unity.py --scene /home/dorwin/HolodeckUnity/Holodeck/data/scenes/a_Scadinavian_style_living_roo-2024-04-19-12-09-29-367318/a_Scadinavian_style_living_roo.json

The terminal indicates that port 8200 is active. Navigating to http://localhost:8200/ displays a "Not Found" error. Clicking Unity's play button does not result in a connection to the generated scene.

Environment Operating System: Windows Subsystem for Linux (Ubuntu) Unity Version: 2020.3.25

1 screenshot1 a_Scadinavian_style_living_roo.json

YueYANG1996 commented 3 months ago

I haven't tried to load scenes in Unity using Windows (wsl). Do you think it is possible @Lucaweihs @AlvaroHG ?

Lucaweihs commented 2 months ago

I also haven't tried this but I believe it should be possible; @elimvb and @winthos use Windows and I believe they have some tools exactly for loading Objaverse assets in AI2-THOR in the editor. I'll let one of them say more.

AlvaroHG commented 2 months ago

From your screenshots I see that you have missmatched ai2thor versions; the repository and the python package install, I would install ai2thor from your local repo in your python env or from a commit id that matches the commit checked out in your local repo.

To install from your local repo pip install -e <path_to_ai2thor_repo_root>

I would suggest using commit 8b74622599da96cfd6c502082ab55550e99f03bb which is our candidate release. But you can follow Holodeck instructions too. To install, check out that commit and run either the above command or this (commit you choose after @): pip install ai2thor git+https://github.com/allenai/ai2thor@8b74622599da96cfd6c502082ab55550e99f03bb

For any commit you check out make sure you also install the dependencies listed in that commit so running python <path_to_ai2thor_repo_root>/setup.py install

Finally make sure you have the following python packages at the specific versions Flask==2.0.1 Werkzeug==2.0.1 for unity editor communication to work.

Hope this helps!