search

allenai / Holodeck

CVPR 2024: Language Guided Generation of 3D Embodied AI Environments.
https://yueyang1996.github.io/holodeck
Apache License 2.0
304 stars 25 forks source link

Generation gets stuck during GPT interaction #16

Closed mlodel closed 6 months ago

mlodel commented 6 months ago

Thanks for open-sourcing this great project!

I am running into a problem when running the example command python main.py --query "a living room" --openai_api_key <OPENAI_API_KEY>. The interaction with GPT4 gets stuck after the following line is printed

AI: Here is the window plan:
living room | east | fixed | (240, 120) | 3 | 100

AI: Selecting objects for living room...

There is no error printed, and the processes continue running, but nothing happens anymore, for multiple hours (then I killed the process).

Can you give me some pointers on what could be the problem here? Thank you!

YueYANG1996 commented 6 months ago

Could you check this issue? It could be because of the multiprocessing problems.

mlodel commented 6 months ago

@YueYANG1996 Thanks for referring to that issue! I make the example command run by code modifications in 3 different files, as suggested in #3 (multiprocessing=False, tensors to GPU, spawn mode). This leads to a json file being generated and an image opening, however the process end with the same problem as in #13.

So the issue with getting stuck during GPT4 queries clearly stems from multiprocessing and torch. I wonder if some setup of the machine/system where I run it has influence on that? I suppose that you could run it with the code in the repo, so there must be some setup difference.

YueYANG1996 commented 6 months ago

Could you follow the instructions here to see if you can load the scene into Unity?