Unity process has exited

[Aoihigashi]

Thanks for sharing your work! I am here asking when I run python main.py --query "a living room" --openai_api_key sk-XXXX,I have encountered the error below:(I am running on a headless server, and I added the argument platform=CloudRendering followed the instructions before.)

and before this run, I encountered this:

but it takes so long and I KeyboardInterrupt it, btw my server is configured with a Linux system, Ubuntu 22.04. Looking forward to your reply。

[YueYANG1996]

Could you follow the solution in this issue to set the multiprocessing to False?

[Aoihigashi]

Could you follow the solution in this issue to set the multiprocessing to False?

yeah, I already set the multiprocessing to Fasle, but still...this is the trace back after I set the multiprocessing to False,

I guess I got something wrong with Vulkan, my new error shows below:

[Aoihigashi]

Could you follow the solution in this issue to set the multiprocessing to False?

yeah, I already set the multiprocessing to Fasle, but still...this is the trace back after I set the multiprocessing to False,

I guess I got something wrong with Vulkan, my new error shows below:

and I have installed Vulkan-tools

[YueYANG1996]

@sunfanyunn Hi Sun, how did you resolve this issue before?

[Aoihigashi]

Could you follow the solution in this issue to set the multiprocessing to False?

yeah, I already set the multiprocessing to Fasle, but still...this is the trace back after I set the multiprocessing to False, I guess I got something wrong with Vulkan, my new error shows below:

and I have installed Vulkan-tools

this is my recent error and its traceback:

[YueYANG1996]

@Lucaweihs Hi Luca, do you have any ideas for this Vulkan issue for running AI2-THOR on the headless server?

[sunfanyunn]

I believe you can solve this by starting the controller under a headless setup. Try adding the argument platform=CloudRendering here: https://github.com/allenai/Holodeck/blob/db15e52238135b9595bbd344eadde81b8a1b25ff/modules/small_objects.py#L220

[Lucaweihs]

Hi @Aoihigashi , can you confirm that if you run vulkaninfo from the command line it executes without an error? The thor code that causes the above error to print is just trying to run vulkaninfo using subprocess:

                    try:
                        vulkan_result = subprocess.run(
                            ["vulkaninfo"],
                            stdout=subprocess.PIPE,
                            stderr=subprocess.DEVNULL,
                            universal_newlines=True,
                        )
                    except FileNotFoundError:
                        pass
                    if vulkan_result is None or vulkan_result.returncode != 0:
                        raise RuntimeError(
                            "vulkaninfo failed to run, please ask your administrator to"
                            " install `vulkaninfo` (e.g. on Ubuntu systems this requires running"
                            " `sudo apt install vulkan-tools`)."
                        )

If it works, can you save the output from vulkaninfo to a file and add it here?

Also, can you also include the Player.log error file that's referenced in your second error message?

Also, just to confirm, are you running holodeck from within a docker container?

[Aoihigashi]

Hi @Aoihigashi , can you confirm that if you run vulkaninfo from the command line it executes without an error? The thor code that causes the above error to print is just trying to run vulkaninfo using subprocess:

                    try:
                        vulkan_result = subprocess.run(
                            ["vulkaninfo"],
                            stdout=subprocess.PIPE,
                            stderr=subprocess.DEVNULL,
                            universal_newlines=True,
                        )
                    except FileNotFoundError:
                        pass
                    if vulkan_result is None or vulkan_result.returncode != 0:
                        raise RuntimeError(
                            "vulkaninfo failed to run, please ask your administrator to"
                            " install `vulkaninfo` (e.g. on Ubuntu systems this requires running"
                            " `sudo apt install vulkan-tools`)."
                        )

If it works, can you save the output from vulkaninfo to a file and add it here?

Also, can you also include the Player.log error file that's referenced in your second error message?

Also, just to confirm, are you running holodeck from within a docker container?

thanks for helping me @Lucaweihs ,indeed I was running holodeck from within a docker container earlier, but to avoid the Vulkan issues, I tried download the whole repository and run holodeck on my own laptop, which is MacBook Pro with M2 chip, although I'm not sure if my 16GB of RAM is sufficient for this project, after my attempts, I successfully ran main.py with results below:

and my backend automatically opened a Unity AI2-THOR window as shown below, but this window has been stuck here.

Then I checked the Unity log, and below are some of the results I captured. It seems there is an issue like 'Setup Scene Called False‘ These are the issues I am currently facing. I look forward to your reply. Thank you for your patience。

[Aoihigashi]

Hi @Aoihigashi , can you confirm that if you run vulkaninfo from the command line it executes without an error? The thor code that causes the above error to print is just trying to run vulkaninfo using subprocess:

                    try:
                        vulkan_result = subprocess.run(
                            ["vulkaninfo"],
                            stdout=subprocess.PIPE,
                            stderr=subprocess.DEVNULL,
                            universal_newlines=True,
                        )
                    except FileNotFoundError:
                        pass
                    if vulkan_result is None or vulkan_result.returncode != 0:
                        raise RuntimeError(
                            "vulkaninfo failed to run, please ask your administrator to"
                            " install `vulkaninfo` (e.g. on Ubuntu systems this requires running"
                            " `sudo apt install vulkan-tools`)."
                        )

If it works, can you save the output from vulkaninfo to a file and add it here? Also, can you also include the Player.log error file that's referenced in your second error message? Also, just to confirm, are you running holodeck from within a docker container?

thanks for helping me @Lucaweihs ,indeed I was running holodeck from within a docker container earlier, but to avoid the Vulkan issues, I tried download the whole repository and run holodeck on my own laptop, which is MacBook Pro with M2 chip, although I'm not sure if my 16GB of RAM is sufficient for this project, after my attempts, I successfully ran main.py with results below: and my backend automatically opened a Unity AI2-THOR window as shown below, but this window has been stuck here. Then I checked the Unity log, and below are some of the results I captured. It seems there is an issue like 'Setup Scene Called False‘ These are the issues I am currently facing. I look forward to your reply. Thank you for your patience。

[Aoihigashi]

Hi @Aoihigashi , can you confirm that if you run vulkaninfo from the command line it executes without an error? The thor code that causes the above error to print is just trying to run vulkaninfo using subprocess:

                    try:
                        vulkan_result = subprocess.run(
                            ["vulkaninfo"],
                            stdout=subprocess.PIPE,
                            stderr=subprocess.DEVNULL,
                            universal_newlines=True,
                        )
                    except FileNotFoundError:
                        pass
                    if vulkan_result is None or vulkan_result.returncode != 0:
                        raise RuntimeError(
                            "vulkaninfo failed to run, please ask your administrator to"
                            " install `vulkaninfo` (e.g. on Ubuntu systems this requires running"
                            " `sudo apt install vulkan-tools`)."
                        )

If it works, can you save the output from vulkaninfo to a file and add it here? Also, can you also include the Player.log error file that's referenced in your second error message? Also, just to confirm, are you running holodeck from within a docker container?

thanks for helping me @Lucaweihs ,indeed I was running holodeck from within a docker container earlier, but to avoid the Vulkan issues, I tried download the whole repository and run holodeck on my own laptop, which is MacBook Pro with M2 chip, although I'm not sure if my 16GB of RAM is sufficient for this project, after my attempts, I successfully ran main.py with results below: and my backend automatically opened a Unity AI2-THOR window as shown below, but this window has been stuck here. Then I checked the Unity log, and below are some of the results I captured. It seems there is an issue like 'Setup Scene Called False‘ These are the issues I am currently facing. I look forward to your reply. Thank you for your patience。

for more details, here is the traceback after KeyboardInterrupt: ^CProcess SpawnPoolWorker-8: Process SpawnPoolWorker-7: Process SpawnPoolWorker-5: Traceback (most recent call last): File "/Users/umi/Holodeck/main.py", line 94, in <module> Process SpawnPoolWorker-6: generate_single_scene(args) File "/Users/umi/Holodeck/main.py", line 21, in generate_single_scene args.model.generate_scene( File "/Users/umi/Holodeck/modules/holodeck.py", line 234, in generate_scene Traceback (most recent call last): top_image = get_top_down_frame(scene, self.objaverse_asset_dir, 1024, 1024) File "/Users/umi/Holodeck/modules/utils.py", line 34, in get_top_down_frame File "/opt/anaconda3/envs/holodeck/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/opt/anaconda3/envs/holodeck/lib/python3.9/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/opt/anaconda3/envs/holodeck/lib/python3.9/multiprocessing/pool.py", line 114, in worker task = get() File "/opt/anaconda3/envs/holodeck/lib/python3.9/multiprocessing/queues.py", line 365, in get res = self._reader.recv_bytes() File "/opt/anaconda3/envs/holodeck/lib/python3.9/multiprocessing/connection.py", line 216, in recv_bytes buf = self._recv_bytes(maxlength) File "/opt/anaconda3/envs/holodeck/lib/python3.9/multiprocessing/connection.py", line 414, in _recv_bytes buf = self._recv(4) File "/opt/anaconda3/envs/holodeck/lib/python3.9/multiprocessing/connection.py", line 379, in _recv chunk = read(handle, remaining) KeyboardInterrupt Traceback (most recent call last): Traceback (most recent call last): File "/opt/anaconda3/envs/holodeck/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/opt/anaconda3/envs/holodeck/lib/python3.9/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/opt/anaconda3/envs/holodeck/lib/python3.9/multiprocessing/pool.py", line 114, in worker task = get() File "/opt/anaconda3/envs/holodeck/lib/python3.9/multiprocessing/queues.py", line 364, in get with self._rlock: File "/opt/anaconda3/envs/holodeck/lib/python3.9/multiprocessing/synchronize.py", line 95, in __enter__ return self._semlock.__enter__() File "/opt/anaconda3/envs/holodeck/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/opt/anaconda3/envs/holodeck/lib/python3.9/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/opt/anaconda3/envs/holodeck/lib/python3.9/multiprocessing/pool.py", line 114, in worker task = get() File "/opt/anaconda3/envs/holodeck/lib/python3.9/multiprocessing/queues.py", line 364, in get with self._rlock: File "/opt/anaconda3/envs/holodeck/lib/python3.9/multiprocessing/synchronize.py", line 95, in __enter__ return self._semlock.__enter__() KeyboardInterrupt KeyboardInterrupt Traceback (most recent call last): File "/opt/anaconda3/envs/holodeck/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/opt/anaconda3/envs/holodeck/lib/python3.9/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/opt/anaconda3/envs/holodeck/lib/python3.9/multiprocessing/pool.py", line 114, in worker task = get() File "/opt/anaconda3/envs/holodeck/lib/python3.9/multiprocessing/queues.py", line 364, in get with self._rlock: File "/opt/anaconda3/envs/holodeck/lib/python3.9/multiprocessing/synchronize.py", line 95, in __enter__ return self._semlock.__enter__() KeyboardInterrupt controller = Controller( File "/opt/anaconda3/envs/holodeck/lib/python3.9/site-packages/ai2thor/controller.py", line 545, in __init__ self.start( File "/opt/anaconda3/envs/holodeck/lib/python3.9/site-packages/ai2thor/controller.py", line 1548, in start self.last_event = self.server.receive(timeout=self.server_start_timeout) File "/opt/anaconda3/envs/holodeck/lib/python3.9/site-packages/ai2thor/fifo_server.py", line 229, in receive metadata, files = self._recv_message( File "/opt/anaconda3/envs/holodeck/lib/python3.9/site-packages/ai2thor/fifo_server.py", line 140, in _recv_message self.server_pipe = open(self.server_pipe_path, "rb") KeyboardInterrupt

[Lucaweihs]

Hi @Aoihigashi,

thanks for helping me @Lucaweihs ,indeed I was running holodeck from within a docker container earlier

Got it, yes running AI2-THOR in a docker container is a bit tricky as you need to ensure certain nvidia config files are mounted correctly and that the /dev/dri device is set, here's a README we have for a different project that describes how to set this up.

although I'm not sure if my 16GB of RAM is sufficient for this project,

16GB of RAM should be sufficient as all the heavy lifting is offloaded to the OpenAI API.

Then I checked the Unity log, and below are some of the results I captured. It seems there is an issue like 'Setup Scene Called False‘ These are the issues I am currently facing. I look forward to your reply. Thank you for your patience。

I'm a bit confused, it sounds like you've been following the Load scene in Unity directions? Can you confirm you've set the --scene and --asset_dir flags? Also, can you send the json file corresponding to the house that you managed to generate?

Also, it would be nice to check if you're able to use AI2-THOR at all, can you try running something like:

from ai2thor.controller import Controller

c = Controller()
c.reset("FloorPlan1")

for i in range(10):
    print(f"Attempting to rotate right ({i})")
    c.step("RotateRight")
    print(f"Rotation successful")

and making sure it prints things out?

[Aoihigashi]

Hi @Aoihigashi,

thanks for helping me @Lucaweihs ,indeed I was running holodeck from within a docker container earlier

Got it, yes running AI2-THOR in a docker container is a bit tricky as you need to ensure certain nvidia config files are mounted correctly and that the /dev/dri device is set, here's a README we have for a different project that describes how to set this up.

although I'm not sure if my 16GB of RAM is sufficient for this project,

16GB of RAM should be sufficient as all the heavy lifting is offloaded to the OpenAI API.

Then I checked the Unity log, and below are some of the results I captured. It seems there is an issue like 'Setup Scene Called False‘ These are the issues I am currently facing. I look forward to your reply. Thank you for your patience。

I'm a bit confused, it sounds like you've been following the Load scene in Unity directions? Can you confirm you've set the --scene and --asset_dir flags? Also, can you send the json file corresponding to the house that you managed to generate?

Also, it would be nice to check if you're able to use AI2-THOR at all, can you try running something like:

from ai2thor.controller import Controller

c = Controller()
c.reset("FloorPlan1")

for i in range(10):
    print(f"Attempting to rotate right ({i})")
    c.step("RotateRight")
    print(f"Rotation successful")

and making sure it prints things out?

hi,@Lucaweihs I followed your instruction to check if I can use AI2-THOR, turned out there is nothing wrong with it, below is the results:

The issue I'm currently facing is that when I run connect_to_unity.py, it returns the following error:

Interestingly, after running main.py, the saved scene image and the AI2-THOR window automatically pop up, but then it gets stuck here, continuously loading.

https://github.com/allenai/Holodeck/assets/130173118/3226d012-ab1b-4cc1-b560-03d6283eaffa

I have thought about two possible reasons for this issue: firstly, my computer may not be able to handle AI2-THOR, causing it to get stuck here. Another reason could be that my computer has an M2 chip, but the Unity editor requires version 2020.3.25f1, which is only available for Intel chips. However, I still downloaded this version. By the way, I have correctly set the --scene and --asset_dir. Below is the JSON file of the scene I want to put into AI2-THOR. a_modern_gym_with_lots_of_equi.json

Thank you very much for your help!!!