Running in headless mode

[vardhandongre]

I see that in sw there is: In https://github.com/Gabesarch/HELPER/blob/main/arguments.py I see' https://github.com/Gabesarch/HELPER/blob/7c69814aef9b78773ef8d04838f5a1556f2f50f9/arguments.py#L63 parser.add_argument("--do_headless_rendering", action="store_true", default=False, help="render in headless mode with new Ai2thor version")

However, it looks like the headless mode isn't available in the main evaluation code, as there isn't an option for it in the SimulatorTHOR(SimulatorBase) class, which the code uses. Is there a way to run it headless without changing the code?

[Gabesarch]

Thank you for bringing that to our attention. Indeed, that argument is currently not supported in the main code.

TEACh by default uses ai2thor version 3.1.0. Headless mode was not introduced until 4.1.0 (I believe). You could try updating to 4.1.0 and adding headless parameters to the controller, although I have not verified this.

If your machine supports Xorg, you can follow the README here to open a X server and run headless on that port.

[vardhandongre]

Thanks for the prompt response Gabe! I will give it a try with what you suggested.

[vardhandongre]

I managed to run it but it seems like its stuck somewhere and not running further than this:

Running 28c8b2c460bd375b_d1d0.tfd.json Iteration 1/1 fov: 90 INIT TfD... [MainThread-717950-INFO] teach.simulators.simulator_THOR: Time to create dataset definitions: 0.008011817932128906 sec [MainThread-717950-INFO] teach.simulators.simulator_THOR: Time to create dataset definitions: 0.008011817932128906 sec INFO:teach.simulators.simulator_THOR:Time to create dataset definitions: 0.008011817932128906 sec [MainThread-717950-INFO] teach.simulators.simulator_THOR: Initializing simulator... time to init Simulator_base: 0.05095052719116211 sec [MainThread-717950-INFO] teach.simulators.simulator_THOR: Initializing simulator... time to init Simulator_base: 0.05095052719116211 sec INFO:teach.simulators.simulator_THOR:Initializing simulator... time to init Simulator_base: 0.05095052719116211 sec [MainThread-717950-INFO] teach.simulators.simulator_THOR: Finished initializing simulator. Total time: 0.051274776458740234 sec [MainThread-717950-INFO] teach.simulators.simulator_THOR: Finished initializing simulator. Total time: 0.051274776458740234 sec INFO:teach.simulators.simulator_THOR:Finished initializing simulator. Total time: 0.051274776458740234 sec [MainThread-717950-INFO] teach.simulators.simulator_base: Resetting dataset object and removing previously stored episodes... INFO:teach.simulators.simulator_base:Resetting dataset object and removing previously stored episodes... [MainThread-717950-INFO] teach.replay.episode_replay: Starting episode... INFO:teach.replay.episode_replay:Starting episode... [MainThread-717950-INFO] teach.simulators.simulator_THOR: In simulator_THOR.start_new_episode, world = FloorPlan1_physics world_type = None [MainThread-717950-INFO] teach.simulators.simulator_THOR: In simulator_THOR.start_new_episode, world = FloorPlan1_physics world_type = None INFO:teach.simulators.simulator_THOR:In simulator_THOR.start_new_episode, world = FloorPlan1_physics world_type = None [MainThread-717950-INFO] teach.simulators.simulator_THOR: In SimulatorTHOR.start_new_episode, before launch_simulator [MainThread-717950-INFO] teach.simulators.simulator_THOR: In SimulatorTHOR.start_new_episode, before launch_simulator INFO:teach.simulators.simulator_THOR:In SimulatorTHOR.start_new_episode, before launch_simulator GPU IDX is 1 /usr/lib/xorg/Xorg.wrap: Only console users are allowed to run the X server [MainThread-717950-INFO] teach.simulators.simulator_THOR: In SimulatorTHOR.launch_simulator, creating ai2thor controller (unity process) [MainThread-717950-INFO] teach.simulators.simulator_THOR: In SimulatorTHOR.launch_simulator, creating ai2thor controller (unity process) INFO:teach.simulators.simulator_THOR:In SimulatorTHOR.launch_simulator, creating ai2thor controller (unity process) DEBUG:ai2thor.build:/home/vdongre2/.ai2thor/releases/thor-Linux64-fdc047690ee0ab7a91ede50d286bd387d379713a/thor-Linux64-fdc047690ee0ab7a91ede50d286bd387d379713a exists - skipping download INFO:root:Initialize return: {'cameraNearPlane': 0.10000000149011612, 'cameraFarPlane': 20.0} [MainThread-717950-INFO] teach.simulators.simulator_THOR: Time to create controller: 23.190205097198486 sec [MainThread-717950-INFO] teach.simulators.simulator_THOR: Time to create controller: 23.190205097198486 sec INFO:teach.simulators.simulator_THOR:Time to create controller: 23.190205097198486 sec

The script I used to run it:

python main.py --mode teach_eval_tfd \ --split valid_seen \ --episode_file 28c8b2c460bd375b_d1d0.tfd.json \ --gpt_embedding_dir ./data/gpt_embeddings \ --teach_data_dir ./data \ --server_port 0 \ --episode_in_try_except \ --use_llm_search \ --use_constraint_check \ --run_error_correction_llm \ --do_headless_rendering \ --zoedepth_checkpoint ./checkpoints/ZOEDEPTH-model-00015000.pth \ --solq_checkpoint ./checkpoints/SOLQ-model-00023000.pth \ --set_name HELPER_teach_tfd_validseen_ep

[Gabesarch]

Can you share the steps you took to run this script headless?

It could be that the controller is not able to render properly. Can you verify that you are able to start a controller and visualize the rendered image?

from ai2thor.controller import Controller
import matplotlib.pyplot as plt
controller = Controller()
event = controller.step("MoveAhead")
plt.figure()
plt.imshow(event.frame)
plt.savefig('test.png')

[vardhandongre]

I tried running it on a machine with direct access without making any changes to the code as well, it does visualize in a separate unity window but it doesn't run further.

[Gabesarch]

Thank you for the additional information and sorry for the delay. When you exit the run, what line of code is it getting stuck on? This would be helpful in further debugging.

[Gabesarch]

Closing due to inactivity.