Cloud Rendering Support

[brandontrabucco]

Hello ai2thor-rearrangement authors,

I am using AI2-THOR in a setting where I will not be able to use an X server for rendering. It looks like support for rendering using Vulkan was recently added to AI2-THOR in one of the recent releases (3.5.0, I think), via the CloudRendering option in the Controller object. However, it looks like only a specific commit ID for the Unity engine currently supports Vulkan with AI2-THOR.

https://github.com/allenai/ai2thor/blob/db856525b770e0ff5af38e9efa27ac0073221be3/ai2thor/build.py#L35

But, the commit ID is set to something different in the rearrangement challenge:

https://github.com/allenai/ai2thor-rearrangement/blob/main/rearrange/constants.py#L12

I find that if I set the commit ID to the one that supports Vulkan rendering, I see the following error when attempting to rotate the agent. If I understand the error, it looks like this version of Unity doesn't support the actionSimulationSeconds argument, which is used by the RearrangeTHOREnvironment class.

File ".../allenact/allenact/base_abstractions/task.py", line 124, in step
    sr = self._step(action=action)
  File ".../ai2thor-rearrangement/rearrange/tasks.py", line 636, in _step
    action_success = getattr(self.walkthrough_env, action_name)()
  File ".../ai2thor-rearrangement/rearrange/environment.py", line 581, in rotate_right
    return execute_action(
  File ".../ai2thor-rearrangement/rearrange/utils.py", line 250, in execute_action
    event = controller.step(thor_action, **kwargs)
  File ".../ai2thor/controller.py", line 960, in step
    raise ValueError(self.last_event.metadata["errorMessage"])
ValueError: 
        Action: "RotateRight" called with invalid argument: 'actionSimulationSeconds'
        Expected arguments: Nullable`1 degrees = , Boolean manualInteract = False, Boolean forceAction = False, Single speed = 1, Boolean waitForFixedUpdate = False, Boolean returnToStart = True, Boolean disableRendering = True, Single fixedDeltaTime = 0.02
        Your arguments: 'actionSimulationSeconds', 'fixedDeltaTime'
        Valid ways to call "RotateRight" action:
                Void RotateRight(Nullable`1 degrees = , Boolean manualInteract = False, Boolean forceAction = False, Single speed = 1, Boolean waitForFixedUpdate = False, Boolean returnToStart = True, Boolean disableRendering = True, Single fixedDeltaTime = 0.02)

If I comment out all instances of the actionSimulationSeconds, the issue goes away and the environment runs without error. However, I'm not sure what effect its removal will have on subsequent agent evaluations I will perform.

Is this a reasonable change for me to make in order to support Vulkan?

Thanks! Brandon

[Lucaweihs]

Hi @brandontrabucco,

That's a great question, I don't think simply removing actionSimulationSeconds is the best solution as this command ensured that certain actions had physically accurate results. I'll need to chat with the AI2-THOR team a bit to get their sense of the best way to accomplish this will be, I'll try to get an answer for you by tomorrow.

[brandontrabucco]

Thanks @Lucaweihs !

[Lucaweihs]

Hi @brandontrabucco,

Just wanted to keep you updated, we're currently working on making this possible but it will likely take to the end of this week (perhaps into early next week). In particular, I'm currently ramping things up for a 2022 version of our Rearrangement Challenge for which a formal announcement is planned for mid-Feb and I'm planning for this to include headless support. For some context, our support for Vulkan is still somewhat new so some aspects of the pipeline are still manual.

[brandontrabucco]

Thanks for the quick reply! Even if the setup is manual, and requires me to compile the system myself, I'd appreciate being given early access to the components needed to support Vulkan rendering, if that's feasible for your team :)

[Lucaweihs]

Hi @brandontrabucco,

The 2022 version of the challenge can be found here. This new version should default to using cloud rendering so long as no x-display is detected. We'll be doing an official release soon but thought you might want to this as quickly as possible.

[brandontrabucco]

Thanks!