Compatibility of "objectType" in AI2THOR

[BlackJ36]

Hi,it is quite impressive to see the scene can be load in AI2THOR, which is soloved in #issues17.However, the objectType is always undefined.Is there any possible way to be competible with the objectType in AI2THOR? Even I tried to modified the objectType, It seem to be still undefined after a controller.step().It can be significant for me to develop in the whole picture of your team.

[YueYANG1996]

@Lucaweihs Could you help to answer this?

[Lucaweihs]

Hi @BlackJ36,

Can I ask how you'd like objectType to work? Do you just want to have the type returned with the object metadata or is there some AI2-THOR action that you want to use that uses the objectType as input?

@AlvaroHG @winthos - how hard would it be to let people specify the object type as a string (e.g. on the call to CreateObjectPrefab)?

[BlackJ36]

@Lucaweihs Hi, thanks to your reply. To be specific, I want to create some item small object in holodeck. The item objectType may be different from what I defined, so I have to try to modify the objectType. Then I want to execute the function in RoboThor like get_shortest_path_to_object_type which take objectType as the parameter.

Or I just want to place all the book type (the type I specify) objecect randomly by PlaceObjectAtPoint and GetSpawnCoordinatesAboveReceptacle.

[Lucaweihs]

Got it, so the get_shortest_path_to_object_type function won't work for now but you should be able to implement something similar by specifying the objectId and using the GetShortestPath action:

controller.step(
  action="GetShortestPath", 
  objectId=object_id,
  position=agent_position,
)

Similarly, for the moment you'll have to manually track which object ids correspond to books and use that for your PlaceObjectAtPoint/GetSpawnCoordinatesAboveReceptacle applications.

We'll look into supporting custom object types more natively in AI2-THOR but for now it's probably best to work around it.

[BlackJ36]

Sure, I have tried get_shortest_path_to_object which implement in the /utils/metrics . There is another trouble I met.

ValueError: Unable to find shortest path for objectId 'tv stand-0 (living room)' due to error 'InvalidOperationException: No point on NavMesh near startPosition (3.5, 0.0, 2.0).

I plot the ReachablePosition of robot, it seems like the initialization of the agent encounter an error.Thus, it can't find the waypoint in the function get_shortest_path_to_object.Further more, the robot can not finish the movement thought the Action like Teleport and MoveAhead.

Is there any possible way to solve this problem? I have check the API document but still in stuck.

[YueYANG1996]

Could you try initializing the agent in a different position without collision with objects?

[Lucaweihs]

Thanks for the picture, that's very helpful! Yes, as @YueYANG1996 said, you'll need to teleport the agent to a collision-free starting position. Here's some code I've used in the past for this (for ProcTHOR houses but I believe the same thing should work here); in particular, see the try_find_collision_free_starting_position function. After teleporting, you might need to also run controller.step("BakeNavMesh").

from typing import Dict, Any

import ai2thor.controller
from shapely import Polygon
from shapely.ops import triangulate

def get_rooms_polymap(house: Dict[str, Any]):
    room_poly_map = {}

    # NOTE: Map the rooms
    for i, room in enumerate(house["rooms"]):
        room_poly_map[room["id"]] = Polygon(
            [(p["x"], p["z"]) for p in room["floorPolygon"]]
        )

    return room_poly_map

def get_candidate_points_in_room(
    room_id: str,
    room_poly_map: Dict[str, Polygon],
):
    polygon = room_poly_map[room_id]

    room_triangles = triangulate(polygon)

    candidate_points = [
        ((t.centroid.x, t.centroid.y), t.area) for t in room_triangles  # type:ignore
    ]

    # We sort the triangles by size so we try to go to the center of the largest triangle first
    candidate_points.sort(key=lambda x: x[1], reverse=True)
    candidate_points = [p[0] for p in candidate_points]

    # The centroid of the whole room polygon need not be in the room when the room is concave. If it is,
    # let's make it the first point we try to navigate to.
    if polygon.contains(polygon.centroid):
        candidate_points.insert(0, (polygon.centroid.x, polygon.centroid.y))

    return candidate_points

def try_find_collision_free_starting_position(
    house: Dict[str, Any],
    controller: ai2thor.controller.Controller,
    room_poly_map: Dict[str, Polygon],
):
    teleport_success = False
    for room_id in sorted(room_poly_map.keys()):
        candidate_points = get_candidate_points_in_room(room_id, room_poly_map)
        for cand in candidate_points:
            event = controller.step(
                action="TeleportFull",
                position={
                    "x": float(cand[0]),
                    "y": house["metadata"]["agent"]["position"]["y"],
                    "z": float(cand[1]),
                },
                rotation=house["metadata"]["agent"]["rotation"],
                standing=True,
                horizon=30,
            )
            if event:
                teleport_success = True
                break

        if teleport_success:
            break

    if teleport_success:
        return True
    else:
        return False

[BlackJ36]

Thanks for your assistance. I tried your method in the scene, unfortunatly it didn't work well. The result of try_find_collision_free_starting_position is False .

I think the problem I met can only be solved in the initalization instead of the postproc ess? The teleoport action can't be done although there are three candidate points returned by get_candidate_points_in_room.

By the way, the function return False in the scene generated by the ProcThor's example.py. So, I think it happens ocasionally, I tested my other scene generated by Holodeck and Robot work well. Thank's for your detailed reply again!

Attachment: The Trouble a_living_room_with_a_book.json

[YueYANG1996]

You could update the JSON file to make sure the robot is in a collision-free position. https://github.com/allenai/Holodeck/blob/7094a014acc6ad0c8ddb9a739e3760940e1f3e21/modules/empty_house.json#L6