Using new materials and objects

[ZheningHuang]

Hi, I have a question about adding new materials and objects into HOLODECK/AI2THOR for retrieval:

For materials:

I tried to add new materials in the local ai2thor asset folder:

ai2thor/unity/Assets

and changed the material names of walls in the JSON file to make corresponding changes. However, such changes create errors, and the scene JSON file cannot connect to Unity when running

python connect_to_unity.py --scene 

Some other attempts:

• Created a new materials file in the asset folder without modifying the JSON file --> Failed to open Scene in JSON file.
• Did not add anything in Asset folder and Changed the material names of walls to some existing materials in Asset folder in Unity --> Worked OK.

I don't know if this sounds expected to you. It would be great if you could provide some guidance here.

Also, regarding adding more objects, I can see from this question (https://github.com/allenai/Holodeck/issues/14#issuecomment-1889974493) that this requires converting objects into THOR format (pkl.gz). It would be great if you could also provide some guidance.

Thanks a lot!

[YueYANG1996]

Hi,

Could you refer to objathor for how to add new assets to AI2-THOR? Especially here.

[ZheningHuang]

Thanks for your reply. I still did not get this work though:

1. In the script you sent here, the extension is --extension=.msgpack.gz, but all Objvaeser assets in 09_23_combine_scale are in the format of pkl.gz. Do I simply change it to --extension=pkl.gz?
2. I am not sure how I can make the newly generated object importable into AI2THor. I did a pilot test by changing the name of the existing object in the 09_23_combine_scale folder to a new name, including all the names that occurred in the metadata.json .

When I tried to load this in Unity, it reports that this asset is not in the database. .

(I'm not an expert in C#, but I did try to debug it a bit, and still did not find the problem. Any suggestion would be valuable.)

When I tried to import the object I generated in step 1 into Unity (using pkl.gz),

.

it reported an error directly:

Traceback (most recent call last):
  File "/Users/zhao/Documents/Holodeck/connect_to_unity.py", line 32, in <module>
    controller.step(action="CreateHouse", house=scene)
  File "/Users/zhao/miniconda3/envs/holodeck/lib/python3.9/site-packages/ai2thor/controller.py", line 1064, in step
    self.run_action_hook(action)
  File "/Users/zhao/miniconda3/envs/holodeck/lib/python3.9/site-packages/ai2thor/controller.py", line 995, in run_action_hook
    event = method(action, self)
  File "/Users/zhao/miniconda3/envs/holodeck/lib/python3.9/site-packages/ai2thor/hooks/procedural_asset_hook.py", line 97, in CreateHouse
    return create_assets_if_not_exist(
  File "/Users/zhao/miniconda3/envs/holodeck/lib/python3.9/site-packages/ai2thor/hooks/procedural_asset_hook.py", line 54, in create_assets_if_not_exist
    evt = create_asset(
  File "/Users/zhao/miniconda3/envs/holodeck/lib/python3.9/site-packages/ai2thor/util/runtime_assets.py", line 165, in create_asset
    evt = controller.step(**create_prefab_action)
  File "/Users/zhao/miniconda3/envs/holodeck/lib/python3.9/site-packages/ai2thor/controller.py", line 1105, in step
    raise ValueError(self.last_event.metadata["errorMessage"])
ValueError: Invalid action: 

Any suggestion would be helpful!

Best, Zhening

[YueYANG1996]

How did you load the asset in Unity? Here are the steps:

1. convert to .glb to the thor format.
2. put the folder into the 09_23_combine_scale folder
3. grab a scene json file and add this new object (using its assetID) to the "objects" list (you can manually do this by assigning a random position, rotation and id).
4. load the scene using the steps here

[ZheningHuang]

Hi, thanks a lot for your reply.

Finally, I got this to work. The issue I had before was:

The assets generated by objathor contained additional keys in the pkl.gz file, seemingly unreadable by AI2Thor, resulting in errors.

To resolve this, I compared the keys of pkl.gz file generated by objathor with those in the 09_23_combine_scale folder. After identifying and deleting the additional keys, the assets could be successful!!

Best, Zhening

[DDDDorwin]

Thanks for the previous answer. But I'm still stuck at the last step of loading a new object into the scene. I would really appreciate it if anyone could provide more help.

I managed to convert .glb file into Thor format using Objathor pipeline. And got the following files in the folder:

And then I put this folder into the _09_23_combinescale folder. And following the above tips, I removed the redundant files, and only kept.pkl.gz, albedo, emission, normal and thor_metadata.

In order to load it into a scene, I add the following part to the "objects" list in the json file of the scene. The name of the folder generated by Objathor is _40562244original , and I manually added an id, position, and other info for it.

    {
        "assetId": "40562244_original",
        "id": "ikeacoffeetable-0 (living room)",
        "kinematic": true,
        "position": {
            "x": 4.575476661719379,
            "y": 0.40774148810617195,
            "z": 3.2
        },
        "rotation": {
            "x": 0,
            "y": 270,
            "z": 0
        },
        "material": null,
        "roomId": "living room",
        "vertices": [
            [
                210.5953323438757,
                106.59884693898007
            ],
            [
                210.5953323438757,
                533.40115306102
            ],
            [
                404.5,
                533.40115306102
            ],
            [
                404.5,
                106.59884693898007
            ]
        ],
        "object_name": "ikeacoffeetable-0",
        "layer": "Procedural0"
    },

However, after I connect the JSON file to Unity, I got the following error: (holodeck) dorwin@dorwin-HP-Z4-G5-Workstation-Desktop-PC:~/Holodeck$ python connect_to_unity.py --scene /home/dorwin/Holodeck/data/scenes/a_living_room-2024-05-28-10-58-41-940166/a_living_room.json Traceback (most recent call last): File "/home/dorwin/Holodeck/connect_to_unity.py", line 32, in <module> controller.step(action="CreateHouse", house=scene) File "/home/dorwin/miniconda3/envs/holodeck/lib/python3.9/site-packages/ai2thor/controller.py", line 1063, in step self.run_action_hook(action) File "/home/dorwin/miniconda3/envs/holodeck/lib/python3.9/site-packages/ai2thor/controller.py", line 994, in run_action_hook event = method(action, self) File "/home/dorwin/miniconda3/envs/holodeck/lib/python3.9/site-packages/ai2thor/hooks/procedural_asset_hook.py", line 88, in CreateHouse return create_assets_if_not_exist( File "/home/dorwin/miniconda3/envs/holodeck/lib/python3.9/site-packages/ai2thor/hooks/procedural_asset_hook.py", line 48, in create_assets_if_not_exist evt = create_asset( File "/home/dorwin/miniconda3/envs/holodeck/lib/python3.9/site-packages/ai2thor/util/runtime_assets.py", line 160, in create_asset evt = controller.step(**create_prefab_action) File "/home/dorwin/miniconda3/envs/holodeck/lib/python3.9/site-packages/ai2thor/controller.py", line 1104, in step raise ValueError(self.last_event.metadata["errorMessage"]) ValueError: Invalid action:

Can anyone tell me what the problem is? Thanks for any kind of help:)