search

allenai / ai2thor

An open-source platform for Visual AI.
http://ai2thor.allenai.org
Apache License 2.0
1.15k stars 215 forks source link

Can I open and edit the scenes from procthor-10k dataset in Unity.editor? #1128

Open MingZwhy opened 1 year ago

MingZwhy commented 1 year ago

Now I can edit the scenes in ai2thor in Unity.editor, but I don't know how to edit scenes in procthor-10k dataset in Unity.editor. I guess the root of the problem is that how to build a unity scene file from a scene.json

winthos commented 1 year ago

So the challenge of editing a procthor scene in-editor is that all procthor scenes are generated at runtime, so they are not saved or cached in a way that you can open up a usual Unity scene.

The best way to "open" a procthor scene in the editor is to take the json and place it in the directory Assets/Resources/rooms

Then, go to Assets/Scenes/Procedural and open up the Procedural.unity scene.

While in this scene, hit Play and then navigate your cursor to the text box that shows up to the bottom left of the Game window that reads "Enter text..."

This box allows actions to be executed directly in the editor without needing to link to the python controller, and all actions are defined in the script DebugInputField.cs located in Assets/Scripts

To start, type initp into this box to initialize a default agent and press enter to execute. Then, type chp <json file's name> and hit enter again to generate a house based on the json file you saved to the Assets/Resources/rooms directory.

From here, you will be able to see and manipulate the objects created based on the json, however editing them here will not save the json. You will need to then go back into the json and edit any new values like changed object positions, by taking the values of manipulated objects in the editor and copying them over into the json itself.

MingZwhy commented 1 year ago

Thank you very much for your guidance！I "open" the procthor scene in unity.editor successfully following your methods.

YandanYang commented 1 year ago

Hi. I really appreciate this question and your answer. However, when I type chp <json file's name>, I got the error Action failed with error message 'NullReferenceException: Object reference not set to an instance of an object.trace : at UnityStandardAssets.Characters.FirstPerson.BaseFPSAgentController.CreaterHouse).... It happens with every json file. Can you help with this? I do not know what's wrong or what should I do.

MingZwhy commented 1 year ago

Hi. I really appreciate this question and your answer. However, when I type chp <json file's name>, I got the error Action failed with error message 'NullReferenceException: Object reference not set to an instance of an object.trace : at UnityStandardAssets.Characters.FirstPerson.BaseFPSAgentController.CreaterHouse).... It happens with every json file. Can you help with this? I do not know what's wrong or what should I do.

You can try to open it using unity editor 2021.3.11f1c1, I found same error message when I try to open it using edition 2020，my solution is to git clone the repo again and using unity editor 2021.3.11f1c1.

YandanYang commented 1 year ago

I have tried the version 2021.3.11f1c1 as well as 2020.3.25f1 LTS in Linux and Windows. Sadly, none of these work.

JiMa25 commented 1 year ago

Thank you! I open the procthor scene in unity.editor successfully ，now,I want to save the scene as an Asset file,how to make it? @winthos

winthos commented 1 year ago

Thank you! I open the procthor scene in unity.editor successfully ，now,I want to save the scene as an Asset file,how to make it? @winthos

The ProcTHOR scenes are all generated dynamically at runtime based on the parameters specified in the json for each scene. Because of this, there is no way to save a scene in the traditional sense as a Unity scene file as the scene itself only exists at runtime. Any changes you want to make in the Unity Editor you will need to log and then make those same updates to the json for that scene in order to "save" your changes.

Singh-sid930 commented 1 year ago

has anyone able to figure out the correct Editor version for this ? I am using linux and trying the 2021.3.11f1 version. It still throws the same error

System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.NullReferenceException: Object reference not set to an instance of an object at UnityStandardAssets.Characters.FirstPerson.BaseFPSAgentController.CreateHouse (Thor.Procedural.Data.ProceduralHouse house) [0x00035] in /home/ANT.AMAZON.COM/hartsid/workspace/progen/src/ai2thor/unity/Assets/Scripts/BaseFPSAgentController.cs:6000 at (wrapper managed-to-native) System.Reflection.RuntimeMethodInfo.InternalInvoke(System.Reflection.RuntimeMethodInfo,object,object[],System.Exception&) at System.Reflection.RuntimeMethodInfo.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x0006a] in <12d27ebfb46146b18b40876234894c77>:0 --- End of inner exception stack trace --- at System.Reflection.RuntimeMethodInfo.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x00083] in <12d27ebfb46146b18b40876234894c77>:0 at System.Reflection.MethodBase.Invoke (System.Object obj, System.Object[] parameters) [0x00000] in <12d27ebfb46146b18b40876234894c77>:0 at ActionDispatcher.Dispatch (System.Object target, DynamicServerAction dynamicServerAction) [0x001f4] in /home/ANT.AMAZON.COM/hartsid/workspace/progen/src/ai2thor/unity/Assets/Scripts/ActionDispatcher.cs:381 at UnityStandardAssets.Characters.FirstPerson.BaseFPSAgentController.ProcessControlCommand (DynamicServerAction controlCommand, System.Object target) [0x0007d] in /home/ANT.AMAZON.COM/hartsid/workspace/progen/src/ai2thor/unity/Assets/Scripts/BaseFPSAgentController.cs:1977 UnityEngine.Debug:Log (object) UnityStandardAssets.Characters.FirstPerson.BaseFPSAgentController:ProcessControlCommand (DynamicServerAction,object) (at Assets/Scripts/BaseFPSAgentController.cs:2022) UnityStandardAssets.Characters.FirstPerson.BaseFPSAgentController:ProcessControlCommand (DynamicServerAction) (at Assets/Scripts/BaseFPSAgentController.cs:1962) UnityStandardAssets.Characters.FirstPerson.DebugInputField:Execute (string) (at Assets/Scripts/DebugInputField.cs:4018) UnityEngine.EventSystems.EventSystem:Update () (at Library/PackageCache/com.unity.ugui@1.0.0/Runtime/EventSystem/EventSystem.cs:501)

Singh-sid930 commented 1 year ago

Also getting this error when running the Procedural.Unity scene Socket exception: System.Net.Sockets.SocketException (0x80004005): Connection refused at System.Net.Sockets.Socket.Connect (System.Net.EndPoint remoteEP) [0x000b6] in <fa31ff7c1e6a4d0894ebb6fee0a2ea37>:0 at AgentManager+<EmitFrame>d__91.MoveNext () [0x001ab] in /home/ANT.AMAZON.COM/hartsid/workspace/progen/src/ai2thor/unity/Assets/Scripts/AgentManager.cs:1045

Singh-sid930 commented 1 year ago

seems like the default jsons present in the code do not work. including the json from the 10k dataset do.

Inshsang commented 1 year ago

@winthos hello，we need an indoor point cloud dataset for our recent task, and I wonder Is it possible to export the scene when Unity is running, using some of the Unity operations?I look forward to your reply.