The place_object keeps running forever

[toannguyen1904]

I'm trying to run your test.py file. Everything went well but I'm currently stuck at performing the place_object() function. As I can observe, the function is a recursion one, which calls itself for the children. My problem is that the function keeps executing forever, without any error. Can you give me some explanations and solutions to address this.

Thank you so much!

[atcelen]

Hello @toannguyen1904 , thank you for your interest in the project! The reason for the place_object() running forever is most probably some spatial implausibility in the generated scene graph. The Layout Corrector is in charge of fixing such errors, but the behaviour of the LLMs are hard to predict and catching all implausibilities has been a dauting task. I have been experimenting lately with the pipeline as well and I have locally added some more guardrails to minimize such non-exiting behavior, which I will commit very shortly, within this week, along with some more updates.

For the moment, I would suggest you to:

1. Deleting the .cache file to generate the scene from scratch for the same prompt.
2. Decrease the number of objects

This is a shortcoming of this work currently that I am trying to improve.

[toannguyen1904]

Thank you @atcelen, I am looking forward to your updates.

[atcelen]

Hello @toannguyen1904 , I have uploaded some changes to the codebase. Feel free to set verbose=True for better debugging of each step!

[ForgeV]

Hello guys @toannguyen1904, @atcelen i have a same problem with place_object func, and fetching changes from yesterday update doesn't help

[toannguyen1904]

Hello guys @toannguyen1904, @atcelen i have a same problem with place_object func, and fetching changes from yesterday update doesn't help

Hi @ForgeV,

I am currently investigating the source code in detail. I'll share more information as I identify any shortcomings (such as why the code might run indefinitely) and possibly suggest some improvements. Personally, I find this work quite impressive, especially given the novelty of the multi-agent framework, the qualitative outcomes and the significant effort @atcelen has put into it. However, I believe the implementation is still a bit underdeveloped and could benefit from further enhancements. It would also be great if @atcelen could update the code progressively.

Best regards.

[atcelen]

Hello all,

Thanks for your interest in this project and your contributions with these comments! I will continue working towards adding some improvements by resolving errors that cause the place_object function to run indefinitely. I would appreciate it if you could upload the scene graphs that the agents output using i_design.to_json(). The scene graphs would help me understand where the issues are occuring and add fixes for them.

One potential solution is to prune objects from the scene graph that cannot be placed in the scene during the runtime of place_object. This approach should lead to better termination, although it will result in fewer objects being included in the scene.

Best

[ForgeV]

Hello all,

scene_graph.json

Thanks for your interest in this project and your contributions with these comments! I will continue working towards adding some improvements by resolving errors that cause the place_object function to run indefinitely. I would appreciate it if you could upload the scene graphs that the agents output using i_design.to_json(). The scene graphs would help me understand where the issues are occuring and add fixes for them.

One potential solution is to prune objects from the scene graph that cannot be placed in the scene during the runtime of place_object. This approach should lead to better termination, although it will result in fewer objects being included in the scene.

Best

I also thought about removing some objects directly from scene_graph, but I couldn't find a moment in which to embed the deletion function and it seems difficult to me to determine which of the objects really interferes and cannot be adequately placed, and which one just hasn't found an adequate position yet and requires rearrangement. Also, I can't get the scene_graph, since it is received in the last function in test.py and the code loops endlessly in the backtrack function

Updated: I was able to get the file scene_graph

[toannguyen1904]

Hi @atcelen, I don't get why the process of json schema validation is performed in the overridden get_human_input(). As I can observe, the previous speakers of json_schema_debugger are always instances of AssistantAgent. To the best of my understandings, get_human_input() is used to deal with input from human via UserProxyAgent. Am I correct? It would be awesome if you could give some explanation on this point.

[atcelen]

Hello,

Yes, that is correct. It was more convenient for me to define the JSONSchemaAgent class by inheriting UserProxyAgent. As you correctly point out, this class is mainly used for human interactions. This function previously gets the human reply from the input streams and returns it as feedback(See here). My aim was to override this by an algorithmic feedback depending on the error type. I hope this clarifies your question.