The specific human feedback on VOYAGER's construction of 3D buildings

[Felixvillas]

Hi~

I am utilizing Voyager to build some complex 3D buildings in Minecraft.

And I noticed you mentioned that Voyager is able to construct 3D structures like the Nether Portal, a house, and so on.

Could you provide the specific human critique on the Voyager's construction of 3D buildings?

[Felixvillas]

@xieleo5 Hi~ How should I design prompts to reproduce the results of building 3D structures from human feedback? Do I need to specify the coordinates of each block that makes up this 3D structure in the prompts?

[xieleo5]

Hi @Felixvillas, sorry for the late reply. The human feedback is integrated in two primary ways, detailed in our supplementary section:

1. Human as a critic: Humans provide a critique to Voyager, allowing it to modify the code from the previous round. For instance, if the agent is constructing a Nether Portal but makes a mistake in the dimensions, we advise GPT-4 in the following manner:
   • “Your portal does not have the correct size. For a 10-obsidian portal, the top and bottom edges should only have 2 blocks, and the two-side edge should have 3 blocks. The 4 corners of the portal should use another material such as cobblestone.”
2. Human as a curriculum: Humans break down complex building tasks into smaller steps, guiding Voyager to complete them incrementally. For example, when constructing a house, we guide Voyager step-by-step, from laying the foundation to adding windows and a door:
   • “First, don't think of any other things, you only need to build the main body using oak_log with size 5x5 and 3-block height. Start from offset Y = 0 level.”
   • “You just built a 5*5 shelter starting at (615, 79, 1206). Now you need to add some wood stairs above the shelter as decoration. Place 9 stairs on each corner, the center of each edge, and the center of the building.”
   • “You just built a 5*5 shelter starting at (615, 79, 1206). You need to add 8 windows and a door. The door should be in the middle of one of the walls. The windows should be on each wall, with 1 block above the ground and 1 block between them.”

[Felixvillas]

Hi @Felixvillas, sorry for the late reply. The human feedback is integrated in two primary ways, detailed in our supplementary section:

1. Human as a critic: Humans provide a critique to Voyager, allowing it to modify the code from the previous round. For instance, if the agent is constructing a Nether Portal but makes a mistake in the dimensions, we advise GPT-4 in the following manner:

   • “Your portal does not have the correct size. For a 10-obsidian portal, the top and bottom edges should only have 2 blocks, and the two-side edge should have 3 blocks. The 4 corners of the portal should use another material such as cobblestone.”

2. Human as a curriculum: Humans break down complex building tasks into smaller steps, guiding Voyager to complete them incrementally. For example, when constructing a house, we guide Voyager step-by-step, from laying the foundation to adding windows and a door:

   • “First, don't think of any other things, you only need to build the main body using oak_log with size 5x5 and 3-block height. Start from offset Y = 0 level.”
   • “You just built a 5*5 shelter starting at (615, 79, 1206). Now you need to add some wood stairs above the shelter as decoration. Place 9 stairs on each corner, the center of each edge, and the center of the building.”
   • “You just built a 5*5 shelter starting at (615, 79, 1206). You need to add 8 windows and a door. The door should be in the middle of one of the walls. The windows should be on each wall, with 1 block above the ground and 1 block between them.”

Thanks for your detailed reply! It helps me. I'll summarize the problems discovered during my experiments under this issue.

[github-actions[bot]]

This issue is stale because it has been open for 30 days with no activity.

[github-actions[bot]]

This issue was closed because it has been inactive for 14 days since being marked as stale.