search

mc-bench / orchestrator

Orchestrator to spin up MC server, connect mineflayer bot, generate + execute build commands and export the structure
MIT License
144 stars 9 forks source link

MVP Feature Set #1

Open nikshepsvn opened 3 weeks ago

nikshepsvn commented 3 weeks ago

wanted to document what I think a good MVP state would be for this repo:

  1. Spin up a unique server for a single LLM
  2. Verify server is running correctly
  3. Create a base for the LLM to build on
  4. Connect Mindcraft Agent to server
  5. Verify Mindcraft Agent can interface with server as expected
  6. Issue eval command to Agent (build building of X)
  7. Track/figure out building / agent complete state
  8. Dump the building as a NBT/Schematic or some saved file
  9. Run saved file through LLM-as-judge eval
  10. Repeat for other LLMs to get preliminary ranking/leaderboard
nikshepsvn commented 3 weeks ago

First 3 should be covered with the base code I've pushed up (might need some manual verification on client side, ie. connecting to server and seeing if things are working as expected), after that we can start from 4.

nikshepsvn commented 3 weeks ago

https://github.com/mc-bench/orchestrator/pull/2 should tackle 4 and 5

SaitejaUtpala commented 3 weeks ago

@nikshepsvn I am interested! Anything specific I can look into ?

nikshepsvn commented 1 week ago

@SaitejaUtpala sorry, saw this late! please join the discord in the README!