search

kolbytn / mindcraft

MIT License
1.98k stars 244 forks source link

image capture with !look(x,y,z) #299

Open MaxRobinsonTheGreat opened 1 week ago

MaxRobinsonTheGreat commented 1 week ago

There is a way to take simple screenshots: https://github.com/PrismarineJS/mineflayer/blob/master/examples/screenshot-with-node-canvas-webgl/screenshot.js

They will be deeply flawed, but better than nothing. thought must go into the command itself, I like !look(x, y, z) to look at a specific block, as opposed to roll,pitch,yaw Also adding text info about the screenshot would be good, like raycast the type of block in the middle of its view, get coords and type, and describe that with the image. some sort of x,y,z coords and info to help ground the image in the gameworld

keneo commented 6 days ago

you can actually send the screenshot to openai newer models as part of the conversation with them. so this could improve the bot experience a lot

JurassikLizard commented 6 days ago

Maybe some sort of info about biomes/structures in the direction of the image? I think the biggest problem is that it needs to accomplish 2 very difficult tasks.

  1. Identify points of interest in the image (Hopefully advanced models will be capable of this)
  2. Figure out where these are in the game world/how to get there (I'm not sure how to solve this one)
Ninot1Quyi commented 4 days ago

@DerJanniku mabe you could submit a PR

DerJanniku commented 4 days ago

@Ninot1Quyi sure

DerJanniku commented 4 days ago

i submit the PR

MaxRobinsonTheGreat commented 3 days ago

yeah dont post all the code here