search

kolbytn / mindcraft

MIT License
2.22k stars 267 forks source link

image capture with !look(x,y,z) #299

Open MaxRobinsonTheGreat opened 2 weeks ago

MaxRobinsonTheGreat commented 2 weeks ago

There is a way to take simple screenshots: https://github.com/PrismarineJS/mineflayer/blob/master/examples/screenshot-with-node-canvas-webgl/screenshot.js

They will be deeply flawed, but better than nothing. thought must go into the command itself, I like !look(x, y, z) to look at a specific block, as opposed to roll,pitch,yaw Also adding text info about the screenshot would be good, like raycast the type of block in the middle of its view, get coords and type, and describe that with the image. some sort of x,y,z coords and info to help ground the image in the gameworld

keneo commented 2 weeks ago

you can actually send the screenshot to openai newer models as part of the conversation with them. so this could improve the bot experience a lot

JurassikLizard commented 2 weeks ago

Maybe some sort of info about biomes/structures in the direction of the image? I think the biggest problem is that it needs to accomplish 2 very difficult tasks.

  1. Identify points of interest in the image (Hopefully advanced models will be capable of this)
  2. Figure out where these are in the game world/how to get there (I'm not sure how to solve this one)
Ninot1Quyi commented 2 weeks ago

@DerJanniku mabe you could submit a PR

DerJanniku commented 2 weeks ago

@Ninot1Quyi sure

DerJanniku commented 2 weeks ago

i submit the PR

MaxRobinsonTheGreat commented 1 week ago

yeah dont post all the code here