search

rese1f / STEVE

⛏💎 STEVE in Minecraft is for See and Think: Embodied Agent in Virtual Environment
https://rese1f.github.io/STEVE/
MIT License
28 stars 1 forks source link

how to use Steve to directly describe a minecraft video? #2

Open gqsmmz opened 9 months ago

gqsmmz commented 9 months ago

hello! I'd like to ask if I have a mp4.video about minecraft, wanting to use steve language model to descrive the video's content, how can I complete it.

after I set the code like following, how can I do next? If I can receive the reply, I'd be extraordinarily appreciated it!!!

self.llm = ChatOpenAI( model_name="STEVE-7B", temperature=temperature, request_timeout=request_timout, )

rese1f commented 9 months ago

U can start with running the whole framework correctly, and the sequence description can be found in log.

gqsmmz commented 9 months ago

U can start with running the whole framework correctly, and the sequence description can be found in log.

ok! I will try it!

gqsmmz commented 9 months ago

I'm afraid that there's no "STEVE/env/mineflayer", does it mean the content in "https://github.com/PrismarineJS/mineflayer"?

and there's no instructions in the link of "Fabric Mods Install"

Thanks for your time!

gqsmmz commented 9 months ago

I'm afraid that in your llama-7B, there're missing some parameters as follows, they are about "inv_freq". if using the llama-7B to replace the LLAMA model in other models like "moviechat" will generate problems? Because I directly replaced the Llama model with the Steve 7b model, I still can't get a language model that can recognize Minecraft.

Some weights of LlamaForCausalLM were not initialized from the model checkpoint at ckpt/LLaMA/steve and are newly initialized: ['model.layers.28.self_attn.rotary_emb.inv_freq', 'model.layers.31.self_attn.rotary_emb.inv_freq', 'model.layers.9.self_attn.rotary_emb.inv_freq', 'model.layers.30.self_attn.rotary_emb.inv_freq', 'model.layers.27.self_attn.rotary_emb.inv_freq', 'model.layers.4.self_attn.rotary_emb.inv_freq', 'model.layers.25.self_attn.rotary_emb.inv_freq', 'model.layers.23.self_attn.rotary_emb.inv_freq', 'model.layers.3.self_attn.rotary_emb.inv_freq', 'model.layers.1.self_attn.rotary_emb.inv_freq', 'model.layers.8.self_attn.rotary_emb.inv_freq', 'model.layers.21.self_attn.rotary_emb.inv_freq', 'model.layers.10.self_attn.rotary_emb.inv_freq', 'model.layers.17.self_attn.rotary_emb.inv_freq', 'model.layers.15.self_attn.rotary_emb.inv_freq', 'model.layers.13.self_attn.rotary_emb.inv_freq', 'model.layers.5.self_attn.rotary_emb.inv_freq', 'model.layers.24.self_attn.rotary_emb.inv_freq', 'model.layers.20.self_attn.rotary_emb.inv_freq', 'model.layers.6.self_attn.rotary_emb.inv_freq', 'model.layers.19.self_attn.rotary_emb.inv_freq', 'model.layers.12.self_attn.rotary_emb.inv_freq', 'model.layers.16.self_attn.rotary_emb.inv_freq', 'model.layers.26.self_attn.rotary_emb.inv_freq', 'model.layers.7.self_attn.rotary_emb.inv_freq', 'model.layers.29.self_attn.rotary_emb.inv_freq', 'model.layers.0.self_attn.rotary_emb.inv_freq', 'model.layers.11.self_attn.rotary_emb.inv_freq', 'model.layers.22.self_attn.rotary_emb.inv_freq', 'model.layers.18.self_attn.rotary_emb.inv_freq', 'model.layers.2.self_attn.rotary_emb.inv_freq', 'model.layers.14.self_attn.rotary_emb.inv_freq'] You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.

ALLISWELL8 commented 8 months ago

I have the same question, the readme for this project did not clearly explain how to proceed. I hope the author can improve it. Thank you to the author.

WZX0Swarm0Robotics commented 7 months ago

I have the same question, the readme for this project did not clearly explain how to proceed. I hope the author can improve it. Thank you to the author.

kiminomiku commented 4 months ago

Bro, have you successfully run it? It seems that the project is missing some content(STEVE/env). I hope the author can provide this part. Thank you to the author.