search

OpenGVLab / InternVideo

[ECCV2024] Video Foundation Models & Data for Multimodal Understanding
Apache License 2.0
1.3k stars 85 forks source link

Demo script for Internvideo2 model for Video Question Answering or Summarization task #150

Open Varun-GP opened 1 month ago

Varun-GP commented 1 month ago

Dear Authors, How can I use the Internvideo2 model for Video Question Answering or Summarization tasks given a video? Please provide a demo script if any for testing on new videos.

Thanks.

WeiminLee commented 3 weeks ago

I can not successfully run the demo code posted on huggingface. I did not know why.

code: https://huggingface.co/OpenGVLab/InternVideo2_chat_8B_HD/blob/main/demo.py

error: File "/data/lwm-data/AwesomeCode/InternVideo2-Chat/demo.py", line 19, in <module> model = AutoModel.from_pretrained( File "/data/lwm-data/lwm-env/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py", line 559, in from_pretrained return model_class.from_pretrained( File "/data/lwm-data/lwm-env/lib/python3.10/site-packages/transformers/modeling_utils.py", line 3482, in from_pretrained raise EnvironmentError( OSError: Error no file named pytorch_model.bin, model.safetensors, tf_model.h5, model.ckpt.index or flax_model.msgpack found in directory /data/model_hub/InterVideo2_chat.

yinanhe commented 3 weeks ago

I can not successfully run the demo code posted on huggingface. I did not know why.

code: https://huggingface.co/OpenGVLab/InternVideo2_chat_8B_HD/blob/main/demo.py

error: File "/data/lwm-data/AwesomeCode/InternVideo2-Chat/demo.py", line 19, in <module> model = AutoModel.from_pretrained( File "/data/lwm-data/lwm-env/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py", line 559, in from_pretrained return model_class.from_pretrained( File "/data/lwm-data/lwm-env/lib/python3.10/site-packages/transformers/modeling_utils.py", line 3482, in from_pretrained raise EnvironmentError( OSError: Error no file named pytorch_model.bin, model.safetensors, tf_model.h5, model.ckpt.index or flax_model.msgpack found in directory /data/model_hub/InterVideo2_chat.

What's your transformers version?

yinanhe commented 3 weeks ago

You can just follow the readme of InternVideo2-Chat

WeiminLee commented 3 weeks ago

I can not successfully run the demo code posted on huggingface. I did not know why. code: https://huggingface.co/OpenGVLab/InternVideo2_chat_8B_HD/blob/main/demo.py error: File "/data/lwm-data/AwesomeCode/InternVideo2-Chat/demo.py", line 19, in <module> model = AutoModel.from_pretrained( File "/data/lwm-data/lwm-env/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py", line 559, in from_pretrained return model_class.from_pretrained( File "/data/lwm-data/lwm-env/lib/python3.10/site-packages/transformers/modeling_utils.py", line 3482, in from_pretrained raise EnvironmentError( OSError: Error no file named pytorch_model.bin, model.safetensors, tf_model.h5, model.ckpt.index or flax_model.msgpack found in directory /data/model_hub/InterVideo2_chat.

What's your transformers version?

Thanks, I have successfully run this demo code. Because I can only run it on local linux server. I changed a lot of configurations.

In particular, the model paths for Q-fomer, Bert-base-uncased and Mistral-7B

yinanhe commented 3 weeks ago

I can not successfully run the demo code posted on huggingface. I did not know why. code: https://huggingface.co/OpenGVLab/InternVideo2_chat_8B_HD/blob/main/demo.py error: File "/data/lwm-data/AwesomeCode/InternVideo2-Chat/demo.py", line 19, in <module> model = AutoModel.from_pretrained( File "/data/lwm-data/lwm-env/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py", line 559, in from_pretrained return model_class.from_pretrained( File "/data/lwm-data/lwm-env/lib/python3.10/site-packages/transformers/modeling_utils.py", line 3482, in from_pretrained raise EnvironmentError( OSError: Error no file named pytorch_model.bin, model.safetensors, tf_model.h5, model.ckpt.index or flax_model.msgpack found in directory /data/model_hub/InterVideo2_chat.

What's your transformers version?

Thanks, I have successfully run this demo code. Because I can only run it on local linux server. I changed a lot of configurations.

In particular, the model paths for Q-fomer, Bert-base-uncased and Mistral-7B

Yes, you need to ensure that your local has the cache of Bert-base-uncased's config and Mistral-7B-v0.3's config. Thank you for your feedback.