search

mbzuai-oryx / Video-ChatGPT

[ACL 2024 🔥] Video-ChatGPT is a video conversation model capable of generating meaningful conversation about videos. It combines the capabilities of LLMs with a pretrained visual encoder adapted for spatiotemporal video representation. We also introduce a rigorous 'Quantitative Evaluation Benchmarking' for video-based conversational models.
https://mbzuai-oryx.github.io/Video-ChatGPT
Creative Commons Attribution 4.0 International
1.05k stars 92 forks source link

Fail to run Video-ChatGPT Demo Offline #50

Closed JuanJia closed 5 months ago

JuanJia commented 10 months ago

Thank you for sharing the good work!

I followed "offline_demo.md" to run offline, but website has no respones.

The terminal shows below. What does line 10 means? What error occurred?

$ python video_chatgpt/demo/video_demo.py --model-name /home/nkd/Documents/jjy/comment_generator/Video-ChatGPT/LLaVA-Lightning-7B-v1-1 --projection_path /home/nkd/Documents/jjy/comment_generator/Video-ChatGPT/video_chatgpt-7B.bin
2023-09-07 14:10:24 | INFO | gradio_web_server | args: Namespace(host='0.0.0.0', port=None, controller_url='http://localhost:210001', concurrency_count=8, model_list_mode='once', share=False, moderate=False, embed=False, model_name='/home/nkd/Documents/jjy/comment_generator/Video-ChatGPT/LLaVA-Lightning-7B-v1-1', vision_tower_name='openai/clip-vit-large-patch14', conv_mode='video-chatgpt_v1', projection_path='/home/nkd/Documents/jjy/comment_generator/Video-ChatGPT/video_chatgpt-7B.bin')
2023-09-07 14:10:24 | INFO | gradio_web_server | Namespace(host='0.0.0.0', port=None, controller_url='http://localhost:210001', concurrency_count=8, model_list_mode='once', share=False, moderate=False, embed=False, model_name='/home/nkd/Documents/jjy/comment_generator/Video-ChatGPT/LLaVA-Lightning-7B-v1-1', vision_tower_name='openai/clip-vit-large-patch14', conv_mode='video-chatgpt_v1', projection_path='/home/nkd/Documents/jjy/comment_generator/Video-ChatGPT/video_chatgpt-7B.bin')
You are using the default legacy behaviour of the <class 'transformers.models.llama.tokenization_llama.LlamaTokenizer'>. If you see this, DO NOT PANIC! This is expected, and simply means that the `legacy` (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set `legacy=False`. This should only be set if you understand what it means, and thouroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565
You are using a model of type llava to instantiate a model of type VideoChatGPT. This is not supported for all configurations of models and can yield errors.
Loading checkpoint shards:   0%|                                        | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:  50%|████████████████                | 1/2 [00:04<00:04,  4.19s/it]
Loading checkpoint shards: 100%|████████████████████████████████| 2/2 [00:05<00:00,  2.68s/it]
Loading checkpoint shards: 100%|████████████████████████████████| 2/2 [00:05<00:00,  2.90s/it]
2023-09-07 14:10:30 | ERROR | stderr | 
You are resizing the embedding layer without providing a `pad_to_multiple_of` parameter. This means that the new embedding dimension will be 32006. This might induce some performance reduction as *Tensor Cores* will not be available. For more details about this, or help on choosing the correct value for resizing, refer to this guide: https://docs.nvidia.com/deeplearning/performance/dl-performance-matrix-multiplication/index.html#requirements-tc

'NOTE: Please make sure you press the ‘Upload Video’ button and wait for it to display 'Start Chatting' before submitting question to Video-ChatGPT.' But Start Chatting button always be gray.

2023-09-07 14:10:30 | ERROR | stderr | 
You are resizing the embedding layer without providing a `pad_to_multiple_of` parameter. This means that the new embedding dimension will be 32006. This might induce some performance reduction as *Tensor Cores* will not be available. For more details about this, or help on choosing the correct value for resizing, refer to this guide: https://docs.nvidia.com/deeplearning/performance/dl-performance-matrix-multiplication/index.html#requirements-tc
2023-09-07 14:10:48 | INFO | stdout | Loading weights from /home/nkd/Documents/jjy/comment_generator/Video-ChatGPT/video_chatgpt-7B.bin
2023-09-07 14:10:49 | INFO | stdout | Weights loaded from /home/nkd/Documents/jjy/comment_generator/Video-ChatGPT/video_chatgpt-7B.bin
2023-09-07 14:10:55 | INFO | stdout | Initialization Finished
2023-09-07 14:10:56 | ERROR | stderr | /home/nkd/miniconda3/envs/video_chatgpt/lib/python3.10/site-packages/gradio/deprecation.py:43: UserWarning: You have unused kwarg parameters in Markdown, please remove them: {'style': 'color:gray'}
2023-09-07 14:10:56 | ERROR | stderr |   warnings.warn(
2023-09-07 14:10:56 | INFO | stdout | Running on local URL:  http://127.0.0.1:7860
2023-09-07 14:14:05 | INFO | gradio_web_server | load_demo.. params: {}
2023-09-07 14:14:18 | INFO | gradio_web_server | add_text. ip:. len: 26
2023-09-07 14:14:19 | ERROR | stderr | Traceback (most recent call last):
2023-09-07 14:14:19 | ERROR | stderr |   File "/home/nkd/miniconda3/envs/video_chatgpt/lib/python3.10/site-packages/gradio/routes.py", line 394, in run_predict
2023-09-07 14:14:19 | ERROR | stderr |     output = await app.get_blocks().process_api(
2023-09-07 14:14:19 | ERROR | stderr |   File "/home/nkd/miniconda3/envs/video_chatgpt/lib/python3.10/site-packages/gradio/blocks.py", line 1075, in process_api
2023-09-07 14:14:19 | ERROR | stderr |     result = await self.call_function(
2023-09-07 14:14:19 | ERROR | stderr |   File "/home/nkd/miniconda3/envs/video_chatgpt/lib/python3.10/site-packages/gradio/blocks.py", line 898, in call_function
2023-09-07 14:14:19 | ERROR | stderr |     prediction = await anyio.to_thread.run_sync(
2023-09-07 14:14:19 | ERROR | stderr |   File "/home/nkd/miniconda3/envs/video_chatgpt/lib/python3.10/site-packages/anyio/to_thread.py", line 33, in run_sync
2023-09-07 14:14:19 | ERROR | stderr |     return await get_asynclib().run_sync_in_worker_thread(
2023-09-07 14:14:19 | ERROR | stderr |   File "/home/nkd/miniconda3/envs/video_chatgpt/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 877, in run_sync_in_worker_thread
2023-09-07 14:14:19 | ERROR | stderr |     return await future
2023-09-07 14:14:19 | ERROR | stderr |   File "/home/nkd/miniconda3/envs/video_chatgpt/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 807, in run
2023-09-07 14:14:19 | ERROR | stderr |     result = context.run(func, *args)
2023-09-07 14:14:19 | ERROR | stderr |   File "/home/nkd/miniconda3/envs/video_chatgpt/lib/python3.10/site-packages/gradio/utils.py", line 549, in async_iteration
2023-09-07 14:14:19 | ERROR | stderr |     return next(iterator)
2023-09-07 14:14:19 | ERROR | stderr |   File "/home/nkd/Documents/jjy/comment_generator/Video-ChatGPT/video_chatgpt/demo/chat.py", line 109, in answer
2023-09-07 14:14:19 | ERROR | stderr |     image_forward_outs = self.vision_tower(image_tensor, output_hidden_states=True)
2023-09-07 14:14:19 | ERROR | stderr |   File "/home/nkd/miniconda3/envs/video_chatgpt/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
2023-09-07 14:14:19 | ERROR | stderr |     return forward_call(*args, **kwargs)
2023-09-07 14:14:19 | ERROR | stderr |   File "/home/nkd/miniconda3/envs/video_chatgpt/lib/python3.10/site-packages/transformers/models/clip/modeling_clip.py", line 958, in forward
2023-09-07 14:14:19 | ERROR | stderr |     return self.vision_model(
2023-09-07 14:14:19 | ERROR | stderr |   File "/home/nkd/miniconda3/envs/video_chatgpt/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
2023-09-07 14:14:19 | ERROR | stderr |     return forward_call(*args, **kwargs)
2023-09-07 14:14:19 | ERROR | stderr |   File "/home/nkd/miniconda3/envs/video_chatgpt/lib/python3.10/site-packages/transformers/models/clip/modeling_clip.py", line 883, in forward
2023-09-07 14:14:19 | ERROR | stderr |     hidden_states = self.embeddings(pixel_values)
2023-09-07 14:14:19 | ERROR | stderr |   File "/home/nkd/miniconda3/envs/video_chatgpt/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
2023-09-07 14:14:19 | ERROR | stderr |     return forward_call(*args, **kwargs)
2023-09-07 14:14:19 | ERROR | stderr |   File "/home/nkd/miniconda3/envs/video_chatgpt/lib/python3.10/site-packages/transformers/models/clip/modeling_clip.py", line 196, in forward
2023-09-07 14:14:19 | ERROR | stderr |     patch_embeds = self.patch_embedding(pixel_values.to(dtype=target_dtype))  # shape = [*, width, grid, grid]
2023-09-07 14:14:19 | ERROR | stderr |   File "/home/nkd/miniconda3/envs/video_chatgpt/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
2023-09-07 14:14:19 | ERROR | stderr |     return forward_call(*args, **kwargs)
2023-09-07 14:14:19 | ERROR | stderr |   File "/home/nkd/miniconda3/envs/video_chatgpt/lib/python3.10/site-packages/torch/nn/modules/conv.py", line 463, in forward
2023-09-07 14:14:19 | ERROR | stderr |     return self._conv_forward(input, self.weight, self.bias)
2023-09-07 14:14:19 | ERROR | stderr |   File "/home/nkd/miniconda3/envs/video_chatgpt/lib/python3.10/site-packages/torch/nn/modules/conv.py", line 459, in _conv_forward
2023-09-07 14:14:19 | ERROR | stderr |     return F.conv2d(input, weight, bias, self.stride,
2023-09-07 14:14:19 | ERROR | stderr | RuntimeError: GET was unable to find an engine to execute this computation
2023-09-07 14:15:59 | INFO | stdout | Running on public URL: https://639177a685ea0e6be8.gradio.live
2023-09-07 14:15:59 | INFO | stdout | 
2023-09-07 14:15:59 | INFO | stdout | This share link expires in 72 hours. For free permanent hosting and GPU upgrades (NEW!), check out Spaces: https://huggingface.co/spaces
mmaaz60 commented 9 months ago

Hi @JuanJia,

Apologies for the delayed reply. It looks like an issue with transformers version. Please try matching the requirements as in requirements.txt and let me know if the issue persists. Thanks

ashmalvayani commented 8 months ago

@JuanJia I've had a similar issue, please use the localhost link instead of the gradio link, your issue will be resolved.