Fail to execute Video-ChatGPT offline demo (when click "Upload Video")

[jeonggwanlee]

Hi, Thank you for sharing Video-ChatGPT open-source.

I tried to run "offline demo" and get gradio public URL.

When I clicked "Upload Video" in gradio web live, there's an error related to gradio library.

I also tried debug mode in vscode, the reason why this error happened is below.

because, the length of "dependency['outputs']" ans "prediction" is different.

I created conda env as following in the readme.md. conda create python 3.10 pip install -r requirements pip install safetensors

Is there anyone having this situation and could you help me? Thank you very much.

[mmaaz60]

Hi @jeonggwanlee,

Thank you for your interest in our work. Can you please manually check the PyTorch, Pydantic and Gradio versions of your environment and see if they match the following,

• torch==2.0.1
• pydantic==1.10.7
• gradio==3.23.0

Also please try installing transformers as

• pip install transformers@git+https://github.com/huggingface/transformers.git@cae78c46

Please let me know if it solves the issues. Thank you.

[jeonggwanlee]

Thank you for the quick response, @mmaaz60!

This is what I did.

• conda create --name=video_chatgpt python=3.10

• pip install -r requirements

Package Version

---

accelerate 0.20.3 aiofiles 23.2.1
aiohttp 3.9.1
aiosignal 1.3.1
altair 5.2.0
anyio 4.2.0
async-timeout 4.0.3
attrs 23.2.0
certifi 2023.11.17 charset-normalizer 3.3.2
click 8.1.7 cmake 3.28.1 contourpy 1.2.0 cycler 0.12.1 decord 0.6.0 einops 0.6.1 exceptiongroup 1.2.0 fastapi 0.109.0 ffmpy 0.3.1 filelock 3.13.1 fonttools 4.47.2 frozenlist 1.4.1 fsspec 2023.12.2 gradio 3.23.0 h11 0.14.0
httpcore 1.0.2 httpx 0.26.0 huggingface-hub 0.20.2
idna 3.6
Jinja2 3.1.3 jsonschema 4.21.1 jsonschema-specifications 2023.12.1 kiwisolver 1.4.5 linkify-it-py 2.0.2 lit 17.0.6 markdown-it-py 2.2.0 markdown2 2.4.8 MarkupSafe 2.1.4 matplotlib 3.8.2 mdit-py-plugins 0.3.3 mdurl 0.1.2 mpmath 1.3.0 multidict 6.0.4
networkx 3.2.1 numpy 1.24.3 nvidia-cublas-cu11 11.10.3.66 nvidia-cuda-cupti-cu11 11.7.101 nvidia-cuda-nvrtc-cu11 11.7.99 nvidia-cuda-runtime-cu11 11.7.99 nvidia-cudnn-cu11 8.5.0.96 nvidia-cufft-cu11 10.9.0.58 nvidia-curand-cu11 10.2.10.91 nvidia-cusolver-cu11 11.4.0.1
nvidia-cusparse-cu11 11.7.4.91 nvidia-nccl-cu11 2.14.3 nvidia-nvtx-cu11 11.7.91 orjson 3.9.12 packaging 23.2 pandas 2.2.0 Pillow 9.5.0 pip 23.3.2 protobuf 4.23.2 psutil 5.9.8 pydantic 1.10.7 pydub 0.25.1 pyparsing 3.1.1 python-dateutil 2.8.2 python-multipart 0.0.6 pytz 2023.3.post1 PyYAML 6.0.1 referencing 0.32.1 regex 2023.12.25 requests 2.30.0 rpds-py 0.17.1 safetensors 0.4.1 semantic-version 2.10.0 sentencepiece 0.1.99 setuptools 69.0.3 six 1.16.0 sniffio 1.3.0 starlette 0.35.1 sympy 1.12 tokenizers 0.13.3 toolz 0.12.0 torch 2.0.1 tqdm 4.65.0 transformers 4.28.0.dev0 triton 2.0.0 typing_extensions 4.9.0 tzdata 2023.4 uc-micro-py 1.0.2 urllib3 2.1.0 uvicorn 0.26.0 websockets 12.0 wheel 0.42.0 yarl 1.9.4

And then,

• pip uninstall transformers (delete original one)

• pip install transformers@git+https://github.com/huggingface/transformers.git@cae78c46

When I tried with this command, python video_chatgpt/demo/video_demo.py --model-name LLaVA-7B-Lightening-v1-1 --projection_path video_chatgpt-7B.bin

I got this error. I expected that this means only supported by hugging face hub repo,

python video_chatgpt/demo/video_demo.py --model-name mmaaz60/LLaVA-7B-Lightening-v1-1 --projection_path video_chatgpt-7B.bin

Then, I got the same error I already reported first.

I will try from the scatch one more time.

[jeonggwanlee]

I tried what I mentioned (above) one more time, But still same error.

[jeonggwanlee]

Or, alternative way would be, Not using gradio but I'd like to manually try in terminal.

Is there more developer friendly inference mode? If not, just naive process (use this function and that function) also might be very helpful!!

Thanks.

[jeonggwanlee]

Or, If you have a docker image, could you let me know? (like docker hub)

[mmaaz60]

Hi @jeonggwanlee,

You can use the script here to run inference on the terminal.

Related to the issue with gradio demo, please check if you have correct pydantic version pydantic==1.10.7 after installing the transformers library using pip install transformers@git+https://github.com/huggingface/transformers.git@cae78c46.

Further, I will try to create and upload the docker image soon. Thank you.

[jeonggwanlee]

I can use your inference script and It works well. :)

Thank you for your kind response and sharing your work, again! 👍