search

mbzuai-oryx / Video-ChatGPT

[ACL 2024 🔥] Video-ChatGPT is a video conversation model capable of generating meaningful conversation about videos. It combines the capabilities of LLMs with a pretrained visual encoder adapted for spatiotemporal video representation. We also introduce a rigorous 'Quantitative Evaluation Benchmarking' for video-based conversational models.
https://mbzuai-oryx.github.io/Video-ChatGPT
Creative Commons Attribution 4.0 International
1.05k stars 92 forks source link

Missing config.json for Video-ChatGPT-7B when deployed on AWS Sagemaker #41

Closed alinaSarwar closed 9 months ago

alinaSarwar commented 10 months ago

I have deployed the Video-ChatGPT-7B model on AWS Sagemaker using the script given on the HuggingFace website. image

However, there are two issues:

  1. I have the predictor now but what exactly should I send in the payload? How do I include the video alongside my prompt in the request to the endpoint created by Sagemaker? Can you please share some example payload?
  2. I tried sending a payload without a video just to test whether the endpoint works or not and it gave me the following error: ModelError: An error occurred (ModelError) when calling the InvokeEndpoint operation: Received client error (400) from primary with message "{ "code": 400, "type": "InternalServerException", "message": "/.sagemaker/mms/models/MBZUAI__Video-ChatGPT-7B does not appear to have a file named config.json. Checkout \u0027https://huggingface.co//.sagemaker/mms/models/MBZUAI__Video-ChatGPT-7B/None\u0027 for available files." } Where can I get this config.json file?
mmaaz60 commented 10 months ago

Hi @alinaSarwar

Thank you for your interest in our work. Please note that currently we do not have any instructions on deploying on AWS SageMaker. And the instructions provided by HuggingFace are automatically generated and may contain multiple errors.

Further some important points to note down,

1. The provided Video-ChatGPT model at HuggingFace contains the checkpoints of only the Linear Layer that we fine-tuned on our video-instruction dataset. And it should be combined with LLaVA Weights to get the complete model. 2. The detailed instructions to run demo on a machine with an NVIDIA GPU are available at run demo offline. As we do not have much experience working with deployment on AWS Sagemaker, we will not be able to help you a lot in this scenario. Instead we recommend creating a GPU instance and try following instructions to run the offline demo.

Please let us know if you have any questions. Thank You.