dosubot[bot]
commented
1 month ago To run a multimodal LLM locally and replace the OpenAI API usage in your video processing project, you can use the HuggingFaceLLM class from the llama_index.legacy.llms.custom module with a model from Huggingface. Here's an example of how to set it up:
from llama_index.legacy.llms.custom import HuggingFaceLLM
# Initialize the HuggingFaceLLM with your chosen model
llm = HuggingFaceLLM(
model_name="your-model-name-here", # Specify your Huggingface model name
tokenizer_name="your-tokenizer-name-here", # Specify your tokenizer name, if different
context_window=2048, # Adjust based on your model's capabilities
max_new_tokens=256, # Adjust based on your needs
)
# Generate a response
response = llm.complete("Your prompt here")
print(response.text)Make sure to replace "your-model-name-here" and "your-tokenizer-name-here" with the actual names of the model and tokenizer you wish to use. The context_window and max_new_tokens parameters should be adjusted according to the capabilities and requirements of your chosen model.
Regarding the availability of multimodal models on Huggingface for video processing, it's important to note that while there are models that support multimodal interactions, their direct application to video processing might be limited. The document mentions models like GPT4V, CLIP, and others, but also indicates that video processing capabilities are not directly supported at the moment. This suggests that while these models can be used for multimodal tasks, additional steps or custom integrations might be necessary for processing video content specifically.
#### Details
**Sources** - [llama-index-integrations/llms/llama-index-llms-huggingface/llama_index/llms/huggingface/base.py](https://github.com/run-llama/llama_index/blob/main/llama-index-integrations/llms/llama-index-llms-huggingface/llama_index/llms/huggingface/base.py) - [llama-index-legacy/llama_index/legacy/llms/huggingface.py](https://github.com/run-llama/llama_index/blob/main/llama-index-legacy/llama_index/legacy/llms/huggingface.py) - [docs/docs/module_guides/models/multi_modal.md](https://github.com/run-llama/llama_index/blob/main/docs/docs/module_guides/models/multi_modal.md) - [llama-index-integrations/llms/llama-index-llms-huggingface/pyproject.toml](https://github.com/run-llama/llama_index/blob/main/llama-index-integrations/llms/llama-index-llms-huggingface/pyproject.toml) - [llama-index-integrations/embeddings/llama-index-embeddings-huggingface/pyproject.toml](https://github.com/run-llama/llama_index/blob/main/llama-index-integrations/embeddings/llama-index-embeddings-huggingface/pyproject.toml) - [llama-index-integrations/vector_stores/llama-index-vector-stores-lancedb/llama_index/vector_stores/lancedb/base.py](https://github.com/run-llama/llama_index/blob/main/llama-index-integrations/vector_stores/llama-index-vector-stores-lancedb/llama_index/vector_stores/lancedb/base.py) - [docs/docs/examples/llm/huggingface.ipynb](https://github.com/run-llama/llama_index/blob/main/docs/docs/examples/llm/huggingface.ipynb) - [llama-index-integrations/embeddings/llama-index-embeddings-huggingface/llama_index/embeddings/huggingface/base.py](https://github.com/run-llama/llama_index/blob/main/llama-index-integrations/embeddings/llama-index-embeddings-huggingface/llama_index/embeddings/huggingface/base.py) - [llama-index-legacy/llama_index/legacy/embeddings/huggingface.py](https://github.com/run-llama/llama_index/blob/main/llama-index-legacy/llama_index/legacy/embeddings/huggingface.py) - [llama-index-legacy/llama_index/legacy/query_engine/multi_modal.py](https://github.com/run-llama/llama_index/blob/main/llama-index-legacy/llama_index/legacy/query_engine/multi_modal.py) - [llama-index-legacy/llama_index/legacy/multi_modal_llms/openai.py](https://github.com/run-llama/llama_index/blob/main/llama-index-legacy/llama_index/legacy/multi_modal_llms/openai.py)**About Dosu** This response is meant to be useful and save you time. It is **not meant to be a precise solution**, but rather a starting point for your own research. Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.
To continue the conversation, mention @dosu.
tsantra
LJ-Hao
Question Validation
Question
I am Referring to this example: https://www.llamaindex.ai/blog/multimodal-rag-for-advanced-video-processing-with-llamaindex-lancedb-33be4804822e to run a multimodal LLM for a video processing use case. I want to know how to run the Multimodal LLM for generation locally? I have highlighted the code block which is now using OpenAI api which needs to be replaced so that we can run a multimodal LLM locally.
I am using the embedding model from Huggingface.
Could anyone please help.
from llama_index.core.response.notebook_utils import display_source_node from llama_index.core.schema import ImageNode from pathlib import Path from moviepy.editor import * import speech_recognition as sr
from llama_index.embeddings.huggingface import HuggingFaceEmbedding from llama_index.core import Settings from PIL import Image import matplotlib.pyplot as plt
from llama_index.core.indices import MultiModalVectorStoreIndex from llama_index.core import StorageContext from llama_index.vector_stores.lancedb import LanceDBVectorStore from llama_index.core import SimpleDirectoryReader import lancedb import numpy as np
video_url = "https://www.youtube.com/watch?v=d_qvLDhkg00"
output_video_path = "./video_data/" output_folder = "./mixed_data/" output_audio_path = "./mixed_data/output_audio.wav" filepath = output_video_path + "input_vid.mp4" Path(output_folder).mkdir(parents=True, exist_ok=True)
def plot_images(image_paths): images_shown = 0 plt.figure(figsize=(16, 9)) for img_path in image_paths: if os.path.isfile(img_path): image = Image.open(img_path)
def download_video(url, output_path): """ Download a video from a given url and save it to the output path.
metadata_vid = download_video(video_url, output_video_path)
def video_to_images(video_path, output_folder): """ Convert a video to a sequence of images and save them to the output folder.
def video_to_audio(video_path, output_audio_path): """ Convert a video to audio and save it to the output path.
def audio_to_text(audio_path): """ Convert an audio file to text.
metadata_vid = download_video(video_url, output_video_path)
video_to_images(filepath, output_folder) video_to_audio(filepath, output_audio_path) text_data = audio_to_text(output_audio_path)
with open(output_folder + "output_text.txt", "w") as file: file.write(text_data) print("Text data saved to file") file.close()
os.remove(output_audio_path) print("Audio file removed")
building the multimodal index and Vector Store
uri= "tmp/sample-lancedb" text_store = LanceDBVectorStore(uri=uri, table_name="text_collection") image_store = LanceDBVectorStore(uri=uri, table_name="image_collection") storage_context = StorageContext.from_defaults( vector_store=text_store, image_store=image_store )
Create the MultiModal index
documents = SimpleDirectoryReader(output_folder).load_data()
Settings.embed_model = HuggingFaceEmbedding( model_name="sentence-transformers/clip-ViT-B-32-multilingual-v1" )
the MultiModalVectorStoreIndex used the embedding model mentioned above
index = MultiModalVectorStoreIndex.from_documents( documents, storage_context=storage_context,
)
retriever_engine = index.as_retriever( similarity_top_k=5, image_similarity_top_k=5 )
import json
metadata_str = json.dumps(metadata_vid)
qa_tmpl_str = ( "Given the provided information, including relevant images and retrieved context from the video, \ accurately and precisely answer the query without any additional prior knowledge.\n" "Please ensure honesty and responsibility, refraining from any racist or sexist remarks.\n" "---------------------\n" "Context: {context_str}\n" "Metadata for video: {metadata_str} \n" "---------------------\n" "Query: {query_str}\n" "Answer: " )
def retrieve(retriever_engine, query_str): retrieval_results = retriever_engine.retrieve(query_str)
query_str = "Using examples from video, explain all things covered in the video regarding the police activity"
img, txt = retrieve(retriever_engine=retriever_engine, query_str=query_str)
image_documents = SimpleDirectoryReader( input_dir=output_folder, input_files=img ).load_data() context_str = "".join(txt) plot_images(img)
######################################## **#how do I replace this module with a local multimodal llm? What class should I import?
What are my options if I want to import a multimodal model from Huggingface? How do I use it instead of the code below?
'''** from llama_index.multi_modal_llms.openai import OpenAIMultiModal openai_mm_llm = OpenAIMultiModal( model="gpt-4-turbo", api_key= "sk-proj-4jX30IwiomEZDezL9pJJT3BlbkFJefAyiBjUhXUlYYW1kFdL", max_new_tokens=1500 ) response_1 = openai_mm_llm.complete( prompt=qa_tmpl_str.format( context_str=context_str, query_str=query_str, metadata_str=metadata_str ), image_documents=image_documents, ) pprint(response_1.text) '''
Referring to this example: https://www.llamaindex.ai/blog/multimodal-rag-for-advanced-video-processing-with-llamaindex-lancedb-33be4804822e
from pathlib import Path from moviepy.editor import * import speech_recognition as sr
from llama_index.embeddings.huggingface import HuggingFaceEmbedding from llama_index.core import Settings from PIL import Image import matplotlib.pyplot as plt
from llama_index.core.indices import MultiModalVectorStoreIndex from llama_index.core import StorageContext from llama_index.vector_stores.lancedb import LanceDBVectorStore from llama_index.core import SimpleDirectoryReader import lancedb import numpy as np
video_url = "https://www.youtube.com/watch?v=d_qvLDhkg00"
output_video_path = "./video_data/" output_folder = "./mixed_data/" output_audio_path = "./mixed_data/output_audio.wav" filepath = output_video_path + "input_vid.mp4" Path(output_folder).mkdir(parents=True, exist_ok=True)
def plot_images(image_paths): images_shown = 0 plt.figure(figsize=(16, 9)) for img_path in image_paths: if os.path.isfile(img_path): image = Image.open(img_path)
def download_video(url, output_path): """ Download a video from a given url and save it to the output path.
metadata_vid = download_video(video_url, output_video_path)
def video_to_images(video_path, output_folder): """ Convert a video to a sequence of images and save them to the output folder.
def video_to_audio(video_path, output_audio_path): """ Convert a video to audio and save it to the output path.
def audio_to_text(audio_path): """ Convert an audio file to text.
metadata_vid = download_video(video_url, output_video_path)
video_to_images(filepath, output_folder) video_to_audio(filepath, output_audio_path) text_data = audio_to_text(output_audio_path)
with open(output_folder + "output_text.txt", "w") as file: file.write(text_data) print("Text data saved to file") file.close()
os.remove(output_audio_path) print("Audio file removed")
building the multimodal index and Vector Store
uri= "tmp/sample-lancedb" text_store = LanceDBVectorStore(uri=uri, table_name="text_collection") image_store = LanceDBVectorStore(uri=uri, table_name="image_collection") storage_context = StorageContext.from_defaults( vector_store=text_store, image_store=image_store )
Create the MultiModal index
documents = SimpleDirectoryReader(output_folder).load_data()
Settings.embed_model = HuggingFaceEmbedding( model_name="sentence-transformers/clip-ViT-B-32-multilingual-v1" )
the MultiModalVectorStoreIndex used the embedding model mentioned above
index = MultiModalVectorStoreIndex.from_documents( documents, storage_context=storage_context,
)
retriever_engine = index.as_retriever( similarity_top_k=5, image_similarity_top_k=5 )
import json
metadata_str = json.dumps(metadata_vid)
qa_tmpl_str = ( "Given the provided information, including relevant images and retrieved context from the video, \ accurately and precisely answer the query without any additional prior knowledge.\n" "Please ensure honesty and responsibility, refraining from any racist or sexist remarks.\n" "---------------------\n" "Context: {context_str}\n" "Metadata for video: {metadata_str} \n" "---------------------\n" "Query: {query_str}\n" "Answer: " )
from llama_index.core.response.notebook_utils import display_source_node from llama_index.core.schema import ImageNode
def retrieve(retriever_engine, query_str): retrieval_results = retriever_engine.retrieve(query_str)
query_str = "Using examples from video, explain all things covered in the video regarding the police activity"
img, txt = retrieve(retriever_engine=retriever_engine, query_str=query_str)
image_documents = SimpleDirectoryReader( input_dir=output_folder, input_files=img ).load_data() context_str = "".join(txt) plot_images(img)
########################################
how do I replace this module with a local multimodal llm? What class should I import?
What are my options if I want to import a multimodal llm from Huggingface? How do I use it instead of the code below?
''' from llama_index.multi_modal_llms.openai import OpenAIMultiModal openai_mm_llm = OpenAIMultiModal( model="gpt-4-turbo", api_key= "sk-proj-4jX30IwiomEZDezL9pJJT3BlbkFJefAyiBjUhXUlYYW1kFdL", max_new_tokens=1500 ) response_1 = openai_mm_llm.complete( prompt=qa_tmpl_str.format( context_str=context_str, query_str=query_str, metadata_str=metadata_str ), image_documents=image_documents, ) pprint(response_1.text) '''