[Colab Demo] Possible misconfiguration // wrong sed params?

[NirantK]

Hey, I'm trying setting up a Colab demo with MCDSE but getting something obvious wrong: https://colab.research.google.com/drive/1aEgITiGDgKb3RSaKHcNjdHa67HVkibZc?usp=sharing

Would appreciate a second pair of eyes 👀

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-11-ded3bcb9bf8d> in <cell line: 3>()
      1 # Encode queries
      2 query_prompt, image = get_query_prompt("Quali erano le passività totali al 31 dicembre 2017?")
----> 3 outputs = llm.encode({"prompt": query_prompt, "multi_modal_data": {"image": [image]}})
      4 outputs[0].outputs.embedding #1536 dimensional embedding

1 frames
/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/llm.py in encode(self, prompts, pooling_params, prompt_token_ids, use_tqdm, lora_request, prompt_adapter_request)
    693         """
    694         if not self.llm_engine.model_config.embedding_mode:
--> 695             raise ValueError(
    696                 "LLM.encode() is only supported for embedding models (XModel)."
    697             )

ValueError: LLM.encode() is only supported for embedding models (XModel).

[Marplex]

Hi, you need to edit the downloaded model config.json file and replace Qwen2VLForConditionalGeneration with Qwen2VLForEmbeddingGeneration.

The local_dir should be 'models/mcdse-2b-v1' and consequentially the sed command will be the following:

sed -i -e 's/Qwen2VLForConditionalGeneration/Qwen2VLForEmbeddingGeneration/g' /content/models/mcdse-2b-v1/config.json