[Question]: How to use MistralAI deployed serverless on Azure AI?

[ziodave]

Question Validation

• [X] I have searched both the documentation and discord for an answer.

Question

I deployed Mistral via Azure AI, how do I use it in LlamaIndex. I tried using the MistralAI LLM in LlamaIndex but I don't see how to configure the endpoint to a custom endpoint.

[dosubot[bot]]

Hey @ziodave, nice to see you around here again! Hope you're doing well.

To configure the MistralAI LLM in LlamaIndex to use a custom endpoint after deploying Mistral via Azure AI, you need to modify the initialization of the MistralClient and MistralAsyncClient within the __init__ method of the MistralAI class to accept a custom endpoint parameter and pass it to these clients. Here's how you can do it:

1. Add a new parameter endpoint: Optional[str] = None to the __init__ method signature.
2. Use this endpoint parameter when initializing MistralClient and MistralAsyncClient, defaulting to DEFAULT_MISTRALAI_ENDPOINT if None is provided.

Here's the modified part of the __init__ method:

def __init__(
        self,
        model: str = DEFAULT_MISTRALAI_MODEL,
        temperature: float = DEFAULT_TEMPERATURE,
        max_tokens: int = DEFAULT_MISTRALAI_MAX_TOKENS,
        timeout: int = 120,
        max_retries: int = 5,
        safe_mode: bool = False,
        random_seed: Optional[int] = None,
        api_key: Optional[str] = None,
        additional_kwargs: Optional[Dict[str, Any]] = None,
        callback_manager: Optional[CallbackManager] = None,
        system_prompt: Optional[str] = None,
        messages_to_prompt: Optional[Callable[[Sequence[ChatMessage]], str]] = None,
        completion_to_prompt: Optional[Callable[[str], str]] = None,
        pydantic_program_mode: PydanticProgramMode = PydanticProgramMode.DEFAULT,
        output_parser: Optional[BaseOutputParser] = None,
        endpoint: Optional[str] = None,  # New parameter for custom endpoint
    ) -> None:
        # Existing code to import MistralClient and MistralAsyncClient and check for api_key

        # Use the custom endpoint if provided, otherwise default to DEFAULT_MISTRALAI_ENDPOINT
        endpoint = endpoint or DEFAULT_MISTRALAI_ENDPOINT

        self._client = MistralClient(
            api_key=api_key,
            endpoint=endpoint,  # Use the custom or default endpoint
            timeout=timeout,
            max_retries=max_retries,
        )
        self._aclient = MistralAsyncClient(
            api_key=api_key,
            endpoint=endpoint,  # Use the custom or default endpoint
            timeout=timeout,
            max_retries=max_retries,
        )
        # Rest of the __init__ method...

With this modification, you can now pass a custom endpoint URL to the MistralAI class upon instantiation, allowing it to communicate with a Mistral deployment on Azure AI or any other platform.

You can find the current structure of the MistralAI class and the DEFAULT_MISTRALAI_ENDPOINT in the LlamaIndex codebase at the following links:

• llama-index-legacy/llama_index/legacy/llms/mistral.py
• llama-index-integrations/llms/llama-index-llms-mistralai/llama_index/llms/mistralai/base.py

Please note that this modification should be done in a fork of the LlamaIndex repository and not directly in the main repository. If you have any further questions, feel free to ask.

For more information, you can refer to the Dosu documentation.

Sources

• llama-index-legacy/llama_index/legacy/llms/mistral.py
• llama-index-integrations/llms/llama-index-llms-mistralai/llama_index/llms/mistralai/base.py

#### About Dosu

This response is meant to be useful and save you time. It is **not meant to be a precise solution**, but rather a starting point for your own research. Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant. If you want to continue the conversation, start your reply with @dosu-bot.