Completion model provider should be traitlet-configurable

[krassowski]

@krassowski Will it also be better if we can set something like default_completion_model and default_chat_model when running jupyter lab --config using config.json? Previously, we can only do something like:

{
    "AiExtension": {
        "default_language_model": "Self-Hosted:deepseek-coder-6.7b-instruct",
        "allowed_providers": "Self-Hosted",
        "model_parameters": {
            "Self-Hosted:deepseek-coder-6.7b-base": {},
            "Self-Hosted:deepseek-coder-6.7b-instruct": {}
        },
        "default_api_keys":{}
    }
}

It may be better to modify the extension.py also. Thank you.

Originally posted by @yuhon0528 in https://github.com/jupyterlab/jupyter-ai/issues/711#issuecomment-2104050127

[yuhon0528]

Also, since we separated into two models (chat and completion), the API keys will most likely be different. However, for example if I use OpenAI Compliable APIs for both chat and completion models, there will only be one OPEN_API_KEY field for authentication.

How can I authenticate both models with different API keys with only one field? Thanks.

[krassowski]

For now, we would probably only want to only add default_completion_model (rather than rename default_language_model to default_chat_model) to avoid breaking compatibility.

[krassowski]

The API keys are a key-value mapping so maybe that is not a problem? If you use cohere for chat you would set COHERE_API_KEY, if you use openai for completions you would set OPENAI_API_KEY, so the default_api_keys would be:

"default_api_keys": {
  "COHERE_API_KEY": "MY_KEY_1",
  "OPENAI_API_KEY": "MY_KEY_1"
}

and respective providers would take the key they need. Does it makes sense?

[yuhon0528]

But in fact is that I am using two models that are both with OpenAI compatible APIs, that is:

{
    "AiExtension": {
        "default_language_model": "Self-Hosted:deepseek-coder-6.7b-instruct",
        "allowed_providers": "Self-Hosted",
        "model_parameters": {
            "Self-Hosted:deepseek-coder-6.7b-base": {},
            "Self-Hosted:deepseek-coder-6.7b-instruct": {}
        },
        "default_api_keys":{
            "OPENAI_API_KEY": "deepseek-coder-6.7b-base-key",
            "OPENAI_API_KEY": "deepseek-coder-6.7b-instruct-key"
        }
    }
}

When we use those self-hosting models for chat and completions, most likely will use something like vllm to provide OpenAI compatible API Servers.

Hence, that is the fact that both completion and chat model will be OpenAI API like.

[dcieslak19973]

Hoping to help get some momentum on this issue. A link to my comments on a related/duplicate issue:

https://github.com/jupyterlab/jupyter-ai/issues/953#issuecomment-2332151326

TL;DR - it does not currently seem possible to configure the Completions solely from the jupyter_jupyter_ai_config.json file