Documentation is incorrect for Azure Autocomplete

[jacekplocharczyk]

Before submitting your bug report

• [X] I believe this is a bug. I'll try to join the Continue Discord for questions
• [X] I'm not able to find an open issue that reports the same bug
• [X] I've seen the troubleshooting guide on the Continue Docs

Relevant environment info

No response

Description

At the https://docs.continue.dev/customize/model-providers/azure#autocomplete-model there is a recommended model "Codestral" which is unavailable at Azure (only mentioned that is coming soon here).

Please fix the documentation to something working.

PS For GPT-4 and GPT-4o I couldn't get tab autocomplete - part of the error message:

The completion operation does not work with the specified model, gpt-4o.

So I would really grateful for working example on Azure.

Cheers!

To reproduce

No response

Log output

No response

[dbwodlf3]

https://github.com/continuedev/continue/issues/1351

The update has been released, but the issue hasn't been fixed yet.

[dbwodlf3]

    "apiKey": "****",
    "engine": "****",
    "apiBase": "https://**.openai.azure.com/",
    "apiVersion": "2024-08-01-preview",
    "apiType": "azure",
    "model": "gpt-4o-mini",
    "title": "****",
    "systemMessage": "You are an expert software developer. You give helpful and concise responses.",
    "maxStopWords": 4,
    "provider": "openai"

It works well.

https://github.com/continuedev/continue/blob/main/core/llm/llms/OpenAI.ts#L24

[jacekplocharczyk]

Yes, it does work. Thank you @dbwodlf3

[Patrick-Erichsen]

@jacekplocharczyk thanks for raising this issue!

Based on what I'm seeing in https://ai.azure.com/explore/models I'm not sure if there is a good autocomplete model that we would recommend. Might need to update the docs to reflect this.

cc @sestinj for your thoughts. I'd lean away from recommending gpt-4o-mini despite it being an available option in Azure.

[sestinj]

I added a note: https://github.com/continuedev/continue/commit/73e75f87e91d5661d86c72da443770c515793590

I want to leave the Codestral reference there so that people know that at least there is an option coming soon, and we will definitely not be recommending gpt-4o-mini or other OpenAI models for autocomplete since they aren't trained for it

[dbwodlf3]

@sestinj What exactly does it mean that GPT-4o-mini isn't trained for autocomplete? Based on the benchmarks alone, GPT-4o-mini seems to perform better. Is this more about the accuracy of output in response to a given prompt?

References

• https://bigcode-bench.github.io/

[sestinj]

It means that the gpt models weren't trained with a FIM (fill in the middle) template. You can still get gpt-4o to generate something like this by creating a super elaborate prompt that asks it to only output the code that should go in between prefix and suffix, but the cost is quite prohibitive given the comparative size to the models trained for autocomplete (Codestral is one of the largest and is still only 22B)