Add Nebius AI Studio support: inference platform with dual performance options and competitive pricing

[demianarc]

Validations

• [X] I believe this is a way to improve. I'll try to join the Continue Discord for questions
• [X] I'm not able to find an open issue that requests the same enhancement

Problem

Developers using Continue face several challenges that impact their coding flow:

Cost barriers when scaling AI usage across teams

High token costs limiting experimentation and usage Difficulty predicting and managing AI development expenses Budget constraints affecting adoption across organizations

Performance vs. cost trade-offs

Need to choose between fast responses and budget optimization Inconsistent performance affecting development flow Limited options for different coding scenarios (quick prototyping vs. production code)

Model accessibility and flexibility

Limited options for specialized coding models Difficulty accessing cutting-edge open-source models Need for different models for various development tasks

Solution

Integrating Nebius AI Studio with Continue would enhance the development experience by providing:

For individual developers:

Uninterrupted coding flow with dual model options: "Fast" flavor for real-time pair programming "Base" flavor for cost-effective batch operations

• Seamless access to specialized coding models like DeepSeek Coder, Meta-Llama-3.1-Nemotron-70B, Qwen/Qwen2.5-72B-Instruct and much more
• $100 in free credits to experiment and find optimal settings

For Teams and organizations:

• Affordable and flexible pricing model with separate input/output token rates
• Centralized access to multiple state-of-the-art models
• Batch processing capabilities for efficient large-scale operations

Technical Integration benefits:

• Drop-in OpenAI API compatibility
• Rich selection of coding-optimized models
• Structured output support for precise code generation
• Simple API key authentication

Implementation approach:

1. Add Nebius AI Studio as a model provider alongside existing options
2. Implement model flavor selection in Continue's configuration
3. Maybe integrate batch processing for improved performance

This integration aligns perfectly with Continue's mission to amplify developers and enhance development through AI, while adding valuable flexibility in model choice and cost optimization. We're ready to contribute to the implementation and collaborate with the Continue team to ensure a seamless integration that enhances the developer experience.

[sestinj]

@demianarc Thanks for the great write-up here! I was already excited about the PR the other day, but sounds like the next outcome would be to add the flavor option in config.json. I think probably this would fit well in the ModelDescription type in index.d.ts, which can then be read from config.json and passed into the LLM class here

[demianarc]

Hey,

thanks for the feedback, forwarded to the team :)

could also be interesting to do some co marketing initiatives, is that something you could be interested in?

Best

Dylan

On Wed, Nov 6, 2024 at 12:47 AM Nate Sesti @.***> wrote:

@demianarc https://github.com/demianarc Thanks for the great write-up here! I was already excited about the PR the other day, but sounds like the next outcome would be to add the flavor option in config.json. I think probably this would fit well in the ModelDescription type in index.d.ts, which can then be read from config.json and passed into the LLM class here https://github.com/continuedev/continue/blob/f7116eaa3ffbddee85bad6ce428c3ff413f86605/core/llm/llms/Nebius.ts#L4

— Reply to this email directly, view it on GitHub https://github.com/continuedev/continue/issues/2805#issuecomment-2458423663, or unsubscribe https://github.com/notifications/unsubscribe-auth/A7RF7HSRGOZE25AF26IXHFLZ7FKIDAVCNFSM6AAAAABREZSPMCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINJYGQZDGNRWGM . You are receiving this because you were mentioned.Message ID: @.***>

[RomneyDa]

@demianarc also would love a contribution on the docs page! Original PR has no docs

For example, https://github.com/continuedev/continue/blob/dev/docs/docs/customize/model-providers/more/nvidia.md

Any Nebius specific notes, direction to the nebius docs, etc. would be great

[vadjs]

Hi @RomneyDa, I'm author of the original PR. Sorry for the missing documentation. I added it in the following PR as well as other improvements. https://github.com/continuedev/continue/pull/2875

[sestinj]

Thanks so much for this and your other PRs @vadjs ! Super glad to have Nebius as a built-in provider : )