search

carlrobertoh / CodeGPT

JetBrains extension providing access to state-of-the-art LLMs, such as GPT-4, Claude 3, Code Llama, and others, all for free
https://codegpt.ee
Apache License 2.0
885 stars 184 forks source link

Support custom model deployed by vLLM as provider #590

Closed jsuii closed 3 weeks ago

jsuii commented 1 month ago

Describe the need of your request

Hello! I am running a server and using vLLM to deploy CodeQwen1.5. When choosing the provider for CodeGPT, it seems like none of the existing options work for me. Below are my three attempts to configure the provider.

  1. In the Custom OpenAI option, I entered the URL as shown by the screenshot below and clicked the test connection button, didn't work. customOpenAI-error

  2. In the LLaMA C/C++ (Local) option, I entered base host and the rest of the fields as the following, clicked OK, but entering stuff in the chat gave me the error message '{"detail":"Not Found"}'. LLaMA-error LLaMA-error2

  3. In the Ollama (Local) option, I entered base host, clicked Refresh Models, I got 'Unable to load models'. Ollama-error

I'm able to use my model in VS Code using the plugin Twinny. Here is a screenshot for the provider configuration I had in there. The major difference I noticed is that Twinny request a Model name where I can enter the path for vLLM to access the model. Twinny-configuration

Proposed solution

Support vLLM deployed CodeQwen1.5. For vLLM to work, there needs to be a space to specify the model name.

Additional context

I'm using: CodeGPT plugin version 2.7.1-241 Intellij IDEA Community Edition version 2024.1.2

jsuii commented 3 weeks ago

Found the solution by myself. It turns out that choosing the provider as Custom OpenAI and filling in the model name in the Body section (model = my_model_name) works!