Can we add support for local LLMs? Or how would you recomend going about making the modifications?

[wakywayne]

Instead of using chat-gpt, I would like to try and use a local LLM. I am sure this would take some modifications, but I think we could potentially make this work, and would be an awesome addition to the plugin as local LLM's are surpassing gpt 3.5.

[andyl]

Ollama recently added openai compatibility. https://ollama.com/blog/openai-compatibility

I gave it a quick test, setting $OPENAI_API_HOST to localhost:11434. No luck. Not sure what changes are required to make this work.

[jaredmontoya]

Ollama recently added openai compatibility. https://ollama.com/blog/openai-compatibility

I gave it a quick test, setting $OPENAI_API_HOST to localhost:11434. No luck. Not sure what changes are required to make this work.

try this:

  {
  "jackMort/ChatGPT.nvim",
    event = "VeryLazy",
    config = function()
      require("chatgpt").setup({
        api_host_cmd = "echo http://localhost:11434",
        api_key_cmd = "echo ''",
        openai_params = {
          model = "codellama:13b",
          frequency_penalty = 0,
          presence_penalty = 0,
          max_tokens = 300,
          temperature = 0,
          top_p = 1,
          n = 1,
        },
        openai_edit_params = {
          model = "codellama:13b",
          frequency_penalty = 0,
          presence_penalty = 0,
          temperature = 0,
          top_p = 1,
          n = 1,
        },
      })
    end,
    dependencies = {
      "MunifTanjim/nui.nvim",
      "nvim-lua/plenary.nvim",
      "folke/trouble.nvim",
      "nvim-telescope/telescope.nvim"
    }
  }

[jaredmontoya]

My config that I provided above is not enough to call it "Local LLM Support". Model name gpt-3.5-turbo is still hard coded for ChatGPTRun actions so they are unusable, and ChatGPTCompleteCode probably has the same issue. I would say that the first step in supporting local LLM's would be to expose those definitions like it was done for openai_params and openai_edit_params.

Best case scenario in my opinion would be to make this plugin a fully backend neutral interface to LLM's in neovim because it's so good. I think LLM.nvim sounds good and it would be practical too because pressing \: and then typing LL(same key) then to complete to LLM is a more convenient than capital C lowercase h(keys further apart) then

I will understand if the maintainer won't like the idea of renaming the project to something more generalized(and OpenAI independent) and it is not necessary to do so in order to add local LLM support but I suggested doing so because I like the Idea of project names reflecting what projects do and ChatGPT.nvim suggests that only ChatGPT is supported.

[andyl]

I like the idea of swappable LLMs very much, but agnostic on repo name change. There is brand awareness already for 'ChatGPT.nvim'. (3.2K stars!) And as we know, OpenAI isn't "Open AI". :laughing:

[jaredmontoya]

That's exactly what I thought about when I said that the maintainer probably won't like my idea to change the name of the project and I am fine with any decision on rebranding it to LLM.nvim or AI.nvim. After all it's the maintainer's choice, not mine. But I think supporting local LLM's is important for this plugin, otherwise it will be sooner or later(probably a lot later than I would like) superceded by https://github.com/huynle/ogpt.nvim the same plugin but with nothing directly related to OpenAI in the name(GPT is not related to OpenAI, it means generative pre-trained transformer), it already supports multiple LLM providers including local LLM's (o in ogpt stands for Ollama) and it's commands are accessible by pressing \:O and \<Tab> (3 keys so more convenient) to autocomplete to OGPT*

The only reason that I suggest this is because having 2 plugins with developer attention split between them is worse than having ONE plugin that everyone works on. ogpt.nvim is less polished and more buggy than ChatGPT.nvim due to how it(for now) focuses primarily on delivering backend support and is less well-known, ChatGPT.nvim on the other hand does not support backends that ogpt.nvim does because it's original goal was to only support ChatGPT and nothing else because there were no alternatives at the time. Today ChatGPT.nvim is as I see in mostly maintanance and bugfixes mode with little to no enhancements made.

That's why I believe that it would be beneficial to everyone if those plugins that forked merged again under a name that represents new goals and gave us best of both worlds.

[wakywayne]

Couldn't agree more with @jaredmontoya teamwork is key in this space

[jaredmontoya]

@andyl Recently a project formerly named Ollama WebUI which is a frontend to Ollama(the backend that runs LLM's locally) changed it's name to Open WebUI to represent that it's a general frontend for LLM's and that it supports it's use with LLM backends other than Ollama, and that it is not a frontend specifically designed only for use with Ollama and if someone has issues with using it with something else their issues will be ignored as off-topic. This project has double the amount of stars(6.2k) ChatGPT.nvim has at the moment so I don't see a problem with rebranding based on that reason alone.

What I do see however is a possibility that some people won't like the idea of rebranding ChatGPT.nvim as AI.nvim or LLM.nvim even if it is for the better due to personal tastes and reasons like "ChatGPT.nvim sounds better" even if in my opinion it doesn't and if their reason is short-sighted.

What I am trying to explain here is that the name of the project is very important and it defines where the project is supposed to go even if it is not there yet as well as holds the project back if the goal set by the name is already met. Sometimes stopping is fine but in other cases it's not.

Names that are not related to what the project actually does like Linux, MacOS or Czkawka do neither of those things and that's why sometimes they are a wise choice in the longrun.

People don't feel comfortable writing a pull request that adds a feature that they are not sure is wanted in the project.

ChatGPT.nvim sure sounds like it's scope of backend support is limited to OpenAI and in fact is even more limited with actual support scope being gpt-3.5-turbo if ChatGPT.nvim already does well enough everything that it has to do as defined by it's name, then without a clear goal new features won't be added even if they are needed like for example local LLM support.

I personally think AI.nvim is a better name for an all in one plugin that supports all latest and greatest AI programming assistance technologies because if some better than LLM technology for programming with AI assistance is introduced in the future the name will be holding this project back again.

But just renaming this project won't be enough to make stuff happen. To deliver the promise of AI.nvim that I described it would require a lot of pull request reviews and because it might be too much for a single person, forming an organization might also be necessary(not that I know exactly how straining maintanance of a hypothetical AI.nvim is. It very well might be doable by a single person).

[andyl]

Dear @jaredmontoya - out of respect to the project maintainer, I believe it's a good practice to suggest ideas and alternatives and leave it at that. This project is wonderful for what is does, and perhaps the maintainer is satisfied with the name and scope of the current implementation, perhaps he is too busy to engage in discussion at the moment. There will be many GPT plugins, some focused on single inference provider, some will have pluggable backends, etc. Maybe you could publish a directory like "Awesome Nvim GPT Plugins", or start a new project, or make PR's to an existing project. Your ideas are good!

[jaredmontoya]

I understand. I don't expect the maintainer to just do the work because I said so or change the direction of the project because I said so. What I am saying here is not a demand but a suggestion that can easily be declined and I won't have a problem with that even if it is not something that I would like to happen.

I felt like an example of a project going through rebranding while having more stars was relevant because it might influence the decision that perhaps will be made here in the future by cancelling out the argument that 3k stars is too much for a rebranding.

What a lot of people would like to hear though, is if expanding the scope of the project will even be considered at all.

If expanding the project's scope will be declined, ChatGPT.nvim can stay as it is now, no problem. But the earlier it is clarified the better. Contributing to ogpt.nvim seems like a good option for now as creating another project will introduce more fragmentation.

[derekwhite]

Is it within scope to limit it to OpenAI API Compatible providers? Inference endpoints from places like Perplexity and TogetherAI offer this. I think all we need to do is not have models hardcoded in places and instead allow for host and model to be configurable and have that config propagate.

[jaredmontoya]

Is it within scope to limit it to OpenAI API Compatible providers? Inference endpoints from places like Perplexity and TogetherAI offer this. I think all we need to do is not have models hardcoded in places and instead allow for host and model to be configurable and have that config propagate.

I support this. I think that adding support for API's that are not compliant with OpenAI API will breed more non-compliant API's because their developers will see that they can get away with not using a format that is already widely accepted for no reason and anyone who wants to use those API's will need to implement support from scratch.

[wakywayne]

@jackMort what do you think?

[h4ckm1n-dev]

Ollama recently added openai compatibility. https://ollama.com/blog/openai-compatibility I gave it a quick test, setting $OPENAI_API_HOST to localhost:11434. No luck. Not sure what changes are required to make this work.

try this:

  {
  "jackMort/ChatGPT.nvim",
    event = "VeryLazy",
    config = function()
      require("chatgpt").setup({
        api_host_cmd = "echo http://localhost:11434",
        api_key_cmd = "echo ''",
        openai_params = {
          model = "codellama:13b",
          frequency_penalty = 0,
          presence_penalty = 0,
          max_tokens = 300,
          temperature = 0,
          top_p = 1,
          n = 1,
        },
        openai_edit_params = {
          model = "codellama:13b",
          frequency_penalty = 0,
          presence_penalty = 0,
          temperature = 0,
          top_p = 1,
          n = 1,
        },
      })
    end,
    dependencies = {
      "MunifTanjim/nui.nvim",
      "nvim-lua/plenary.nvim",
      "folke/trouble.nvim",
      "nvim-telescope/telescope.nvim"
    }
  }

There is an issue whern using this method for edit with instruction and add test as it's using GPT.3.5 as a model and not llama3. is it possible to set the API endpoint globaly ?

[jaredmontoya]

Ollama recently added openai compatibility. https://ollama.com/blog/openai-compatibility I gave it a quick test, setting $OPENAI_API_HOST to localhost:11434. No luck. Not sure what changes are required to make this work.

try this:

  {
  "jackMort/ChatGPT.nvim",
    event = "VeryLazy",
    config = function()
      require("chatgpt").setup({
        api_host_cmd = "echo http://localhost:11434",
        api_key_cmd = "echo ''",
        openai_params = {
          model = "codellama:13b",
          frequency_penalty = 0,
          presence_penalty = 0,
          max_tokens = 300,
          temperature = 0,
          top_p = 1,
          n = 1,
        },
        openai_edit_params = {
          model = "codellama:13b",
          frequency_penalty = 0,
          presence_penalty = 0,
          temperature = 0,
          top_p = 1,
          n = 1,
        },
      })
    end,
    dependencies = {
      "MunifTanjim/nui.nvim",
      "nvim-lua/plenary.nvim",
      "folke/trouble.nvim",
      "nvim-telescope/telescope.nvim"
    }
  }

There is an issue whern using this method for edit with instruction and add test as it's using GPT.3.5 as a model and not llama3. is it possible to set the API endpoint globaly ?

If you see something like here that is a hard coded model name then you can't change it unless you make a personal fork of the plugin or implement it's configuration and your pull request gets merged.

the add_tests task as well as any other task is defined here and I think there is a way to use a custom actions.json file but I don't remember how to and not all people will use some model x so every user who wants to use something other than default has to create a replacement for this file but with the default model replaced to what they want to use.

I am not sure about this one, edit with instructions model is defined in my config under openai_edit_params and if you changed it to the one you want to use for it to still not work then I don't know what's wrong.

[petershutt]

I got this to work with Ollama. Make sure you have the latest version:

https://ollama.com/blog/openai-compatibility

require("chatgpt").setup({
            api_host_cmd = 'echo http://127.0.0.1:11434',
            openai_params = {
                    model = "mistral",
                    frequency_penalty = 0,
                    presence_penalty = 0,
                    max_tokens = 4095,
                    temperature = 0.2,
                    top_p = 0.1,
                    n = 1,
            }
    })

[arnm]

I got this to work with Ollama. Make sure you have the latest version:

https://ollama.com/blog/openai-compatibility

require("chatgpt").setup({
            api_host_cmd = 'echo http://127.0.0.1:11434',
            openai_params = {
                    model = "mistral",
                    frequency_penalty = 0,
                    presence_penalty = 0,
                    max_tokens = 4095,
                    temperature = 0.2,
                    top_p = 0.1,
                    n = 1,
            }
    })

As mentioned in previous comment: https://github.com/jackMort/ChatGPT.nvim/issues/383#issuecomment-2106186657

This works with the ChatGPT command for chat, but ChatGPTRun has gpt-3.5-turbo seemingly hard-coded. I get the following error:

API ERROR: model 'gpt-3.5-turbo' not found,

FYI, I also had to add api_key_cmd` = 'echo -n ""' for ChatGPT to work

[HarrisDePerceptron]

@h4ckm1n-dev you can pass in edit pram through setup as well:

openai_edit_params = { 
     model = "<your model name>", 
     frequency_penalty = 0, 
     presence_penalty = 0, 
     temperature = 0, 
     top_p = 1, 
     n = 1, 
   }, 

This work for my openai compatible chat and "edit with instructions" but does not work with "add_test" or other options. seems like the endpoint "/completions" is deprecated by openai and not included in the compatible server @andyl

354