Cannot select model when using remote Ollama

[hwvenancio]

I'm getting the following error when trying to use Ollama running on another machine:

2024-07-25 09:25:14,882 [ 612290] SEVERE - #com.devoxx.genie.service.PromptExecutionService - Error occurred while processing chat message
java.util.concurrent.CompletionException: com.devoxx.genie.service.exception.ProviderUnavailableException: java.lang.RuntimeException: status code: 404; body: {"error":"model 'llama-3-8b' not found, try pulling it first"}
        at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:315)
        at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:320)
        at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1770)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: com.devoxx.genie.service.exception.ProviderUnavailableException: java.lang.RuntimeException: status code: 404; body: {"error":"model 'llama-3-8b' not found, try pulling it first"}
        at com.devoxx.genie.service.PromptExecutionService.processChatMessage(PromptExecutionService.java:111)
        at com.devoxx.genie.service.PromptExecutionService.lambda$executeQuery$0(PromptExecutionService.java:68)
        at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1768)
        ... 3 more
2024-07-25 09:25:14,885 [ 612293] SEVERE - #com.devoxx.genie.service.PromptExecutionService - IntelliJ IDEA 2024.1.4  Build #IC-241.18034.62
2024-07-25 09:25:14,906 [ 612314] SEVERE - #com.devoxx.genie.service.PromptExecutionService - JDK: 17.0.11; VM: OpenJDK 64-Bit Server VM; Vendor: JetBrains s.r.o.
2024-07-25 09:25:14,907 [ 612315] SEVERE - #com.devoxx.genie.service.PromptExecutionService - OS: Linux
2024-07-25 09:25:14,908 [ 612316] SEVERE - #com.devoxx.genie.service.PromptExecutionService - Plugin to blame: DevoxxGenie version: 0.2.8

Looks like I cannot select the model when I try to use a remote Ollama, and it uses the one from when Exo was selected.

It show the models drop down when a local Ollama is running.

[stephanj]

Only tested it with local Ollama instances, so most likely a bug :)

[stephanj]

It tries to get the installed Ollama models in the class com.devoxx.genie.service.OllamaService in method getModels() So it should throw an UnsuccessfulRequestException("Unexpected code " + response);

    /**
     * Get the models from the Ollama service.
     *
     * @return array of model names
     * @throws IOException if there is an error
     */
    public OllamaModelEntryDTO[] getModels() throws IOException {
        String baseUrl = ensureEndsWithSlash(DevoxxGenieStateService.getInstance().getOllamaModelUrl());

        Request request = new Request.Builder()
            .url(baseUrl + "api/tags")
            .build();

        try (Response response = client.newCall(request).execute()) {
            if (!response.isSuccessful()) {
                throw new UnsuccessfulRequestException("Unexpected code " + response);
            }

            assert response.body() != null;

            OllamaModelDTO ollamaModelDTO = new Gson().fromJson(response.body().string(), OllamaModelDTO.class);
            return ollamaModelDTO != null && ollamaModelDTO.getModels() != null ? ollamaModelDTO.getModels() : new OllamaModelEntryDTO[0];
        }
    }

[stephanj]

Can you confirm this curl command returns a list of models?

Please run the curl command on the local machine accessing the remote Ollama instance?

curl http://[remote ip address]:11434/api/tags 

[hwvenancio]

It returns something like this:

$ curl http://[remote ip address]:11434/api/tags | jq .
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   991  100   991    0     0  51757      0 --:--:-- --:--:-- --:--:-- 52157
{
  "models": [
    {
      "name": "llama3:8b",
      "model": "llama3:8b",
      "modified_at": "2024-07-25T16:31:05.852972867-03:00",
      "size": 4661224676,
      "digest": "365c0bd3c000a25d28ddbf732fe1c6add414de7275464c4e4d1c3b5fcb5d8ad1",
      "details": {
        "parent_model": "",
        "format": "gguf",
        "family": "llama",
        "families": [
          "llama"
        ],
        "parameter_size": "8.0B",
        "quantization_level": "Q4_0"
      }
    },
    {
      "name": "codellama:13b",
      "model": "codellama:13b",
      "modified_at": "2024-07-25T16:25:39.193006661-03:00",
      "size": 7365960935,
      "digest": "9f438cb9cd581fc025612d27f7c1a6669ff83a8bb0ed86c94fcf4c5440555697",
      "details": {
        "parent_model": "",
        "format": "gguf",
        "family": "llama",
        "families": null,
        "parameter_size": "13B",
        "quantization_level": "Q4_0"
      }
    },
    {
      "name": "mistral:latest",
      "model": "mistral:latest",
      "modified_at": "2024-05-13T08:22:09.008741305-03:00",
      "size": 4109865159,
      "digest": "61e88e884507ba5e06c49b40e6226884b2a16e872382c2b44a42f2d119d804a5",
      "details": {
        "parent_model": "",
        "format": "gguf",
        "family": "llama",
        "families": [
          "llama"
        ],
        "parameter_size": "7B",
        "quantization_level": "Q4_0"
      }
    }
  ]
}

[hwvenancio]

I don't know if it's the same bug or if it's another, but here it's using a hard-coded localhost: https://github.com/devoxx/DevoxxGenieIDEAPlugin/blob/91c6882f069c037ea89f5eef0d1b217757cfc814/src/main/java/com/devoxx/genie/service/OllamaApiService.java#L23

        Request request = new Request.Builder()
            .url(OLLAMA_API_URL)
            .post(body)
            .build();

[stephanj]

Great find, that's obviously the issue.

Can you confirm with this new version of the plugin?

DevoxxGenie-0.2.9.zip

[hwvenancio]

Worked like a charm, thanks!