On-demand model loading feature load the same model multiple times

[captain-corgi]

I tried On-demand model loading on my Macbook M1. Every time I run my program, which interferes with LMS, it loads the same model again.

Steps to reproduce:

1. lms server start
2. Open a simple application that interferes with LMS. In my example, the LLM is gemma-2-2b-it.
3. After the program finishes, run it again.
4. You will see the same model loaded multiple times with the suffix :<index> by using lms status or viewing it from the LMStudio GUI.

LMStudio:

Terminal:

❯ lms status

   ┌ Status ───────────────────────────┐
   │                                   │
   │   Server:  ON  (Port: 1234)       │
   │                                   │
   │   Loaded Models                   │
   │     · gemma-2-2b-it - 1.71 GB     │
   │     · gemma-2-2b-it:2 - 1.71 GB   │
   │     · gemma-2-2b-it:3 - 1.71 GB   │
   │     · gemma-2-2b-it:4 - 1.71 GB   │
   │     · gemma-2-2b-it:5 - 1.71 GB   │
   │                                   │
   └───────────────────────────────────┘

My program:

package main

import (
    "context"
    "fmt"

    "github.com/tmc/langchaingo/llms"
    "github.com/tmc/langchaingo/llms/openai"
)

func main() {
    ctx := context.Background()

    model := "lmstudio-community/gemma-2-2b-it-GGUF/gemma-2-2b-it-Q4_K_M.gguf"
    baseUrl := "http://localhost:1234/v1"

    clientOptions := []openai.Option{
        openai.WithBaseURL(baseUrl),
        openai.WithModel(model),
        openai.WithToken("dummy-token"), // It always be dummy
    }

    llm, err := openai.New(clientOptions...)
    if err != nil {
        panic(err)
    }

    prompt := "What is the differences between Generative AI and Traditional AI?"

    completion, err := llms.GenerateFromSinglePrompt(ctx, llm, prompt)
    if err != nil {
        panic(fmt.Errorf("call LLM failed: %s", err.Error()))
    }

    fmt.Println(completion)
}

[yagil]

Thanks for the bug report @captain-corgi , we’re investigating

[yagil]

@captain-corgi thanks again for the detailed bug report, super helpful. We fixed this in 0.3.5 build 2.

Get it from: https://lmstudio.ai/download (you need to manually redownload it to get it, or wait for 0.3.6)

[captain-corgi]

Working perfectly! Thanks for all your hard work.