Unable to load all-MiniLM-L6-v2-f16.gguf OR all-MiniLM-L6-v2-f16.bin

[SidAtBluB0X]

I am trying to load an embedding model - all-MiniLM-L6-v2-f16.gguf.

I am interested in generating embeddings.

Here is the code I have tried, I always get an error when the model is loading.

  // Load model
  string modelPath = "./models/all-MiniLM-L6-v2-f16.gguf";

  var modelParams = new ModelParams(modelPath);            
  using var weights = LLamaWeights.LoadFromFile(modelParams);

  var embedder = new LLamaEmbedder(weights, modelParams);

Error says : [2024-03-04T03:04:30.115Z] llama_model_load: error loading model: unknown model architecture: 'bert' [2024-03-04T03:04:30.116Z] llama_load_model_from_file: failed to load model [2024-03-04T03:04:30.117Z] Fatal error. System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.

I also tried loading the model file as a bin as I also have a bin.

string modelPath = "./models/all-MiniLM-L6-v2-f16.bin";

Then I get, [2024-03-04T03:04:30.117Z] Fatal error. System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.

Any help would be appreciated.

Thanks

[AsakusaRinne]

It seems like the feature of bert support from llama.cpp 3 weeks ago.

@martindevans Would #565 support the bert model well or need some extra support?

[martindevans]

I believe #565 should support this model, but I haven't tested it. @SidAtBluB0X if you could pull that branch and test that model out it'd be very helpful :)

[abhiaagarwal]

I was able to load all-MiniLM-L6-v2-f16.gguf and embed in the 0.12.0 build!

[martindevans]

Thanks for confirming that