Unable to load Quantized mistral

[AbhishekBose]

I am able to load quantised_mistral

For the model_id and revision I have chosen this

let model_id = "mistralai/Mistral-7B-v0.1".to_string();
let revision = "26bca36bde8333b5d7f72e9ed20ccda6a618af24".to_string();

let filenames = hub_load_safetensors(&api_repo, safetensors_file_name)?; where filenames is of Vec

What should I choose for the Varbuilder?

When I choose the Varbuilder form the quantised_mistral file?

QuantizedMistralVarBuilder::from_gguf(&filenames, &device);

I am getting this error

error[E0277]: the trait bound `Vec<PathBuf>: AsRef<std::path::Path>` is not satisfied
  --> src/model/backends/candle/text_generation.rs:39:57
   |
39 |                  QuantizedMistralVarBuilder::from_gguf(&filenames, &device);
   |                  -------------------------------------  ^^^^^^^^^ the trait `AsRef<std::path::Path>` is not implemented for `Vec<PathBuf>`, which is required by `&Vec<PathBuf>: AsRef<std::path::Path>`
   |                  |
   |                  required by a bound introduced by this call
   |
   = help: the following other types implement trait `AsRef<T>`:
             <Vec<T, A> as AsRef<Vec<T, A>>>
             <Vec<T, A> as AsRef<[T]>>
   = note: required for `&Vec<PathBuf>` to implement `AsRef<std::path::Path>`

And If I use

candle_nn::VarBuilder; let vb = unsafe { VarBuilder::from_mmaped_safetensors(&filenames, dtype, &device)? }; QuantizedMistral::new(&config, vb)?

I get

error[E0308]: mismatched types
   --> src/model/backends/candle/text_generation.rs:41:44
    |
41  |             QuantizedMistral::new(&config, vb)? 
    |             ---------------------          ^^ expected `VarBuilder`, found `VarBuilderArgs<'_, Box<...>>`
    |             |
    |             arguments to this function are incorrect
    |
    = note: expected struct `VarBuilder`
               found struct `VarBuilderArgs<'_, Box<dyn SimpleBackend>>`

What am I doing wrong here?

[R5byte]

If you are using Quantized model then you can load varbuilder like:

let vb =candle_transformers::quantized_var_builder::VarBuilder::from_gguf(filename, &device)?; let model = QMistral::new(&config, vb)?;

And for safetensors: let vb = unsafe { VarBuilder::from_mmaped_safetensors(&filenames, dtype, &device)? }; let model = Mistral::new(&config, vb)?;