Converted msmarco-distilbert-cos-v5 model with convert_model.py fails when loaded with Tch tensor error: cannot find the tensor named distilbert.transformer.layer.0.attention.v_lin.bias

guillaume-be / rust-bert

Rust native ready-to-use NLP pipelines and transformer-based models (BERT, DistilBERT, GPT2,...)

https://docs.rs/crate/rust-bert

Apache License 2.0

2.67k stars 216 forks source link

Converted msmarco-distilbert-cos-v5 model with convert_model.py fails when loaded with Tch tensor error: cannot find the tensor named distilbert.transformer.layer.0.attention.v_lin.bias #453

Open owi1972 opened 7 months ago

owi1972 commented 7 months ago

I converted the model https://huggingface.co/sentence-transformers/msmarco-distilbert-cos-v5 with the current branch and the ./util/convert_model.py script

In the the example/sentence_embeddings_local.rs it says that you should use --prefix distilbert and --suffix for Distil-based models.

I tried with or without --prefix or --suffix but always got the same error when using the model in the sentence_embeddings_local.rs example

Tch tensor error: cannot find the tensor named distilbert.transformer.layer.0.attention.v_lin.bias

Is there an example on how to convert a model or model classes? What am I doing wrong?

linkedlist771 commented 7 months ago

I have also met the same error as you, this rustber only supports some models. This error raises because this model's layer has not been manually implemented yet. It seems that this project is not maintaining. I suggest you to use the onnx type to make inference. Here is a project about it https://github.com/nbigaouette/onnxruntime-rs.

guillaume-be commented 7 months ago

Hello,

The conversion script outputs the weights that are being converted. Can you please check if a prefix or suffix is mismatching the expected weights (i.e. distilbert.transformer.layer.0.attention.v_lin.bias)? Is the error always referring to distilbert.transformer.layer.0.attention.v_lin.bias or a different set of weights at each tryout?