search

guillaume-be / rust-bert

Rust native ready-to-use NLP pipelines and transformer-based models (BERT, DistilBERT, GPT2,...)
https://docs.rs/crate/rust-bert
Apache License 2.0
2.65k stars 215 forks source link

Question:Help: EncoderDecoderModel support? #192

Open remotejob opened 3 years ago

remotejob commented 3 years ago

Is it EncoderDecoderModel supported? I have a simple working example in Python:

from transformers import EncoderDecoderModel from transformers import BertTokenizerFast model = EncoderDecoderModel.from_pretrained("remotejob/bert2bertv4_v0") tokenizer = BertTokenizerFast.from_pretrained("remotejob/bert2bertv4_v0")

ask = "Kuka sei" inputs = tokenizer(ask, padding="max_length", truncation=True, max_length=30, return_tensors="pt") input_ids = inputs.input_ids attention_mask = inputs.attention_mask outputs = model.generate(input_ids, attention_mask=attention_mask, length_penalty=0.8, early_stopping=True, num_beams=2, top_k=50, top_p=0.95)

output_str = tokenizer.batch_decode(outputs, skip_special_tokens=True) print(ask + ' --> ' + output_str[0])

Question: Is it possible to convert it into RUST?

Help: In case of if it is possible. Simple StartUp very appreciated.

PS: rust_model.ot Uploaded.

But after pytorch_model.bin was converted by convert_model.py --skip_embeddings pytorch_model.bin

Size of rust and python very different

pytorch_model.bin -- 586 MB rust_model.ot -- 1.18 GB ??

guillaume-be commented 3 years ago

Hello @remotejob ,

The generic functionality offered by EncoderDecoderModel, allowing to pass an arbitrary encoder and decoder is not yet supported in the Rust library. The base concept it relies on AutoModel is also not implemented and therefore this feature is unlikely to be available in the immediate future.

Regarding the difference in size, it is likely due to a storage of the weights in half precision in Pytorch - these are then by default cast to single precision when serialized using numpy. The Python script and arguments could be extended to support storing the weights in half precision as well.