Does the text generation instructions work for the ByT5 models? (byte-by-byte T5)?

ELS-RD / transformer-deploy

Efficient, scalable and enterprise-grade CPU/GPU inference server for 🤗 Hugging Face transformer models 🚀

https://els-rd.github.io/transformer-deploy/

Apache License 2.0

1.64k stars 150 forks source link

Does the text generation instructions work for the ByT5 models? (byte-by-byte T5)? #140

Open NOT-HAL9000 opened 1 year ago

NOT-HAL9000 commented 1 year ago

As the question states, does transformer-deploy text generation work on the byt5 models? (google/byt5-small, google/byt5-base etc..)? Basically ByT5 works on raw UTF-8 bytes and can be used without a tokenizer. Thanks

model: https://huggingface.co/google/byt5-base

ayoub-louati commented 1 year ago

@NOT-HAL9000 For the moment we don't support ByT5 models but it should not be very different from T5 models. I think it can work with some minor manipulations and modifications in the present code.