Tokenizer.from_bytes() not available in python bindings

RamvigneshPasupathy commented 3 months ago

Looking for a "Tokenizer.from_bytes()" support in python, similar to the one in Rust - https://github.com/huggingface/tokenizers/issues/1013

Currently, it is not available in the python bindings code - https://github.com/huggingface/tokenizers/blob/v0.19.1/bindings/python/src/tokenizer.rs

Why this is needed?

We have "safetensors" format to serialize models and them store as remote objects and which then can loaded back in memory and deserialized into Model objects safely - without any need for file write operations in an app server.
But in case of Tokenizers, I have to use a "zip" file for pre_trained tokenizers to save them as remote objects - where I have to download the remote zip and perform a file write operation in my server before loading the Tokenizer object back - performing this file write is not convenient. If I have to skip the file write operation, only way left for me is to save a pickled copy of my Tokenizer object created at runtime and then unpickled from my remote object server - which may not be a safer option.. Having a "from_bytes" option for Tokenizer can hence be helpful.

ArthurZucker commented 3 months ago

Would you like to open a PR to add this featuyr? 🤗

RamvigneshPasupathy commented 3 months ago

Hi @ArthurZucker

I was going through the code once more with a view of contributing the method that I asked Tokenizer.from_bytes(); but then I figured out that the feature that I am expecting is already available in a different method name Tokenizer.from_buffer().

Tried a PoC of loading the tokenizer from file bytes of a tokenizer.json, and it works. Attaching screenshots; Plz close this issue if you find the PoC is good and this code will be an enough reference for anyone who is using huggingface tokenizers..

Page 1	Page 2

ArthurZucker commented 3 months ago

yeah maybe update the doc to make from buffer more findable?

github-actions[bot] commented 2 months ago

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

huggingface / tokenizers

Tokenizer.from_bytes() not available in python bindings #1567