search

xenova / transformers.js

State-of-the-art Machine Learning for the web. Run 🤗 Transformers directly in your browser, with no need for a server!
https://huggingface.co/docs/transformers.js
Apache License 2.0
11.04k stars 678 forks source link

Support transformers original behavior to load processors #808

Open hans00 opened 3 months ago

hans00 commented 3 months ago

As title.

I'm trying implement custom processor which has multiple tokenizer. The original behavior will help to load it.

xenova commented 3 months ago

Thanks for this! This is exactly what I was planning to do for v3. Can you give the model id of the processor/tokenizer you are loading so I can add a unit test?

hans00 commented 3 months ago

Thanks for this! This is exactly what I was planning to do for v3. Can you give the model id of the processor/tokenizer you are loading so I can add a unit test?

The processor still in WIP. https://github.com/mybigday/Bert-VITS2/tree/convert-hf/hf

this model uses multiple tokenizer.