Open xd009642 opened 2 years ago
Alongside this, if the maintainers think it's a good idea, I would be happy to look into making the functions exposed from the rust crate generic over the type of the alphabet (right now, the alphabet must be a String
).
I'm doing this already in a local copy of the repo because I'm using this library to decode output from an ASR model, and the network output is a distribution over WordPiece tokens (which are not necessarily one letter characters).
I'm very happy to upstream improvements.
Sounds good. I'm also looking into adding language model support because I need it, so it will take me a while before I submit even the "expose-a-rust-API-and-make-it-generic" feature.
Can there be a version published for use within Rust code, as well as existing python and wasm versions? Seems a bit unfortunate some colleagues can use this in their python projects but I need to fork it to use it in Rust to remove the crate-type cdylib etc.