Closed albertsgarde closed 4 months ago
This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.
SOrry for the late reply here, I think pyo3 is required for this TBH. We have example of this, we use wrappers and match types to switch between model
or decoder
objects.
I am implementing a part of my Python project in Rust. I need to use a tokenizer deep in the Rust code, but I want the Python code to decide what tokenizer to use. I would therefore like to pass a tokenizer from Python to Rust. Since the Python wrapper is supposed to be thin, I have hope that this is possible. If it is not, I'll probably just use Pyo3 to call Python from Rust, but that would both be uglier and feel silly to call Python code that only calls right back into Rust.
What I have tried looks something like this:
but Pyo3 gives me the error "the trait bound
&tokenizers::Tokenizer: pyo3::PyClass
is not satisfied". Looking at the source, this makes sense since it is thePyTokenizer
struct in the cratetokenizers-python
that wrapsTokenizer
and is turned into a Python class. If I could access thePyTokenizer
struct in Rust I might therefore be able to fix the problem, but sincetokenizers-python
is not available on crates.io this is difficult.Is there some way to pass a tokenizer from Python to Rust?