dottxt-ai / outlines-core

Structured generation in Rust
Apache License 2.0
124 stars 7 forks source link

Interface with `text-generation-inference` (TGI) #1

Closed rlouf closed 3 weeks ago

rlouf commented 3 months ago

From my understanding of the code in TGI (building the FSM and using the FSM to bias logits) and discussions on the hf-dottxt-outlines-collab TGI interacts with Outlines in the following way:

  1. The FSM is compiled in Python using RegexFSM
  2. The instance of RegexFSM is used to bias logits

While (2) can remain in Python we would like to port the logic necessary to build the FSM (1) to Rust. This means porting:

rlouf commented 3 weeks ago

If we need extra work to be done for this it would be better to open specific issues. Closing for now.