Open lapp0 opened 1 month ago
The idea is interesting. Do we really want to increase the surface area of the library? There's a risk of spreading ourselves too thinly by adding extra code to maintain.
Yes that's a risk but IMO it wouldn't substantially increase the maintenance requirements of the library:
Supporting your argument, there would be more PRs to review even if a downstream maintainer contributes, and for new processors there may be edge cases which require priority fixes.
Presentation of the new feature
Logits processors in outlines.processors support nearly every inference engine, offering a "write once, run anywhere" implementation of business logic.
Currently, vLLM, TGI, LoRAX, and xinference depend on Outlines for structured generation. Inference libraries can also utilize Outlines for various sampling techniques. To solidify Outlines' role in the LLM ecosystem, we should focus on developing robust logits processors that can be shared across multiple libraries.
A good case study is the new MLXLM library, which doesn't support structured generation and only supports a handful of sampling techniques
We can help downstream libraries by removing the potential complexity and bugs of structured generation.
A good starting point would be implementing:
LogitsProcessors for sampling augmentation:
Logits Processors for logging:
Where does it fit in Outlines?
We can expand our existing strong foundation of "write once, run anywhere"
outlines.processors
Are you willing to open a PR?
Yes, I have some progress as well https://github.com/lapp0/outlines/pull/35