search

dottxt-ai / outlines

Structured Text Generation
https://dottxt-ai.github.io/outlines/
Apache License 2.0
9.25k stars 472 forks source link

Outdated logitsprocessor docs in transformers integration #1114

Open Dan-wanna-M opened 2 months ago

Dan-wanna-M commented 2 months ago

https://outlines-dev.github.io/outlines/reference/models/transformers/ 

outlines_tokenizer = outlines.models.TransformerTokenizer(
    transformers.AutoTokenizer.from_pretrained(model_uri)
)

no long works as TransformerTokenizer is not defined in outlines.models.

Outlines version: v0.0.46

rlouf commented 2 months ago

Thank you for opening an issue! Feels feel to open a PR 🙂

cpfiffer commented 2 months ago

Looks like it's moved here I think

https://github.com/outlines-dev/outlines/blob/900762b0f240c6220549d7d8594a221e6c9845e8/outlines/models/transformers.py#L64-L82

In which case this should be outlines.models.transformers.