For #789 in SequenceGenerator if input_ids is empty, it fails.
Generally, models.transformers has an inconsistent implementation with other modules making the codebase harder to manage.
Solution
Use SequenceGeneratorAdapter for transformers instead of SequenceGenerator
Implement Transformers.generate and Transformers.stream which use model.generate(logits_processor=...) argument with outlines.processors.OutlinesLogitsProcessor
In draft until https://github.com/huggingface/transformers/pull/31448 makes it into a newtransformers
releaseFixes #789
Addresses #806 (does everything except remove the issues requirement of "remove
torch
")Closes https://github.com/outlines-dev/outlines/pull/910 (device inconsistency issue handled through other means)
Problem
SequenceGenerator
ifinput_ids
is empty, it fails.models.transformers
has an inconsistent implementation with other modules making the codebase harder to manage.Solution
SequenceGeneratorAdapter
for transformers instead ofSequenceGenerator
Transformers.generate
andTransformers.stream
which usemodel.generate(logits_processor=...)
argument withoutlines.processors.OutlinesLogitsProcessor
Additional Changes
Temporarily disablesstop_at
argument for transformers and implements test to determine whether upstream is fixed (https://github.com/huggingface/transformers/issues/31435)seed
instead ofrng
outlines.generate.cfg
for now (https://github.com/outlines-dev/outlines/issues/959)TODO:
GenerationConfig
defaultmax_tokens
being 20rng
withseed
in all tests and all documentation.transformers
fix for https://github.com/huggingface/transformers/issues/31435 sostop_at
works.