search

dottxt-ai / outlines

Structured Text Generation
https://dottxt-ai.github.io/outlines/
Apache License 2.0
9.44k stars 479 forks source link

Update logits array in-place #859

Open brandonwillard opened 6 months ago

brandonwillard commented 6 months ago

What behavior of the library made you think about the improvement?

The current structured generation code is creating a -inf copy of the logits array and setting the allowed token ID indices to the corresponding values in the original logits array. See here.

How would you like it to behave?

When possible, the original logits array should be updated in-place and completely avoid creating a new array. This change would likely require the set of disallowed token IDs instead of the allowed ones.

brandonwillard commented 1 month ago

@rlouf @lapp0

lapp0 commented 1 month ago

This is dead code which should be cleaned up after

https://github.com/dottxt-ai/outlines/pull/1010

brandonwillard commented 1 month ago

This is dead code which should be cleaned up after

1010

The relevant code now seems to be here; is that what will be updated in #1010?

lapp0 commented 1 month ago

No, that is a problem, thanks for pointing it out. I'll look at it once my current task is complete.

brandonwillard commented 1 month ago

1192 does not appear to update the logits arrays in place, at least not without creating another array the size of the logits.

brandonwillard commented 4 weeks ago

Looks like this might need to be moved after https://github.com/dottxt-ai/outlines/issues/1206.