Open samrat opened 6 months ago
Hi Samrat, thanks for catching this bug. You are correct that the JSON is being incorrectly enforced in that tutorial notebook.
Looking at the code, I see that the sample is setting the general structure of the JSON, but there is no stop token being enforced. This allows the underlying library to close the JSON object and continue with text generation. A quick fix is to add the ' quote character as a stop token for each of the gen calls for the fields.
Let me make that quick fix, and then I'll also think about whether there's a more elegant way to do this --- like having the first token of the next fixed string be a stop token (though that might create other surprises as an implicit behavior).
It is fixed? Can aici work with tiny llama (even 70M models?). Please update.
Yes, AICI does not place limits on the model size. However, you do need to have it integrated in whatever system you're using for inference!
Hello 👋,
When I try to generate JSON using the example in the
basics_tutorial
notebook with thephi2
model, the JSON generated doesn't conform to the structure specified:(The generation doesn't stop-- I had to interrupt the kernel for it to stop)
There is JSON at the beginning, but it is missing some keys(
quality_level
,excitement_level
).I realize that phi2 is a relatively small model and limited in capability, but my understanding was that
aici
would constrain the output forcing specific tokens. In this case, shouldn't it have ensured that the keys specified are output?