There is now a mismatch between the predictions and feedback arrays in Agent.learn() where the predictions come out as pieces of the pydantic model generated from the skill's schema, and the feedback comes out as plaintext (the previous behavior), which causes false negative matches.
As learn() is currently implemented, I believe the previous behavior is correct, since the field names (eg Label) interfere with the model's understanding of the problem. However, if learn() was using structured generation end2end, for example like this: https://python.useinstructor.com/examples/examples/, then the new behavior could be an improvement. However, the scope of this ticket is just to resolve the mismatch.
See the comments in the latest commit for an example.
There is now a mismatch between the
predictions
andfeedback
arrays inAgent.learn()
where the predictions come out as pieces of the pydantic model generated from the skill's schema, and the feedback comes out as plaintext (the previous behavior), which causes false negative matches.As
learn()
is currently implemented, I believe the previous behavior is correct, since the field names (egLabel
) interfere with the model's understanding of the problem. However, iflearn()
was using structured generation end2end, for example like this: https://python.useinstructor.com/examples/examples/, then the new behavior could be an improvement. However, the scope of this ticket is just to resolve the mismatch.See the comments in the latest commit for an example.