search

HumanSignal / Adala

Adala: Autonomous DAta (Labeling) Agent framework
https://humansignal.github.io/Adala/
Apache License 2.0
953 stars 75 forks source link

fix: DIA-1328: [adala BE] learn method is not working with structured generation #176

Closed matt-bernstein closed 2 months ago

matt-bernstein commented 2 months ago

There is now a mismatch between the predictions and feedback arrays in Agent.learn() where the predictions come out as pieces of the pydantic model generated from the skill's schema, and the feedback comes out as plaintext (the previous behavior), which causes false negative matches.

As learn() is currently implemented, I believe the previous behavior is correct, since the field names (eg Label) interfere with the model's understanding of the problem. However, if learn() was using structured generation end2end, for example like this: https://python.useinstructor.com/examples/examples/, then the new behavior could be an improvement. However, the scope of this ticket is just to resolve the mismatch.

See the comments in the latest commit for an example.

robot-ci-heartex commented 2 months ago

Commit https://github.com/HumanSignal/infra/commit/bd729b219fd57bd12c71473593ac89f63a53059b is created.

robot-ci-heartex commented 2 months ago

Commit https://github.com/HumanSignal/infra/commit/db85c836ee718cc00d4257f8b2a53898ff19ed63 is created.

robot-ci-heartex commented 2 months ago

Commit https://github.com/HumanSignal/infra/commit/ea7f8d7e626acb0d4a85ae15e014cb1a25ed6916 is created.

robot-ci-heartex commented 2 months ago

Commit https://github.com/HumanSignal/infra/commit/a37d9f07334179c5c2d49b7c39522810ecd8585c is created.

matt-bernstein commented 2 months ago

fixed in #168

robot-ci-heartex commented 2 months ago

Commit https://github.com/HumanSignal/infra/commit/820d95c5d042d5347023fbaa8c0d19b4cdcd794d is created.