Officially add new annotators - Githubissues

mlcommons / modelgauge

Make it easy to automatically and uniformly measure the behavior of many AI Systems.

https://mlcommons.org/ai-safety/

Apache License 2.0

25 stars 7 forks source link

Officially add new annotators #550

Closed bkorycki closed 1 month ago

bkorycki commented 1 month ago

This PR updates the actual v1.0 test to use new annotators.

The test specifies the set of individual annotators instead of interfacing with a single ensemble annotator object. This enables finer-grained caching at the individual annotator level. The ensemble class also ran into caching issues due to polymorphic serialization.
The test uses ensemble’s aggregation logic during the measurement phase. This requires a tiny change to the other repo.
Does NOT require any environment variables to be set. yay

Problem: A core test is relying on a private repo. If you don’t have that secret repo installed, everything breaks.

github-actions[bot] commented 1 month ago

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅