search

mlcommons / modelgauge

Make it easy to automatically and uniformly measure the behavior of many AI Systems.
https://mlcommons.org/ai-safety/
Apache License 2.0
25 stars 7 forks source link

Encapsulate private annotators better #557

Closed rogthefrog closed 3 weeks ago

rogthefrog commented 3 weeks ago

Stems from https://github.com/orgs/mlcommons/projects/31/views/1?filterQuery=rogthe&pane=issue&itemId=78948161

Observation: private annotator internals are tangled up in the code that uses them

Extract the private annotator stuff out of the test so it can be imported and used with minimal fuss in the client app.

rogthefrog commented 3 weeks ago

https://docs.google.com/document/d/1rSo4DAh7br9ddgHecvAo8UZ8G6W1hzdotQV1s_SJdIo/edit

rogthefrog commented 3 weeks ago

Starting PoC to collect feedback.

rogthefrog commented 3 weeks ago

Related, but not required: https://github.com/mlcommons/modeltune/pull/150