search

mlopscommunity / open-questions-ai-quality

0 stars 0 forks source link

Specifying requirements for Generative AI #21

Open adamboazbecker opened 6 months ago

adamboazbecker commented 6 months ago

How do we best specify requirements given the massive action space of Generative AI?

skirmer commented 6 months ago

I'm not sure I quite understand the question. Are we talking about acceptance criteria for a finished model, or resource requirements going in to the development process, or something else?

dpbrinkm commented 6 months ago

Today I heard a story about a company that spent a good chunk of time trying to sort out edge cases with their evaluation test set only to find that when there was a new model release these problems went away.

I know it doesn't really answer the question and wait til there is a new model released but it does feel like there should be a way to game out what could change in the next 6 months that would turn everything we are working on up side down