IterablePipeline + Image Selection

CarperAI / cheese

Used for adaptive human in the loop evaluation of language and embedding models.

MIT License

300 stars 24 forks source link

IterablePipeline + Image Selection #17

Closed shahbuland closed 2 years ago

shahbuland commented 2 years ago

Changes:

IterablePipeline: Pipeline that reads from anything iterator, can be used to read from many different kinds of datasets
image_selection: Added extensively documented image selection task which makes use of IterablePipeline
audio_rating: Moved audio rating to its own example as to not clutter backend code
Fixed a bug where base pipeline had method named "get" instead of "post", whereas all subclasses made reference to "post"

shahbuland commented 2 years ago

Looks fine. My only comment is that I feel like we could provide extra magic to make our examples shorter (so that certain default cases are easy to build) but that can definitely come later

Yep, agreed. Best course of action is to abstract away BatchElements entirely. I have some solutions in mind but can make this a future PR. This should remove the need for adding any new Pipeline code at all.