search

CarperAI / cheese

Used for adaptive human in the loop evaluation of language and embedding models.
MIT License
300 stars 24 forks source link

Pipeline can't send/receive data directly to/from model #21

Closed shahbuland closed 1 year ago

shahbuland commented 2 years ago

Multiple setups might require data to go to model before client, or to go to model than immediately client. Couple things preventing this currently.

  1. Pipeline doesn't have an event subscriber for model to publish to
    • Fix this by adding a subscriber to pipeline (easy)
  2. Client ends up waiting for model rather then getting new data in the case where model is the last to touch data before it goes to pipeline
    • Fix this by checking trip and comparing it to trip_max on client before deciding if we are going to put it into idle or waiting state