search

jlewi / foyle

Foyle is a copilot to help developers deploy and operate their applications.
https://foyle.io
Apache License 2.0
111 stars 10 forks source link

Speed up evaluation #301

Closed jlewi closed 1 month ago

jlewi commented 1 month ago

Experiments take a long time to run. It looks like the processing time per example is on average 1 minute.

I suspect the bulk of this processing time is due to waiting for blocklogs. Can we verify that using OTEL?

Our latest dataset have 424 examples. At 1 minute per example thats 7 hours. Thats way too slow for fast iteration.

Two things we could try to do to speed things up

  1. We could add a mechanism to flush logs and trigger sync'ing of logs immediately
  2. We could continue processing examples without while waiting for blocklog and data for previous example to be available.
jlewi commented 1 month ago

302 should hopefully good enough for now. In the future we might want to look into not blocking while waiting for the traces to be available.