search

zeno-ml / zeno-build

Build, evaluate, understand, and fix LLM-based apps
MIT License
484 stars 33 forks source link

Adjust chatbot example on re-run #137

Closed neubig closed 1 year ago

neubig commented 1 year ago

Description

I re-ran the chatbot example end-to-end to re-generate data in the new file format. In doing so I discovered a variety of little issues that I resolved:

  1. I implemented a search space that exactly reproduces all and only the systems in the online report using the new CompositeSearchSpace functionality. I also fixed a bug in creating report names when using this search space.
  2. The alpaca model was throwing an error with the most recent version of transformers, and Cohere was not responding well so I removed these (for now).
  3. I switched to using the ExhaustiveOptimizer to enumerate over exactly these systems.
  4. I made it so that evaluation results are saved after each run so that evaluation doesn't need to be run in duplicate.
  5. I bumped the version of zeno.

Blocked by