argilla-io / distilabel

Distilabel is a framework for synthetic data and AI feedback for engineers who need fast, reliable and scalable pipelines based on verified research papers.
https://distilabel.argilla.io
Apache License 2.0
1.45k stars 111 forks source link

Add `examples/arena_hard.py` and remove from `distilabel` core #741

Closed alvarobartt closed 3 months ago

alvarobartt commented 3 months ago

Description

As recently discussed internally, maintaining the benchmarks may be too time consuming and hard, since there are a lot of things going on and benchmarks tend to be maintained separately. So on, we internally decided to remove the implementation from the distilabel core for the moment, and provide that as an example instead, that anyone can use as reference for building benchmarks with distilabel.

codspeed-hq[bot] commented 3 months ago

CodSpeed Performance Report

Merging #741 will not alter performance

Comparing remove-benchmarks (b1ac3c7) with develop (356a4a3)

Summary

✅ 1 untouched benchmarks