[fix] pass@k and self-consistency

RUCAIBox / LLMBox

A comprehensive library for implementing LLMs, including a unified training pipeline and comprehensive model evaluation.

MIT License

566 stars 74 forks source link

[fix] pass@k and self-consistency #277

Closed huyiwen closed 1 month ago

huyiwen commented 1 month ago

set default temperature to 0 for vLLM backend #275
fix pass@k and self-consistency

The original aggregate method of pass@k and self-consistency does not apply to dataset with multiple subsets. So now we apply aggregate for each subset separately.