Pin to a particular MultiPL-E revision (SantaCoder)

bigcode-project / bigcode-evaluation-harness

A framework for the evaluation of autoregressive code generation language models.

Apache License 2.0

745 stars 193 forks source link

Pin to a particular MultiPL-E revision (SantaCoder) #80

Closed arjunguha closed 1 year ago

arjunguha commented 1 year ago

I am about to publish the version of MultiPL-E used for StarCoder to the HF Hub. Before I do that, let's create a checkpoint of the eval harness that should reproduce the SantaCoder results.