This patch updates the version of MultiPL-E used in the evaluation harness to the version that was used in the StarCoder paper. The previous version were the prompts used for the SantaCoder paper.
For the StarCoder paper, we fixed a number of bugs that were artificially lowering the scores for a couple of languages -- almost all typed languages. Most notable is that Java's performance will go up about 2-3% on StarCoderBase with this update.
This patch updates the version of MultiPL-E used in the evaluation harness to the version that was used in the StarCoder paper. The previous version were the prompts used for the SantaCoder paper.
For the StarCoder paper, we fixed a number of bugs that were artificially lowering the scores for a couple of languages -- almost all typed languages. Most notable is that Java's performance will go up about 2-3% on StarCoderBase with this update.