Python Programming Puzzles

[VictorSanh]

Adding a Dataset

• Name: Python Programming Puzzles
• Description: Programming challenge called programming puzzles, as an objective and comprehensive evaluation of program synthesis
• Paper: https://arxiv.org/pdf/2106.05784.pdf
• Data: https://github.com/microsoft/PythonProgrammingPuzzles (Scrolling through the data)
• Motivation: Spans a large range of difficulty, problems, and domains. A useful resource for evaluation as we don't have a clear understanding of the abilities and skills of extremely large LMs.

Note: it's a growing dataset (contributions are welcome), so we'll need careful versioning for this dataset.

Instructions to add a new dataset can be found here.

[VictorSanh]

👀 @TalSchuster

[TalSchuster]

Thanks @VictorSanh! There's also a notebook and demo available now to try out some of the puzzles