search

hendrycks / apps

APPS: Automated Programming Progress Standard (NeurIPS 2021)
MIT License
414 stars 55 forks source link

Running instructions #17

Closed PulkitMadan closed 2 years ago

PulkitMadan commented 2 years ago

Reference: https://github.com/hendrycks/apps#how-to-use

Files train/README and eval/README are not present in the repository. Would really appreciate if the instructions are added for training and evaluation of the models.

xksteven commented 2 years ago

I'll check what happened to them. I remember they used to be there but maybe I'm mistaken. Will double check soon. Thanks for raising the issue.

PulkitMadan commented 2 years ago

Thanks. For reproducibility, under training-instructions, do you also plan to release the filtered dataset utilized during pretraining? Please let me know if you would like to raise a separate issue for this.

xksteven commented 2 years ago

I do not believe we plan to release the filtered dataset as the hosting costs are too much for the size of dataset.

xksteven commented 2 years ago

The links in the README were broken but the actual README instructions for train and eval were present in their respective folders. I've updated the links and will now close the issue.

Thanks for reporting it!