Download script improvements

openai / gpt-2-output-dataset

Dataset of GPT-2 outputs for research in detection, biases, and more

MIT License

1.93k stars 548 forks source link

Open akx opened 5 years ago

akx commented 5 years ago

This PR:

makes sure the download script doesn't clobber any existing files if they seem correct enough (same size as remote)
ensures 404, etc. errors don't get written as outputs (by raising an error when they occur)
uses a larger read chunk size for a modest speed boost; there's no need to read and write things 1000 bytes at a time

WuTheFWasThat commented 4 years ago

sorry for getting to this so late, could you resolve the conflicts and i'll take another look?

akx commented 1 year ago

@WuTheFWasThat Rebased!