Useful utility: auto download dataset

[pengzhenghao]

I come across snippet code downloading model automatically:

    def download_tokenizer(self):
        if self.is_verbose: print("downloading tokenizer params")
        suffix = '' if self.is_mega else '_mini'
        _ = requests.get(MIN_DALLE_REPO + 'config.json') # trigger HF download
        vocab = requests.get(MIN_DALLE_REPO + 'vocab{}.json'.format(suffix))
        merges = requests.get(MIN_DALLE_REPO + 'merges{}.txt'.format(suffix))
        with open(self.vocab_path, 'wb') as f: f.write(vocab.content)
        with open(self.merges_path, 'wb') as f: f.write(merges.content)

we can use this in this repo.

[pengzhenghao]

From here: https://github.com/kuprel/min-dalle/blob/main/min_dalle/min_dalle.py