Unofficial PyTorch implementation of "Step-unrolled Denoising Autoencoders for Text Generation"
Work-in-progress, more experiments soon.
Currently implements unconditional character-level text generation on the text8 dataset.
Requires x-transformers
and ptpt
to be installed.
Run SUNDAE training using config file cfg.toml
:
python main.py --cfg-path cfg.toml
Sample text from trained model saved at ckpt.pt
:
python main.py --cfg-path cfg.toml --resume ckpt.pt --sample --nb-samples 16
Other useful flags:
--seed # set RNG seed
--no-save # disable saving of checkpoints [False]
--no-cuda # disable the use of CUDA device [False]
--no-amp # disable the use of automatic mixed precision [False]
--nb-workers # set number of dataloader workers. [4]
Step-unrolled Denoising Autoencoders for Text Generation
Nikolay Savinov, Junyoung Chung, Mikolaj Binkowski, Erich Elsen, Aaron van den Oord
@misc{savinov2021stepunrolled, title={Step-unrolled Denoising Autoencoders for Text Generation}, author={Nikolay Savinov and Junyoung Chung and Mikolaj Binkowski and Erich Elsen and Aaron van den Oord}, year={2021}, eprint={2112.06749}, archivePrefix={arXiv}, primaryClass={cs.CL} }