search

cheind / autoregressive

:kiwi_fruit: Autoregressive Models in PyTorch.
MIT License
72 stars 5 forks source link

Reconsider the softmax distribution #24

Open cheind opened 2 years ago

cheind commented 2 years ago

PixelCNN++ argues that neighboring intensity usually correlate is not captured by the softmax distribution. Instead, they propose a mixture model consisting of the logitic distribution (like normal but with heavier tails). https://arxiv.org/pdf/1701.05517.pdf

cheind commented 2 years ago

also add bits-per-dim metric that measures how many bits are required to encode pixel intensities. see E.2 of https://arxiv.org/pdf/1705.07057.pdf https://uvadlc-notebooks.readthedocs.io/en/latest/tutorial_notebooks/tutorial11/NF_image_modeling.html

cheind commented 2 years ago

See also for how to optimize based on cdf https://github.com/Rayhane-mamah/Tacotron-2/issues/155