convolution-autoregressive-nns

[utterances-bot]

Convolutions in Autoregressive Neural Networks

This post explains how to use one-dimensional causal and dilated convolutions in autoregressive neural networks such as WaveNet.

https://theblog.github.io/post/convolution-in-autoregressive-neural-networks/

[mlnoone]

Awesome, really well explained.. have been trying to understand this for a while. Thanks a lot!

[VigneshBaskar]

Hi Kilian, Thank you very much for the excellent article. In the section Mixing stride with dilation, the following statement does not hold for Pytorch

The Conv1D layer does not support specifying both a stride greater than one and a dilation rate greater than one.

It is possible to have strides as well as dilations greater than 1 in Pytorch. It is just a small remark but I thought its worth mentioning for Pytorch users. Here is a code snippet for interested readers:

import torch
from torch import nn

# Input
N, C_in, L_in = 1, 1, 18
test_input = torch.arange(1, 19, dtype=torch.float)
test_input = test_input.view(N, C_in, L_in)

# Convolution model
C_in = 1
C_out = 1
kernel_size = 2
stride = 3 # Later on try out stride = 1
dilation = 2
m = nn.Conv1d(in_channels=C_in, out_channels=C_out, kernel_size=kernel_size, stride=stride, dilation=dilation)
m.weight.data = torch.ones(1, 1, 2)
m.bias.data = torch.zeros(1)

# Ouptut
output = m(test_input)

print(f'Input: {test_input}')
print(f'Output: {output.data}')
print(f'Shape of input: {test_input.shape}')
print(f'Shape of output: {output.shape}')

And here is the output for the above code snippet:

Input: tensor([[[ 1.,  2.,  3.,  4.,  5.,  6.,  7.,  8.,  9., 10., 11., 12., 13., 14.,
          15., 16., 17., 18.]]])
Output: tensor([[[ 4., 10., 16., 22., 28., 34.]]])
Shape of input: torch.Size([1, 1, 18])
Shape of output: torch.Size([1, 1, 6])

[pranavraikote]

Excellent article. Keep up the good work :)