Pay Less Attention with Lightweight and Dynamic Convolutions

[kweonwooj]

Abstract

• self-attention is strong, but its effect on long-range dependency is in question
• propose lightweight convolution and dynamic convolution, a convolution as a function of timestep which is lightweight and cost is linear in input length + performs better or on-par with self-attention in machine translation, summarization and language modeling
• in machine translation, WMT14 EnDe SoTA of 29.7 BLEU

Details

Background

• Depth-wise Convolution : performs convolution independently over every channel

  Light-weight Convolution

• weight sharing (H = 16) + softmax normalized depth-wise convolution
• DropConnect is used for regularization

  Dynamic Convolution

• timestep dependent kernel function + light-weight convolution

  Overall Structure

  

Results

• DynamicConv achieves 29.7 BLEU on WMT EnDe with same param count as Transformer Big
• Ablation
  • speed is 20% faster with DynamicConv

Personal Thoughts

• impressive result, improving both performance and speed against Transformer
• I wonder what timestep dependent kernel is capturing
• will the performance with small number of layers be equivalent? because CNN seem to gather contextual information via stacking, whereas self-attention can obtain global context in single operation

Link : https://openreview.net/pdf?id=SkVhlh09tX Authors : Wu et al. 2018

[demdecuong]

As i guess, We have GLU to expand the dimension into nx2d then we go to conv to rescale it into nxd right ? I still dont understand how to apply the softmax in LightweightConv . We would softmax all the kernel weight of the Conv layer right ?

Moreover, i am not clear about the weight-sharing of the author since i try to re-implement this architecture.

Please give me some explanation .

Thank you so much.