Autoregressive Transformer Model

[agemagician]

Hello,

First, I want to thank you for the great framework.

Second, the current implementation of transformer model uses non-autoregressive, correct ? If I want to switch to autoregressive transformer, what I need to modify in the framework to make it work ?

[orhanf]

hi, thank you for your interest. the current implementation is a vanilla Transformer, and it is autoregressive by default. unfortunately we don't have a non-autoregressive Transformer yet under Lingvo.

[agemagician]

Thanks a lot for your quick feedback. This is actually perfect for my use-case, because unfortunately, tensor2tensor library only supports non-autoregressive during training and only support both "autoregressive and non-autoregressive" during evaluation.

[agemagician]

Just to double check, you don't use the target output shifted as an input to the decoder. You only use the output of the decoder as input to the decoder in the next step, correct ?

[orhanf]

correct

[agemagician]

Great, thanks a lot for the quick response :)