search

lsdefine / attention-is-all-you-need-keras

A Keras+TensorFlow Implementation of the Transformer: Attention Is All You Need
702 stars 188 forks source link

Transformer encoder layer instead of Bidirectional LSTM #19

Open Eugen2525 opened 5 years ago

Eugen2525 commented 5 years ago

So I want to change below Keras bidirectional LSTM layer into Transformer encoder:

lstmLayer = keras.layers.Bidirectional( keras.layers.CuDNNLSTM(args.rnnSize, return_sequences = True, recurrent_initializer = 'glorot_uniform' ) )(inputLayer)

so can this be accomplished using your library? The rest of the code remains same, I just want to replace bidirectional LSTM layers with Transformer.

I would really appreciate your help. Thanks.

mostafaalishahi commented 4 years ago

@Eugen2525 did you figure it out how to do it?