search

kpot / keras-transformer

Keras library for building (Universal) Transformers, facilitating BERT and GPT models
MIT License
533 stars 140 forks source link

decoder and encoder model #14

Open LuCeHe opened 5 years ago

LuCeHe commented 5 years ago

Hey!

nice code man!

Can you reproduce the results of the original code? If I understand correctly you only implemented the encoder side?

Best, Luca

IbrH commented 5 years ago

Not sure if this only an encoder, a decoder, or both, but there's an option for masking, so I'm assuming it's a decoder or the full transformer.

gregory112 commented 5 years ago

You can create the encoder/decoder as you wish. TransformerBody however, is an encoder. The wikitext example in this repo seems to be a universal transformer (a single TransformerBody is called multiple times, not stacked like plain transformer). I have tried building an encoder that has masking enabled (masking is originally enabled in the decoder).