geek-ai / Texygen

A text generation benchmarking platform
MIT License
863 stars 203 forks source link

How to prepare the data for text generation task. Thank you very much. #34

Open guotong1988 opened 5 years ago

guotong1988 commented 5 years ago

First, I'm not sure whether the model contains the encoder during training.

EOS means end-of-sentence. Encoder and decoder are part of transformer network.

If without-encoder, training time:

target: [E, F, G, H, EOS]
decoder input: [0, E, F, G, H]

If without-encoder, testing time:

decoder input: [0]

If with encoder, training time:

encoder input: [A, B, C, D]
target: [E, F, G, H, EOS]
decoder input: [0, E, F, G, H]

If with-encoder, testing time:

encoder input: [A, B, C, D]
decoder input: [0]

Am I exact right?

I know it is beyond the topic of this project, but hope you could help. Thank you and thank you.