search

asyml / texar

Toolkit for Machine Learning, Natural Language Processing, and Text Generation, in TensorFlow. This is part of the CASL project: http://casl-project.ai/
https://asyml.io
Apache License 2.0
2.39k stars 372 forks source link

BERT as encoder and a transformer as a decoder. #269

Closed salokr closed 4 years ago

salokr commented 4 years ago

Hey there! I need to create a model which can perform BIO sequence labeling task and a generation task. Using the example in seq2seqatten.py I was able to create a single encoder and two decoders. Now, following the comment in an issue https://github.com/asyml/texar/issues/190#issuecomment-518776064 I replaced the Encoder with BERTEncoder but the results were very poor. The scores for the tags 'B' and 'I' were not even close to.10. Is this due to the preprocessing part? As in the beginning, PairedTextData is used to feed the data because of which correct indices from BERT's vocab are not fed to the decoder part?

Can you also please provide an example of using BERTEncoder and transformer decoder?

Thanks!

Bastian commented 4 years ago

You can use my repo as an example: https://github.com/Bastian/Abstractive-Summarization-of-Meetings. I'm using it to generate abstractive summaries of meetings, but it should be able to handle every other seq2seq task as well.

salokr commented 4 years ago

Thank you so much for the reply and example. Really appreciate it.