Closed xuy2 closed 6 years ago
I'd like to use the code you provided to finish seq2seq task such as POS and Named Entity Recognition. However, the preprocessing way in your code is to divide the words which are not frequent. I don't know how to use the divided word to finish Named Entity Recognition... Therefore, I'd like to retrain the transfer language model by myself. Can I kindly know the training time?
I'm trying to training the language model on Chinese. Maybe we can discussion some experience.
@xuy2 we're using this base model for sequence labeling tasks over at our fork of the repo. Docs for the SequenceLabeler
wrapper class are here. We've thought a little bit about how to deal with the problem of predictions being made at the subtoken level and opted to simply round predictions off to the nearest token as a post processing step. You can optionally disable this behavior in the config.
If you end up adapting the code in this repo you may want to consider adding one more attention block on top of the model to give the model some forward context, because as it currently stands future tokens are masked off from the model's representation of the sequence.
@madisonmay Thanks for your excellent work.
@wugh I sent you an e-mail to the address in your profile.
Have you released the code for language model training?