search

yikangshen / Ordered-Neurons

Code for the paper "Ordered Neurons: Integrating Tree Structures into Recurrent Neural Networks"
https://arxiv.org/pdf/1810.09536.pdf
BSD 3-Clause "New" or "Revised" License
577 stars 101 forks source link

Tuning contextual embeddings with hierarchical relations #19

Closed goaaron closed 4 years ago

goaaron commented 4 years ago

I have a masked LM pretrained with bert.

The embeddings are poor on the sentence level, but do well for base tokens. There is a natural tree structure to my corpus that I believe stands to gain from something like on-lstm.

Do you think swapping out the embedding layer of the on-lstm with pretrained bert embeddings could be fruitful?

yikangshen commented 4 years ago

If you have a sentence-level task, I believe it's worth to try.