search

sinovation / ZEN

A BERT-based Chinese Text Encoder Enhanced by N-gram Representations
Apache License 2.0
642 stars 104 forks source link

how to initialize n-gram tower and emb? #25

Open FengYue95 opened 1 year ago

FengYue95 commented 1 year ago

Hi~

1、Is ZEN trained from any base bert（e.g. google） or trained from scratch? If from scrach, I guess the n-gram emb is randomly initialized, If from base bert, the n-gram emb maybe the average of characters included?

2、According to "We use the same parameter setting for the n-gram encoder as in BERT" in the paper，I want to know that the params of n-gram encoder is shared and the same with bert tower（maybe the bottom six layer?），or is initialized and trained independently?

thank you~

GuiminChen commented 1 year ago

您好，您的来信已收到，我会尽快回复您的邮件。

祝您生活愉快！

shizhediao commented 1 year ago
  1. There are two models in our paper. (R): randomly initialized parameters and (P): pre-trained model, which is the Google released Chinese BERT base model.
  2. Sorry I don't quite get your question, could you elaborate on it? Thanks