How to apply the GeDi model on the models with different tokenizer?

salesforce / GeDi

GeDi: Generative Discriminator Guided Sequence Generation

https://arxiv.org/abs/2009.06367

BSD 3-Clause "New" or "Revised" License

208 stars 47 forks source link

How to apply the GeDi model on the models with different tokenizer? #20

Open victorup opened 12 months ago

victorup commented 12 months ago

Hi, I want to apply the GeDi model on a model with BertTokenizer. However, a dimension problem will occur when combining the probability distributions of two models. In addition, the tokens tokenized by two Tokenizers are different, so I can't match the two vocabularies directly. Is there any approach to applying the GeDi model? Thanks!