search

uber-research / PPLM

Plug and Play Language Model implementation. Allows to steer topic and attributes of GPT-2 models.
Apache License 2.0
1.13k stars 202 forks source link

A different discriminator? #10

Open ehsan-soe opened 4 years ago

ehsan-soe commented 4 years ago

Hi,

I really enjoyed your paper. In this regard, I have following questions that I appreciate your reply,

1) I wonder if it possible to replace the current discriminator which is build on top of the LMHead with any other trained discriminator? In other words, does it really need to be built on top of the LM (p(x)) itself, or it can be any discriminator? 2) If the answer to previous is yes, then is it possible to fine-tune gpt-2 on our own data and then generate from that with our specific discriminator (attr model)?

mshlis commented 4 years ago

@ehsan-soe

  1. It needs to be built on top of the LM to get gradients for backprop (if otherwise you would have to use some form of expensive/noisy gradient approximation scheme to use the same approach)
  2. You can still fine tune this approach on your own data, just fine tune the descriminator ontop of the LM

(Note: i have no actual association to this project)

srulikbd commented 4 years ago

@mshlis so can I just replace GPT2 with like "distilbert-base-multilingual-cased"?

mshlis commented 4 years ago

@srulikbd sure but for what result? The goal of this is to do text generation which you can’t do effectively with distilbert. You probably want a backbone that is trained on next word/letter prediction.

srulikbd commented 4 years ago

OK, you are right. I saw experiments that tried to use BERT like LM but got bad results, because of the bidirectionality training. so I think I should find GPT2 that was trained on my specific language, or train one myself..

thanks for the quick answer!

liya-gafurova commented 4 years ago

Hello! Thank you for your great work!

In your article you have mentioned that instead of GPT2 it could be used another transformer-based text generator ("but the method applies in any representation space from any transformer-based text generator"). Is it possible to replace GPT2 with CTRL? And how to manage using CTRL with its different codes (like "Links", "Wikipedia", "Rewies", "Reddit " etc) within PPLM codebase?