Why gpt-2 could apply to other tasks without fine-tune?

[guotong1988]

Thank you!

[gionapaolini]

Imho it's because of the dataset they have used.

It's huge and somewhat clean (they have taken specific text that had more than a specific score from reddit I believe, not sure). This means that in the dataset there were a lot example of QA (question-answer), TL;DR: (summarization), and so on.

Since the model is very good at learning patterns, if you set your input in a specific way (for example for summarization you concatenate TL;DR: at the end of the input), the model will recognize the pattern from the training dataset (TL;DR:) and try to generate sentences that follow the same pattern.

[MrKrzYch00]

My observations are:

• it could complete my item codes at work giving ideas what new ones could be maintaining the pattern (very good, didn't go off topic as well)
• could complete a bit of Twitch chat, but usually got confused and mixed it with forums and other stuff (good)
• could maintain your writing patter but sometimes gave me unclean followup (broken grammar, even though I tried my best to keep mine correct)
• acted funny when one writes a sci-fi story and puts some computer code, like ones brain is directed by it. It mostly liked to do C/PHP like syntax, writing anything else like Python or SQL usually resulted in unwanted output or switched to C/PHP altogether. (usually "if(something == somethingelse) {" produced quite funny code that could sometimes still relate to the normal human written input)

So imo, due to the various input it was trained on, it can't be perfect and there is a risk of unwanted output. I tried fine-tuning. Yes, it helps a lot to be on topic but only if your input follows your samples in a way as well. If not, your trained data will result in unwanted output then. Said so, I had not a lot data to train it with.

[guotong1988]

Thank you！

[shayneoneill]

I'm not gonna lie. The part of my brain where i stored my philsophy minor is currently running around with its hair on fire. This thing is terrifying, and I'm not sure if I can articulate why.

Its like someone actually built Searles chinese room, and it started asking for a payrise.