search

openai / gpt-2

Code for the paper "Language Models are Unsupervised Multitask Learners"
https://openai.com/blog/better-language-models/
Other
22.36k stars 5.5k forks source link

Abstractive summarization #33

Closed bertagknowles closed 5 years ago

bertagknowles commented 5 years ago

How can I create an abstractive summary of a text document?

WuTheFWasThat commented 5 years ago

See paper for details!

bertagknowles commented 5 years ago

According to the paper "To induce summarization behavior we add the text TL;DR: ". Where do we add the document text to be summarized? Is it after this command at the prompt - "python3 src/interactive_conditional_samples.py " ? I tried to run this, but a huge error pops up if the sample text is long.

WuTheFWasThat commented 5 years ago

unfortunately I'm not sure if you can add newlines when using interactive_conditional_samples, so you may need to tweak the script - but normally it would be 

[article]
TL;DR:
HaleTom commented 5 years ago

For other googlers, section 3.6 of the paper relates to this, but the method is described in very high-level terms.

Also look at interactive_conditional_samples.py

WuTheFWasThat commented 5 years ago

@HaleTom if you have specific questions about the methodology feel free to ask! it really is very simple

bertagknowles commented 5 years ago

It seems the summary is based on the context of trained data and not the document I present to it. The model forms opinions on the global news and entertainment data and not on specific topic that I want.

WuTheFWasThat commented 5 years ago

I would guess that's probably a bug in your code

bertagknowles commented 5 years ago

I got my answer on quora that the summary is more imaginative than abstractive or extractive. See below:

_Example 9. Summarization

Input:

norway delivered a diplomatic protest to russia on monday after three norwegian fisheries research expeditions were barred from russian waters . the norwegian research ships were to continue an annual program of charting fish resources shared by the two countries in the barents sea region TL;DR:

Output:

Norwegian fishing boat was caught in a fishing accident in the Barents Sea.

The Norwegian fishing vessel was caught in a fishing accident in the barents sea.

Norway's Fisheries Research Program (FPR) was suspended on Tuesday after a Norwegian ship carrying a Norwegian fishing boat was caught in a fishing accident in the barents sea region TL;DR: Norway's Fisheries Research Program was suspended on Tuesday after a Norwegian ship carrying a Norwegian fishing boat was caught in a fishing accident in the barents sea region.

The Norwegian fishing vessel was caught in a fishing accident in the barents sea.

The Norwegian fishing vessel was caught in a fishing accident in the barents sea.

The Norwegian fishing vessel was caught in a fishing accident in the barents sea.

Notes. More on the imaginative side than abstractive or extractive. Again could be because of low model capacity , though even GPT-2 didn’t do well in summarization compared to supervised model performances_

ckpm commented 5 years ago

I don't think the summarization mode is implemented in the src

federicosan commented 4 years ago

Maybe the model was only trained for content generation, not text summarization.

0x1za commented 4 years ago

This worked for me python src/interactive_conditional_samples.py --nsamples=3 --length=100 --temperature=1

Then added [article] TL;DR: