search

abisee / pointer-generator

Code for the ACL 2017 paper "Get To The Point: Summarization with Pointer-Generator Networks"
Other
2.18k stars 812 forks source link

Output for the baseline model seems wrong after training for 962 steps #61

Closed LeenaShekhar closed 6 years ago

LeenaShekhar commented 7 years ago

I am training the baseline model by using the following command:

python run_summarization.py --mode=train --data_path=/pointer-gen/finishedfiles/chunked/train* --vocab_path=/pointer-gen/finished_files/vocab --log_root=/thesis/pointer-gen/log/ --exp_name=baseline --pointer_gen=False

I trained it for 962 steps and started decoding using the following command:

python run_summarization.py --mode=decode --data_path=/pointer-gen/finishedfiles/chunked/val* --vocab_path=/pointer-gen/finished_files/vocab --log_root=/pointer-gen/log/ --exp_name=baseline --max_enc_steps=400 --max_dec_steps=120 --pointer_gen=False

I get the following output:

INFO:tensorflow:Loading checkpoint /pointer-gen/log/baseline/train/model.ckpt-962 INFO:tensorflow:Restoring parameters from/pointer-gen/log/baseline/train/model.ckpt-962

INFO:tensorflow:ARTICLE: the missing philae space probe that bumped down onto the surface of a comet 284 million miles from earth could wake up from today . scientists at the european space agency have restarted efforts to listen for a signal from the stricken lander on comet 67p/churyumov-gerasimenko . they lost contact with the washing-machine sized spacecraft in november last year after it bounced several times before landing in a hollow below a cliff . starved of the sunlight it needed to power its suite of experiments , the lander 's batteries went flat after around just 60 hours on the surface . esa scientists have renewed efforts to contact the philae lander on comet 67p , shown in the illustration above . however , as the comet has travelled closer to the sun in its journey through the solar system , researchers hope philae will get enough light to come back to life . the rosetta probe , which was carrying philae , launched into space in 2004 , using the gravity of earth and mars to slingshot its way towards comet 67p . it chased comet 67p/churyumov -- gerasimenko through space for more than ten years , and entered orbit in august 2014 . after a four billion mile -lrb- 6.5 billion km -rrb- journey , the probe then successfully released the philae lander from its grip to land on the comet on 12 november 2014 , travelling at 1 metre -lrb- 40 inches -rrb- per second relative to the comet . but when it first made contact with the surface it failed to fire harpoons that would have kept it attached to the comet . this resulted in it bouncing to a height of 0.62 miles -lrb- 1km -rrb- above the comet before again landing on the surface . it then bounced again , but to a much lower altitude . scientists recently said that water on the comet was different to that on earth - suggesting water on our world came from an asteroid , not a comet . the european space agency has now turned on the communication unit on the rosetta spacecraft that is orbiting the comet . stephen ulamec , lander project manager at the german aerospace... CONTD

INFO:tensorflow:REFERENCE SUMMARY: european space agency has begun attempting to contact the lost probe . philae has been missing since november when it landed on comet 67p . scientists believe the lander bounced into a ditch in the shade of a cliff . but they say it may already be awake but not yet able to send messages . the probe needs 19 watts of power from its solar panels to communicate . as the comet gets closer to the sun the lander may receive more sunlight . scientists have sent commands to philae for it to begin warming itself up . they will attempt to contact the lander with rosetta over the next 8 days .

INFO:tensorflow:GENERATED SUMMARY: [UNK] [UNK] [UNK] of of of of . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

INFO:tensorflow:Wrote visualization data to /pointer-gen/log/baseline/decode/attn_vis_data.json

This output looks as it the model has not been trained at all. I expected a better result after these any steps.

Questions:

  1. Is there anything wrong in the way I am training? Or the model simply needs more time to train? Log for the last training step:

INFO:tensorflow:loss: 7.387758 INFO:tensorflow:running training step... INFO:tensorflow:Saving checkpoint to path /home/leena/Documents/thesis/pointer-gen/log/baseline/train/model.ckpt INFO:tensorflow:seconds for training step: 49.206

  1. Is it possible to make the baseline model available as well?
LeenaShekhar commented 7 years ago

Based on the Issue: https://github.com/abisee/pointer-generator/issues/7 I think I need to train it longer, which I intend to do anyway. I still expected some meaningful output so not sure whether something is wrong during training.

LeenaShekhar commented 7 years ago

Generated sentences seem to be better than before:

INFO:tensorflow:GENERATED SUMMARY: [UNK] [UNK] , the [UNK] , was a [UNK] ' in the new [UNK] 's [UNK] . the [UNK] was a [UNK] ' to be a [UNK] ' in the new [UNK] . the [UNK] was a [UNK] ' and [UNK] 's i 'm the ' to be a [UNK] '

I will keep training to get better results. Paper says something around 600, 000 iterations.