Model producing largely extractive summaries

[hate5six]

After much work I was finally able to get the model to train successfully (220k iterations, 400/100 max enc/dec steps, 3k iterations with coverage bringing the coverage loss to 0.2).

I'm noticing when running the model (with beam size = 4) on my own data (financial news) the decoded summaries are almost entirely extractive. What could be causing this? I've left the vocab size at the default size of 50k. Is it possible that this is a result of having too small of a vocab?

Example (bold added to source to show extracted regions):

Decoded

air line pilots association said friday that 79 % of voting aviators approved a deal running through january 2019 that the union said provided industry-leading pay and benefits .

investors are closely watching the outcome of contract talks involving other united labor groups and at other carriers , concerned about a repeat of previous industry cycles when market conditions deteriorated .

pilots at delta air lines inc and southwest airlines co. both rejected proposed deals last year . flight attendants have been unable to reach agreement on a joint contract even though they have been bargaining since 2012 .

Source

Pilots at United Continental Holdings Inc. overwhelmingly approved a two-year contract extension, continuing the momentum of the airline's efforts to restore labor peace and complete the integration of staff following its creation in a 2010 merger. The Air Line Pilots Association said Friday that 79% of voting aviators approved a deal running through January 2019 that the union said provided industry-leading pay and benefits. Investors are closely watching the outcome of contract talks involving other United labor groups and at other carriers, concerned about a repeat of previous industry cycles when record profits resulted in more generous deals that hobbled carriers' finances when market conditions deteriorated. Pilots at Delta Air Lines Inc and Southwest Airlines Co. both rejected proposed deals last year. The United contract provides for higher pay, restores benefits for previously furloughed pilots, and enhances scheduling rules for long-haul flights, according to the pilot union. Both sides declined to provide contract details, though a person familiar with the situation said it included a 13% pay rise this year followed by a 3% increase in 2017 and 2% in 2018. United executives said this week that the pilot contract and a new deal being considered by its technicians would raise its unit costs excluding fuel by 2.5 percentage points this year compared with 2015. The airline forecast its unit costs this year excluding fuel and the two labor deals would rise by between 0.5% and 1.5%. The airline has suffered from rocky labor relations since its 2010 merger with Continental Airlines and has been trying to smooth that friction under new management led by Chief Executive Oscar Munoz that took over in September. United has fresh deals or tentative agreements with the majority of its unionized staff, with the results on a new contract for its mechanics due to be revealed next week, but the toughest challenge remains securing a pact with flight attendants. Flight attendants have been unable to reach agreement on a joint contract even though they have been bargaining since 2012. United presented fresh proposals covering pay and work practices last week in talks brokered by federal mediators. Flight attendants this week held a world-wide protest in pursuit of a joint contract. The airline in November reached a deal to start negotiations with the International Association of Machinists union more than a year before the contract covering 30,000 ramp workers, customer-service agents and reservation staff opens for renewal at the end of 2016. It also reached a new proposed joint labor agreement for its 9,000 mechanics, with the International Brotherhood of Teamsters union due to issue the results on Jan. 25. Pilots at Delta, its only union-represented labor group, resumed contract talks last month after rejecting a deal endorsed by union leaders. Southwest pilots turned down a proposed deal in November, with flight attendants having rejected a tentative new pact earlier in the year.

[abisee]

Hi @hate5six,

Yes - the pointer-generator model produces mostly extractive summaries. This is discussed in section 7.2 of the paper. It is the main area for future work!

[hate5six]

Ah, great! Thank you so much for the clarification. Very excited I've been able to replicate the model. Your help and work has been greatly appreciated!

[MoAbd]

Hi @hate5six Can u share the model with us ? I need to test and try it but I don`t have a GPU for the training Thanks in advance.

[tianjianjiang]

@hate5six May I ask what are your final training and validating losses, and how do you decide it (they) converged? @abisee I would like to take this chance to ask the same questions to you, too. Although the paper indeed mentioned the final losses of pgen and coverage were 0.5 and 0.3, I'm not sure whether they're of training or of validating. Perhaps I misread it. Meanwhile it's also unclear to me that how to determine a convergence. Thank you!

[abisee]

@tianjianjiang

1. In the paper, when we report that the coverage loss drops from about 0.5 to about 0.2 during the coverage training phase, we're referring to the validation set.
2. When we say the average value of p_gen starts at around 0.30 and converges to around 0.53, we're also referring to the validation set.
3. Determining "convergence" of these values (or the overall loss function) is usually a very inexact empirical observation. Usually we deem the validation loss function to have "converged" when it has gone some time without lowering, or when it begins to increase.

[JafferWilson]

@hate5six Can you please share your checkpoints or model with us? I am in need of testing the model. Kindly help me.