search

karpathy / neuraltalk

NeuralTalk is a Python+numpy project for learning Multimodal Recurrent Neural Networks that describe images with sentences.
5.4k stars 1.32k forks source link

Update multi-bleu.perl #5

Closed elliottd closed 9 years ago

elliottd commented 9 years ago

Section 2.2 of Papineni et al. (2002) argues that the brevity penalty should be determined from the effective corpus to balance the precision-only effect of BLEU.

The current implementation of multibleu used in this project uses a fixed brevity penalty, which will give undue credit to overly short descriptions.

This commit reinstates the original intention behind the brevity penalty calculation.

karpathy commented 9 years ago

Hey Eliott,

The script, as currently presented, is exactly what's being used by Vinyals et al. and Kiros et al., with no penalty. In practice, I'm told this barely changes the results because most of the generated sentences are not too short, so in practice the computed BP ~= 1.

I'll keep it fixed for consistency. COCO is setting up evaluation server soon, so all of this will be properly standardized.