search

dmlc / gluon-nlp

NLP made easy
https://nlp.gluon.ai/
Apache License 2.0
2.55k stars 538 forks source link

Beam search scorer question #1420

Closed carter54 closed 3 years ago

carter54 commented 3 years ago

Description

Hello , I have a question about the beam search scorer function scores = (log_probs + scores) / length_penalty length_penalty = (\frac{K + length}{K + 1})^\alpha https://github.com/dmlc/gluon-nlp/blob/0484e6494edf0a40c7bac220b5a10d8245324750/src/gluonnlp/sequence_sampler.py#L74

if K = 5, alpha = 2 image

if K = 5, alpha =1 image

if K =5, alpha = 0.5 image

In these three cases, the length_penalty function is decreasing when length > 0

The log_probs are all negative number, so the score is negative as well. a negative number divided by a decreasing function... output will alway prefer shorter sequences results...

carter54 commented 3 years ago

sorry, I made a mistake in drawing length_penalty function, it is a increasing function