search

EleutherAI / project-menu

See the issue board for the current status of active and prospective projects!
65 stars 4 forks source link

[RFP] Token-length-weighted LM loss #30

Closed leogao2 closed 1 year ago

leogao2 commented 3 years ago

Background

(this idea due to alstroemeria313) Weight the loss of tokens by their decoded lengths, so that the loss objective corresponds to BPB rather than bits per token.

What to plot?

Obviously this will do better on BPB. Will it also do better on eval harness?