(this idea due to alstroemeria313)
Weight the loss of tokens by their decoded lengths, so that the loss objective corresponds to BPB rather than bits per token.
What to plot?
Obviously this will do better on BPB. Will it also do better on eval harness?
Background
(this idea due to alstroemeria313) Weight the loss of tokens by their decoded lengths, so that the loss objective corresponds to BPB rather than bits per token.
What to plot?
Obviously this will do better on BPB. Will it also do better on eval harness?