Set logits of entity related special tokens to -infinity

bigscience-workshop / metadata

Experiments on including metadata such as URLs, timestamps, website descriptions and HTML tags during pretraining.

Apache License 2.0

30 stars 12 forks source link

Set logits of entity related special tokens to -infinity #190

Closed manandey closed 1 year ago

manandey commented 1 year ago

Hi @tianjianjiang, as discussed with Timo, setting the logits of entity related special tokens to -infinity. Feel free to merge this PR if it looks good.