Function to compute word-level probabilities from calls to token_score

sathvikn commented 3 months ago

The current implementation sums surprisals over individual subword tokens with a new method called word_score, applicable to all LMScorer objects. It currently splits sentences on whitespaces and punctuation, we might need to add workarounds for special characters. This doesn't implement the fixes suggested in https://github.com/tpimentelms/probability-of-a-word/tree/main, but I thought I could make this PR as a starting point to get word-level measures.

Testing: I ran it with the example that splits the text into multiple subword tokens (confirmed for GPT2 and RoBERTa).

from minicons import scorer
gpt2 = scorer.IncrementalLMScorer("gpt2")
gpt2.word_score(stimuli)
[[('The', 0.0),
  ('sketch', -10.879680633544922),
  ('of', -2.51055908203125),
  ('those', -6.663204193115234),
  ('trucks', -8.962379455566406),
  ("hasn't", -8.682258605957031)],
 [('The', 0.0),
  ('sketch', -10.879680633544922),
  ('of', -2.51055908203125),
  ('those', -6.663204193115234),
  ('trucks', -8.962379455566406),
  ("haven't", -10.670646667480469)]]

Please let me know if you have any further suggestions/changes. Thanks in advance!

netlify[bot] commented 3 months ago

Deploy Preview for pyminicons canceled.

Name	Link
Latest commit	e3c568df6ded7ada6065da7b1beb93b327383f0e
Latest deploy log	https://app.netlify.com/sites/pyminicons/deploys/66a353fdf154ec0008648fa3

sathvikn commented 3 months ago

I could also add a warning if a sentence fails to get aggregated properly & just return the output of token_score so this fails elegantly.

sathvikn commented 3 months ago

As discussed, I implemented the warning and checked if a token is the beginning of sentence token. I tested this with both Llama and GPT2.

kanishkamisra / minicons