task: accept sentence(s) and pretty print score-table (shown as below), with scores for each token.
score --scorer incremental --model gpt2 --text "this is a sentence" --prob --logprob --surprisal --rank --device cpu
sentence_id token_id token prob logprob surprisal
1 1 this 0.0 0.0 0.0
2 2 is 0.80
score --scorer incremental --model gpt2 --file "path/to/file.txt"
# same thing but data grabbed from file.
CLI command for sentence score
sentence-score --scorer incremental --model gpt2 --text "this is a sentence" --prob --logprob --device cpu
sentence_id sentence prob logprob
1 this is a sentence 1e-09 -10.213
sentence-score --scorer incremental --model gpt2 --file "some path" --prob --logprob --device cpu
sentence_id sentence prob logprob
1 this is a sentence 1e-09 -10.213
2
cli command for score (prioritize this)
task: accept sentence(s) and pretty print score-table (shown as below), with scores for each token.
CLI command for sentence score