What is simple LM method

[panwarnaveen9]

Just the question for clarification

• what is simple LM method ?
• How the sentence grammatically acceptability is being computed ?

Let's take example from paper -

a. The cats annoy Tim. (grammatical)
b. *The cats annoys Tim. (ungrammatical)

How do we check which having grammatically acceptability based in GPT-2

[Alicia-Parrish]

• what is simple LM method ?
• How the sentence grammatically acceptability is being computed ? Let's take example from paper -

a. The cats annoy Tim. (grammatical)
b. *The cats annoys Tim. (ungrammatical)

The simple LM method is a comparison of the probability assigned to two different sentences (so it's based on the probability of each whole sentence). Taking the above example, if the probability a model assigns to (a) is greater than the probability it assigns to (b), then we conclude that the model 'prefers' (a). The model is correct whenever P(a) > P(b) because, by design, (a) is always grammatical and (b) is always ungrammatical in this dataset. I think that addresses both your questions, but let me know if you have followup questions.

[panwarnaveen9]

Thanks a lot @Alicia-Parrish for quick response.

Just the follow up question on

• How do we get sentence "Probability" ? or Method of getting p(a) and p(b) ?

All the existing model generally provide sentence representation in form embeddings, which are generally of two types a) Representation/Embeddings of all word level tokens b) Representation /Embeddings at the end of the sequence

probability of each whole sentence

Just wondering how we compute this ? or how we go from embeddings provided by models to probabilities ?

[jorendorff]

@panwarvaneen9 The models compute P(x | s[..]), the probability that the next word in a sentence is x, given the preceding words s.

Then the sentence probability can be computed as P(s) = product(P(s[i] | s[..i]) for i in range(len(s)))).