top-p < 1 fails inf assertion

[artsobolev]

When I set top-p in the arguments to a value less than 1, I have this assert failing. Which seems to make sense since this line puts them in.

What's the solution and what is the inf_mask needed for?

[t1101675]

inf_mask ensures that when the output distribution (probs) has zero values (like using top-p), logprobs will be assigned -float("inf") instead of nan. You can try commenting out this line to use top-p.