from this figure, we know green 1 is conditioned on blue 0, so it is the final true token. therefore since red 2 is conditioned on green 1 and blue 0, red 2 should also the correct token.
now we already have correct token on position 1 and 2, we should verify ngram start from 2. why we only verify ngram start from 0, isn't it a waste?
from this figure, we know green 1 is conditioned on blue 0, so it is the final true token. therefore since red 2 is conditioned on green 1 and blue 0, red 2 should also the correct token.
now we already have correct token on position 1 and 2, we should verify ngram start from 2. why we only verify ngram start from 0, isn't it a waste?