Changing tokenization means there is no way to look up the saved annotation without manually retokenizing. This is because annotations appear to be indexed by the sentence string, not by a sentence ID.
In the meantime, the solution is to manually keep the sentences list in sync with any changes to the tokenization (and rerun initpriorities after any change).
Changing tokenization means there is no way to look up the saved annotation without manually retokenizing. This is because annotations appear to be indexed by the sentence string, not by a sentence ID.
In the meantime, the solution is to manually keep the sentences list in sync with any changes to the tokenization (and rerun initpriorities after any change).