Currently we're using tokens to construct match spans in the input bar. This requires a call to the tokenize endpoint in the function justGetTokensAndAddThemToIGCTextData, to prevent the originalInput and the token offsets and length from being out-of-sync. We want to instead use the match data to construct these spans.
Currently we're using tokens to construct match spans in the input bar. This requires a call to the tokenize endpoint in the function justGetTokensAndAddThemToIGCTextData, to prevent the originalInput and the token offsets and length from being out-of-sync. We want to instead use the match data to construct these spans.