Currently we tokenize and normalize and operate with just the normalized values. CollateX holds original and normalized versions of each token as a complex object and currently ignore the non-normalized version after normalization. We need to get back to the non-normalized tokens for eventual output.
djbpitt
commented
1 year ago
Currently we tokenize and normalize and operate with just the normalized values. CollateX holds original and normalized versions of each token as a complex object and currently ignore the non-normalized version after normalization. We need to get back to the non-normalized tokens for eventual output.