Closed ZachNagengast closed 2 months ago
For whatever reason, certain config files for unigram tokenizers end up reading scores as NSNumber instead of float, and throw malformedVocab errors.
Expected a Float but found __NSCFNumber
This doesn't convert directly to Float, so I'm adding one fallback here to attempt to convert it to float before throwing.
Interesting! Maybe because of precision issues?
For whatever reason, certain config files for unigram tokenizers end up reading scores as NSNumber instead of float, and throw malformedVocab errors.
Expected a Float but found __NSCFNumber
This doesn't convert directly to Float, so I'm adding one fallback here to attempt to convert it to float before throwing.