huggingface / swift-transformers

Swift Package to implement a transformers-like API in Swift
Apache License 2.0
536 stars 46 forks source link

Allow unigram tokenizer to handle NSNumber score values #94

Closed ZachNagengast closed 2 months ago

ZachNagengast commented 2 months ago

For whatever reason, certain config files for unigram tokenizers end up reading scores as NSNumber instead of float, and throw malformedVocab errors.

Expected a Float but found __NSCFNumber

This doesn't convert directly to Float, so I'm adding one fallback here to attempt to convert it to float before throwing.

pcuenca commented 2 months ago

Interesting! Maybe because of precision issues?