Tokenizer's default behavior on unknown token

BiomedSciAI / fuse-med-ml

A python framework accelerating ML based discovery in the medical field by encouraging code reuse. Batteries included :)

Apache License 2.0

137 stars 34 forks source link

Tokenizer's default behavior on unknown token #375

Closed SagiPolaczek closed 1 month ago

SagiPolaczek commented 1 month ago

Support a configurable default value for the tokenizer op. Meaning we can change the default behavior of each instance, additionally to override it in the __call__() function itself.