tokenization: log spm usage only in debug mode to avoid console spamming

google-research / albert

ALBERT: A Lite BERT for Self-supervised Learning of Language Representations

Apache License 2.0

3.23k stars 571 forks source link

tokenization: log spm usage only in debug mode to avoid console spamming #268

Open stefan-it opened 2 years ago

stefan-it commented 2 years ago

Hi,

I'm not sure if the repo is currently maintained, but this PR increases the log level (from info to debug) for the SPM tokenization usage message. It will be displayed whenever a line is tokenized/detokenized which is very annoying, because the default log level is info.