lru_cache max size is not set in tokenizer_spm.py

mjpost / sacrebleu

Reference BLEU implementation that auto-downloads test sets and reports a version string to facilitate cross-lab comparisons

Apache License 2.0

1.07k stars 164 forks source link

lru_cache max size is not set in tokenizer_spm.py #217

Closed chikiulo closed 2 years ago

chikiulo commented 2 years ago

For this reason, memory consumption exploded when computing spBLEU (-tok spm) and setting SACREBLEU_DISABLE_CACHING environment variable does not seem to work for spBLEU (-tok spm) as well.

mjpost commented 2 years ago

Wow, that was bad! I just fixed this and released it as 2.3.1.