Using the flores101 or flores200 tokenizers is resulting in a memory leak.
I am using version 2.4.2 on Windows 11, but the same was also occurring on version 2.4.0.
Running the following results in memory usage increasing linearly until crashing:
import sacrebleuwhile True:sacrebleu.sentence_bleu("Hello world.", ["Hello world."], tokenize="flores101")
This is also the case for corpus_bleu.
I do not think that it is due to caching since I am running it over and over on the same sentence.
Using the flores101 or flores200 tokenizers is resulting in a memory leak. I am using version 2.4.2 on Windows 11, but the same was also occurring on version 2.4.0.
Running the following results in memory usage increasing linearly until crashing:
import sacrebleuwhile True:sacrebleu.sentence_bleu("Hello world.", ["Hello world."], tokenize="flores101")This is also the case for corpus_bleu. I do not think that it is due to caching since I am running it over and over on the same sentence.