out of memory for summarization accuracy

[gilinachum]

Hi, I'm getting OOM when running with m5.large (8GB RAM). What's the minimum memory required? Worth putting in README

[keerthanvasist]

Thanks for reaching out @gilinachum. The memory requirement depends on the evaluation algorithm being used. I believe this can happen in cases where we use LLM as evaluator - for example, in toxicity evaluation algorithm we use LLMs to evaluate the toxicity of the model responses, and as you have found out, when we try to get BertScore for SummarizationAccuracy.

This tends to happen because the models are loaded on to memory. Since we use multi-processing through Ray, we load the model once per core on your machine. This can end up being too much for some machines. If you run into OOM in these cases, a good way to get around it by reducing the parallelization. You could do that setting an env PARALLELIZATION_FACTOR to a value that can be managed by your machine. For instance, you could start with PARALLELIZATION_FACTOR of 1, and then increase it until the max your computer can handle. By default, we try to use all the cores in your machine.

This information is not really captured in documentation as you pointed out. I will add this to our troubleshooting guide in the README.

[keerthanvasist]

I have created a PR to add this to the troubleshooting section in the README. I will close this issue now, but please feel free to reopen if you have other concerns.

[gilinachum]

TY. Didn't use toxicity, just BertScore.