IBM / low-resource-text-classification-framework

Research framework for low resource text classification that allows the user to experiment with classification models and active learning strategies on a large number of sentence classification datasets, and to simulate real-world scenarios. The framework is easily expandable to new classification models, active learning strategies and datasets.
Apache License 2.0
97 stars 20 forks source link

Fix GPU OOM errors upon evaluation #1

Closed jysdoran closed 3 years ago

jysdoran commented 3 years ago

I was getting an OOM error with larger datasets even with a smaller batch size and model (base with batch 10): tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[10,100,3072] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [Op:Erf] The lines that were causing it are those changed in the commit. I found that keeping a list of the model calls was using uneccessary memory during evaluation and it works after this change. (Also I would like to add that I really appreciate how organised and well documented this repository is!)