Memory issue in running existing model with XLM roberta

[n-akram]

The current implementation runs out of memory (with one GPU system, of memory 12 GB). The problem exists for both Local GPU and google colab.

[n-akram]

What I have tried so far:

• Running on google colab TPU
• Running on google colab GPU
• Running on local GPU
• Running on local CPU

GPU and TPU always result in memory overflow error. Running on local machine results in ETA of 300+ hours.

[n-akram]

Potential solutions:

• Use xlm-base instead of xlm-large model.
• Use of parallelization in TPU

[n-akram]

Changed the model to xlm-base, and it is working in both google Colab and local GPU. Local GPU: 3 hours, Colab 9 hours ETA.

However, colab seems to use around 8.14 GB. Whereas local machine is using complete assigned GPU.

[n-akram]

Another implementation seems to use parallelization.

Next task is to try that today on TPU with xlm roberta base.