OOM at II. Joint Optimization in Training, Validation, and Testing

[Woolseyyy]

It keeps out of memory at II. Joint Optimization in Training, Validation, and Testing. Note that it is NOT out of GPU memory but cpu memory. And it seems to happen at for batch_i, batch in enumerate(datapipe_train): I run on a machine with 1 3090 GPU, 20 cpu cores, 80 GB memory. Any suggestion would help!

[xiumingzhang]

Try reducing MLP chunk size? Feel free to reopen this if you still have problems.

[Woolseyyy]

The code rquires at least 100GB cpu memory. I run on a 120GB memory machine and every thing is ok. Besides, it needs 2 3090 GPUs to avoid out of cuda memory.