Memory gets full during training.

TUMFTM / CameraRadarFusionNet

Apache License 2.0

407 stars 131 forks source link

Open hume2011 opened 4 years ago

hume2011 commented 4 years ago

I got this issue when I tried to train model with distance regression on 8 blobs of nuScenes:

During training, the usage of memory continues to rise until the usage of swap reaches 100%, then the training stops, and no error is reported.

Please help me with this.Thanks!

config: batchsize=1

Fellfalla commented 3 years ago

@hume2011 This might be related to multiprocessing, which keeps opening new processes. Maybe the processes don't get shut down properly?

Davidleoncrespo commented 2 months ago

Anyone could solve that?