search

Lyken17 / Efficient-PyTorch

My best practice of training large dataset using PyTorch.
1.08k stars 139 forks source link

LMDB getting slower after iterations #27

Open mikezhang95 opened 2 years ago

mikezhang95 commented 2 years ago

I customize a dataset with LMDB format. Then I train the model in pytorch's DDP mode with 4 GPUs.

The training process is suddenly going down after some iterations (about 6 times slower). The memory doesn't change a lot.

Do you know the possible reasons?

bring728 commented 2 years ago

i have same problem..