search

Fangyh09 / Image2LMDB

Convert image folder to lmdb, adapted from Efficient-PyTorch
66 stars 10 forks source link

Training speed #6

Closed hkunzhe closed 3 years ago

hkunzhe commented 3 years ago

Hi, thanks for this script! Will converting the ImageFolder to the LMDB file make training faster?

buzhangjiuzhou commented 3 years ago

It depends on your bottleneck of training. In my case, the cloud I using was based on NFS file system, which is surprisingly slow when loading large amount of small files. So it do faster 1.5 fold than before(Without it I train 2K steps per hour, and now 3K.).