Open more than once "lmdb.open()" in one process would lead to performance drop #23

[weiaicunzai]

Hi, thanks for your great work, sorry to bother, I do not know if this is the right place to ask my question.

Affected Operating Systems

• ubuntu18.04

Affected py-lmdb Version

'1.0.0'

py-lmdb Installation Method

conda install lmdb

Using bundled or distribution-provided LMDB library?

Bundled

Distribution name and LMDB library version

(0, 9, 24)

Machine "free -m" output

e.g.

              总计         已用        空闲      共享    缓冲/缓存    可用
内存：      128581        5857       17666         311      105057      121383
交换：        2047           2        2045

Other important machine info

Running under cgroups? Containers? Weird filesystems in use? Network filesystem? Patched kernel? ...

Describe Your Problem

I've tried to write my data into lmdb format to boost my pytorch performance, however, I've encountered a very strange situation. Here is the code snippet in my train.py:

    train_dataset = CamVid(
        'data',
        image_set='train',
        download=args.download
    )

    valid_dataset = CamVid(
        'data',
        image_set='val',
        download=args.download
    )

the nvidia-smi command gives me around 80% GPU-Util during training, and if I do this:

    train_dataset = CamVid(
        'data',
        image_set='train',
        download=args.download
    )
     valid_dataset = train_dataset   # do not initialize another dataset instance

GPU-Util immediately boost to 97%.

I've tried to save my training data and validation data into 1 lmdb file or separately, nothing changes, is there a reason why this would happen?Why use lmdb.open() function twice in a process would cause this difference, is there anything I did wrong? Thanks

There is my dataset script: https://github.com/weiaicunzai/pytorch-camvid/blob/refactor/optimize_transformations/dataset/camvid.py

Now my project code is on: https://github.com/weiaicunzai/pytorch-camvid/tree/refactor/optimize_transformations. If you need. Please note it's not on the master branch Simply do

python train.py -net segnet -b 5

to run my code

Thanks in advance.

Describe What You Expected To Happen

Use multiple lmdb.open()function in one process won't effect reading speed.

Describe What Happened Instead

Use multiple lmdb.open()function in one process affected the reading speed

[jnwatson]

This is completely speculation here, but if your data is already loaded from disk, your process will be waiting on disk less, and able to keep the GPU busy. I don't think this has anything to do with lmdb.

[weiaicunzai]

This is completely speculation here, but if your data is already loaded from disk, your process will be waiting on disk less, and able to keep the GPU busy. I don't think this has anything to do with lmdb.

Thanks, I've rewrite a lmdb dataset class, and above situation has disappeared, seems like the old code I wrote has come performance bottleneck.