Bad performance on training

[xhzhao]

When i train this model(split=1) on GPU(M40), the training speed is so slow that i can hardly wait for the result. The training speed is about 3789sec/600iter, and the batchsize is 20, so one epoch(121512 images) is about 6075 iters. The total time will be 3121 hour to train the full 250 epochs.

Is this a normal training speed?

The training log is as follows : iter 600: 2.853175, 3.237587, 0.000398, 3986.911091 iter 1200: 2.764053, 2.890393, 0.000397, 7775.439858 iter 1800: 4.232092, 2.867312, 0.000395, 11630.438926 iter 2400: 3.718163, 2.826459, 0.000394, 15476.820226 iter 3000: 2.618628, 2.725287, 0.000392, 19317.520915 iter 3600: 1.765032, 2.695489, 0.000391, 23153.146176 iter 4200: 2.188749, 2.651189, 0.000389, 26987.516512 iter 4800: 2.323560, 2.658700, 0.000388, 30819.787844 iter 5400: 2.473269, 2.561281, 0.000386, 34647.776586 iter 6000: 1.226942, 2.620815, 0.000385, 38466.522832

When i train the mode from this github, the training speed is very fast and i could train the model to the target accuracy in 2 hours.

[zhwis]

Hello，when i run train.lua,but there are the following problems:

HDF5-DIAG: Error detected in HDF5 (1.8.16) thread 140697829553984:

000: ../../../src/H5Dio.c line 161 in H5Dread(): selection+offset not within extent

major: Dataspace
minor: Out of range

/home/caffe/torch/install/bin/luajit: /home/caffe/torch/install/share/lua/5.1/hdf5/dataset.lua:160: HDF5DataSet:partial() - failed reading data from [HDF5DataSet (83886080 /images_train DATASET)] stack traceback: [C]: in function 'assert' /home/caffe/torch/install/share/lua/5.1/hdf5/dataset.lua:160: in function 'partial' ./misc/DataLoaderDisk.lua:123: in function 'getBatch' train.lua:245: in function 'lossFun' train.lua:310: in main chunk [C]: in function 'dofile' ...affe/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:145: in main chunk [C]: at 0x00405d50

I don't know where the problem is,I use the dataset provided by the author .I hope to get your help

[jiasenlu]

Hi @xhzhao I wonder why not use larger batch size ? I think I use 300 as batch size.

[jiasenlu]

Hi @zhihuiwisdom I think this means there are some problem with the data prepossessing. Could you tell me how did you generate the Hdf5 data?

[zhwis]

@jiasenlu Thank you very much for your reply.you are right,I didn't get HDF5 data，because during the data process of my computer display not enough memory, So can you share the image feature data(.h5) with me? Or do you have any suggestions? Thank you

[panfengli]

@zhihuiwisdom
I made a little change as:

local ndims=4096
local batch_size = opt.batch_size
local sz=#img_list_train
local feat_train=torch.FloatTensor(sz, 14, 14, 512) --ndims)
print(string.format('processing %d images...',sz))
for i=1,sz,batch_size do
    xlua.progress(i, sz)
    r=math.min(sz,i+batch_size-1)
    local ims=torch.CudaTensor(r-i+1,3,448,448)
    for j=1,r-i+1 do
        ims[j]=loadim(img_list_train[i+j-1]):cuda()
    end
    net:forward(ims)
    feat_train[{{i,r},{}}]=net.modules[37].output:permute(1,3,4,2):contiguous():float()
    collectgarbage()
end

local train_h5_file = hdf5.open(opt.out_name_train, 'w')
train_h5_file:write('/images_train', feat_train)
train_h5_file:close()
feat_train = nil

local ndims=4096
local batch_size = opt.batch_size
local sz=#img_list_test
local feat_test=torch.FloatTensor(sz,14, 14, 512)
print(string.format('processing %d images...',sz))
for i=1,sz,batch_size do
    xlua.progress(i, sz)
    r=math.min(sz,i+batch_size-1)
    local ims=torch.CudaTensor(r-i+1,3,448,448)
    for j=1,r-i+1 do
        ims[j]=loadim(img_list_test[i+j-1]):cuda()
    end
    net:forward(ims)
    feat_test[{{i,r},{}}]=net.modules[37].output:permute(1,3,4,2):contiguous():float()

    collectgarbage()
end

local test_h5_file = hdf5.open(opt.out_name_test, 'w')
test_h5_file:write('/images_test', feat_test)
test_h5_file:close()
feat_test = nil

For me, it takes about 40 GB CPU memory for the new VQA v1.9 dataset. I think you could try to save the data into hdf5 file iteratively, but not as a whole.

[badripatro]

@zhihuiwisdom, size of vqa_data_img_vgg_train.h5 is 49 Gb and vqa_data_img_vgg_test.h5 is 37Gb .

Since size of files are huge, I am not able to share them.

I have extracted image feature by taking batch_size=10. I didn't face any memory issue. please try by decreasing batch_size.

On Sun, Apr 9, 2017 at 7:07 AM, Panfeng LI notifications@github.com wrote:

@jiasenlu https://github.com/jiasenlu For me, it takes about 40 G memory for the new VQA v1.9 dataset.

I made a little change as:

local ndims=4096 local batch_size = opt.batch_size local sz=#img_list_train local feat_train=torch.FloatTensor(sz, 14, 14, 512) --ndims) print(string.format('processing %d images...',sz)) for i=1,sz,batch_size do xlua.progress(i, sz) r=math.min(sz,i+batch_size-1) local ims=torch.CudaTensor(r-i+1,3,448,448) for j=1,r-i+1 do ims[j]=loadim(img_list_train[i+j-1]):cuda() end net:forward(ims) feat_train[{{i,r},{}}]=net.modules[37].output:permute(1,3,4,2):contiguous():float() collectgarbage() end

local train_h5_file = hdf5.open(opt.out_name_train, 'w') train_h5_file:write('/images_train', feat_train) train_h5_file:close() feat_train = nil

local ndims=4096 local batch_size = opt.batch_size local sz=#img_list_test local feat_test=torch.FloatTensor(sz,14, 14, 512) print(string.format('processing %d images...',sz)) for i=1,sz,batch_size do xlua.progress(i, sz) r=math.min(sz,i+batch_size-1) local ims=torch.CudaTensor(r-i+1,3,448,448) for j=1,r-i+1 do ims[j]=loadim(img_list_test[i+j-1]):cuda() end net:forward(ims) feat_test[{{i,r},{}}]=net.modules[37].output:permute(1,3,4,2):contiguous():float()

collectgarbage()

end

local test_h5_file = hdf5.open(opt.out_name_test, 'w') test_h5_file:write('/images_test', feat_test) test_h5_file:close() feat_test = nil

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/jiasenlu/HieCoAttenVQA/issues/25#issuecomment-292757584, or mute the thread https://github.com/notifications/unsubscribe-auth/AP3EvMgaF0Zn-g7xXxFc_T967X4kD_41ks5ruDZrgaJpZM4LlH2C .

--

With Regards, Badri Narayana Patro Ph: +91-9076237295