Evaluation metric details

[YNX940214]

Can you provide the evaluation metric that generate exactly result with the paper? Here is my "bicubic" method and evaluation metric matlab code, but it has a little difference with the paper score: (example for Dataset Lu and scale 16)

scale=16; count=0; total2=0; for i=1:6 depths_image=imread(sprintf('/home/xx/Depth_Enh/03_RGBD_Dataset/RGBD_0%d_output_depth.png',i)); rgb=imread(sprintf('/home/xx/Depth_Enh/03_RGBD_Dataset/RGBD_0%d_ouput_color.png',i)); depth=single(depths_image); [hei,wid,channel]=size(depth); max2=max(max(depth)); min2=min(min(depth)); depth = (depth-min2)/(max2-min2); depthGT = uint8(depth255); max1=255; min1=0; startpoint=scale; low1=depth(startpoint:scale:end,startpoint:scale:end); upsample1=double(imresize(low1,[hei,wid],'bicubic')); padding=6 upsample2=upsample1(padding+1:hei-padding,padding+1:wid-padding)255; depth2=depth(padding+1:hei-padding,padding+1:wid-padding)255; res2=upsample2-depth2; temp2=sum(sum(res2.^2)); num=(hei-2padding)(wid-2padding); avg2=sqrt(temp2/num) total2=total2+avg2; end total2/6

[sola303]

I use the same approach as you. For NYUv2 dataset,upsample scale factor is 8. the bicubic result is relatively 7 not 14 in the result Table.But I notice, in the paper, it says ''For the NYU v2 dataset [26], the depth values are measured in centimeter.'' it's not just scaled to the range [0,255] from range[0,1].I don't know what it means by measured in centimeter. @YNX940214 @Yijunmaverick

[YNX940214]

this is my code fragment for NYUv2 RMSE calculate:

depth_data=nyu_depth_test['data'][:] lossSum_modify=0 lossSum_original=0; lossSum_bicubic=0; count=0 for i in range(1000,1449): print(i) test_rgb=nyu_data['images'][i] test_rgb=torch.FloatTensor(test_rgb/255) i1=i-1000 depthGT=nyu_data['depths'][i] test_depth=torch.FloatTensor(depth_data[i1,:,:]) input_depth=test_depth min1=nyu_depth_test['min1'][i1] max1=nyu_depth_test['max1'][i1] test_depth = np.expand_dims(test_depth, 0) test_depth = np.expand_dims(test_depth, 0) test_rgb=np.expand_dims(test_rgb,0) rgb1=Variable(torch.from_numpy(test_rgb)).cuda() depth1=Variable(torch.from_numpy(test_depth)).cuda()

total11=cnn1.forward(rgb1,depth1)

total22=cnn.forward(rgb1,depth1)
k1=depthGT.shape[0]
k2=depthGT.shape[1]
# total11=depth1[0,0,padding:k1-padding,padding:k2-padding]
# res1=total11.cpu()[0][0].data
# res1=total11.cpu().data
res2=total22.cpu()[0][0].data
delta=max1-min1
bicubic_depth=input_depth*delta.item()+min1.item()
bicubic_depth=bicubic_depth[padding:k1-padding,padding:k2-padding]*100
# res1=res1*delta.item() + min1.item()
res2=res2*delta.item()+min1.item()
depthGT1=depthGT[padding:k1-padding,padding:k2-padding]   # can the problem lie in the index of matlab and python starts differently from 0 or 1?
# res=res[padding:k1-padding,padding:k2-padding]
# depthGT1=depthGT
depthGT1*=100
# res1*=100
res2*=100
thisloss2=((depthGT1-res2)**2).sum()
thisloss3=((depthGT1-bicubic_depth)**2).sum()
# print('original: ' + str(thisloss1) + " original modifyied" +str(thisloss2) + 'bicubic: '+str(thisloss3))
thisloss3/=(k1*k2)
thisloss3=thisloss3**0.5
# thisloss1/=(k1*k2)
# thisloss1=thisloss1**0.5
thisRMSE = thisloss2/(k1*k2)
thisRMSE = thisRMSE**0.5
lossSum_bicubic+=thisloss3
lossSum_modify+=thisRMSE
# lossSum_original+=thisloss1
count+=1

print('modify: '+str(lossSum_modify/count) + 'original : '+str(lossSum_original/count) + 'bicubic: '+str(lossSum_bicubic/count))

sorry for the mess, there are too many versions. And i tried another ECCV2016 paper for depth upsampling, but can't re-implement it. Maybe you can have a try. @sola303

[sola303]

@YNX940214 you're right.NYU v2 dataset is given in meters.I should do (im(max-min)+min)100 and then caculate the rmse. Now I get 14 when scale factor is 8 and method is bicubic,which corresponds to the result Table.I will try the author's CNN model when I have time and see whether it is right. If you have time,I suggest you to see the this @YNX940214 @Yijunmaverick https://hal.archives-ouvertes.fr/hal-01857016/file/kernelnetwork_tpami.pdf The author use bicubic downsample (we use nearest-neighbor) and his result is so good that I don't believe he use (im(max-min)+min)100 for NYUv2 dataset.(Maybe just rmse to im.) I don't know,what do you think.