how to get RMSE

[zf-666]

I try to get the rmse by the groud_truth and output_depth folders generated by the model，I use _data_utils.load_depth_with_validitymap read the ground_truth and validity_map and _data_utils.loaddepth read the output_depth，but can't get the initial result. Thanks

images = []
for path in image_paths:
    image = data_utils.load_image(path)
    images.append(np.stack(image, axis=-1))

ground_truths = []
for path in ground_truth_paths:
    ground_truth, validity_map = data_utils.load_depth_with_validity_map(path)
    ground_truths.append(np.stack([ground_truth, validity_map], axis=-1))

output_depths = []
for path in output_depth_paths:
    output_depth = data_utils.load_depth(path)
    output_depths.append(np.stack(output_depth, axis=-1))

for i in range(length):
    ground_truth = ground_truths[i,:,:,:]
   # ground_truth = np.squeeze(ground_truth)
    output_depth = output_depths[i,:,:]

    validity_map = ground_truth[:, :, 1]
    ground_truth = ground_truth[:, :, 0]

    validity_mask = np.where(validity_map > 0, 1, 0)
    min_max_mask = np.logical_and(
        ground_truth > 0,
        ground_truth < 100)
    mask = np.where(np.logical_and(validity_mask, min_max_mask) > 0)
    #output_depth = output_depth[mask]
    #ground_truth = ground_truth[mask]

    mae[i] = eval_utils.mean_abs_err(1000 * output_depth, 1000 * ground_truth)
    rmse[i] = eval_utils.root_mean_sq_err(1000 * output_depth, 1000 * ground_truth)

    print(eval_utils.root_mean_sq_err(1000.0 * output_depths[i,:,:][np.where(validity_map > 0,1,0)], 1000.0 * ground_truths[i,:,:,0][np.where(validity_map > 0,1,0)]))
    imae[i] = eval_utils.inv_mean_abs_err(0.001 * output_depth, 0.001 * ground_truth)
    irmse[i] = eval_utils.inv_root_mean_sq_err(0.001 * output_depth, 0.001 * ground_truth)
    #print(rmse[i])
    # Compute mean metrics
mae   = np.mean(mae)
rmse  = np.mean(rmse)
imae  = np.mean(imae)
irmse = np.mean(irmse)

[alexklwong]

Hi @zf-666 which dataset are you using? If VOID, do you see that there are 800 files? If KITTI, are there 1000 files?

It looks like you are using KITTI based on this:

min_max_mask = np.logical_and(
        ground_truth > 0,
        ground_truth < 100)

Can you verify on your end?

We provide bash scripts for evaluation in

https://github.com/alexklwong/calibrated-backprojection-network/blob/master/bash/kitti/run_kbnet_kitti_validation.sh

and

https://github.com/alexklwong/calibrated-backprojection-network/blob/master/bash/void/run_kbnet_void1500.sh

Have you considered running those?

[zf-666]

Thanks a lot，I use VOID dataset and set wrong parameters. Thank you!