[Competition] Error in online evaluation

[doubleZ0108]

Hi, thanks for your amazing work! I tried to make my very beginning submission for Track 1, I evaluate 500 images and generate disp.zip as the guidance. But when I submit the online evaluation on CodaLab, I got the following errors:

WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap.
Traceback (most recent call last):
  File "/tmp/codalab/tmpDDvi2f/run/program/evaluate.py", line 102, in <module>
    errors.append(compute_errors(gt_depth, pred_depth))
  File "/tmp/codalab/tmpDDvi2f/run/program/evaluate.py", line 11, in compute_errors
    thresh = np.maximum((gt / pred), (pred / gt))
ValueError: operands could not be broadcast together with shapes (17281,) (17281,640) 

I also observe the data shape of disp.npy as the figure shown:

How can I make a successful submission? Looking forward to your reply.

[ldkong1205]

Hi @doubleZ0108, thanks for providing this information!

The error message appears after reading all the predictions from your .zip file. This happens when computing the errors, i.e., from the following function:

def compute_errors(gt, pred):
    """
    Computation of error metrics between predicted and ground truth depths.
    """
    thresh = np.maximum((gt / pred), (pred / gt))
    a1 = (thresh < 1.25     ).mean()
    a2 = (thresh < 1.25 ** 2).mean()
    a3 = (thresh < 1.25 ** 3).mean()

    rmse = (gt - pred) ** 2
    rmse = np.sqrt(rmse.mean())

    rmse_log = (np.log(gt) - np.log(pred)) ** 2
    rmse_log = np.sqrt(rmse_log.mean())

    abs_rel = np.mean(np.abs(gt - pred) / gt)

    sq_rel = np.mean(((gt - pred) ** 2) / gt)

    return abs_rel, sq_rel, rmse, rmse_log, a1, a2, a3

We didn't encounter an error like this before. I would conjecture that the shape of your prediction is not correct. Could you try a squeeze the predictions before appending them to the list? For example:

pred_disps = []

with torch.no_grad():
    for data in dataloader:
        input_color = data[("color", 0, -1)].to(device)
        output = depth_decoder(encoder(input_color))
        pred_disp, _ = disp_to_depth(output[("disp", 0)], opt.min_depth, opt.max_depth)
        pred_disp = pred_disp.cpu()[:, 0].numpy()
        pred_disps.append(pred_disp)

This would generate a pred_disps of shape (500, 192, 640), instead of (500, 1, 192, 640).

Please let me know if this works for you. Thanks!

[doubleZ0108]

After changing the data shape to 3-dim like (500, 192, 640), I made a successful submission. Thanks for your timely reply.

[ldkong1205]

After changing the data shape to 3-dim like (500, 192, 640), I made a successful submission. Thanks for your timely reply.

Glad to hear this! Feel free to contact us again if you encounter any other problems.