Problem running evaluate_kittic

olmerg commented 1 year ago

Dear kong,

When I try to run the valuation of kittic I get an error in tensor dimension

CUDA_VISIBLE_DEVICES=0 python3 zoo/MonoDepth2/evaluate_kittic.py --eval_mono --load_weights_folder "zoo/MonoDepth2/models/mono_1024x320"

Error RoboDepth/zoo/MonoDepth2/networks/depth_decoder.py", line 60, in forward x = torch.cat(x, 1) RuntimeError: Sizes of tensors must match except in dimension 1. Expected size 48 but got size 47 for tensor number 1 in the list.

I have changed the weights to other pretrained models and always I get this problem. The monodepth is working because depth_prediction_example is working without a problems.

thanks,

ldkong1205 commented 1 year ago

Hi @olmerg, thanks for putting up this problem. We will upload runnable .sh training scripts to the Repo in the next few days. We will let you know once they are ready.

hhuang-code commented 1 year ago

Hi @ldkong1205 ,

We encountered the same issue when running evaluate_kittic.py, and we found the "clean" folder under kitti_c is empty.

Another issue is the "gt_depths.npz" used in competition/evaluator_color.py file is not found, so how can I get this .npz file?

Thanks!

ldkong1205 commented 1 year ago

Hi @ldkong1205 ,

We encountered the same issue when running evaluate_kittic.py, and we found the "clean" folder under kitti_c is empty.

Another issue is the "gt_depths.npz" used in competition/evaluator_color.py file is not found, so how can I get this .npz file?

Thanks!

Hi @aaron-h-code, thanks for pointing out this! We will re-upload the evaluation sets and update the download link by tomorrow. We will let you know once they are ready.

ldkong1205 commented 1 year ago

Hi @aaron-h-code, we have updated the KITTI-C dataset. You can download the new one with the following link: https://drive.google.com/file/d/1bqd2fpVE0Ac-58H6zxk07rM_H3Lzbyf_/view

Alternatively, you can directly download them to the server by running:

wget --load-cookies /tmp/cookies.txt "https://docs.google.com/uc?export=download&confirm=$(wget --quiet --save-cookies /tmp/cookies.txt --keep-session-cookies --no-check-certificate 'https://docs.google.com/uc?export=download&id=1bqd2fpVE0Ac-58H6zxk07rM_H3Lzbyf_' -O- | sed -rn 's/.*confirm=([0-9A-Za-z_]+).*/\1\n/p')&id=1bqd2fpVE0Ac-58H6zxk07rM_H3Lzbyf_" -O kitti_c.zip && rm -rf /tmp/cookies.txt

Then unzip with:

unzip kitti_c.zip

Hope the above solves your problem. Please let me know if there is any other problem. Thanks a lot!

hhuang-code commented 1 year ago

Hi @aaron-h-code, we have updated the KITTI-C dataset. You can download the new one with the following link: https://drive.google.com/file/d/1bqd2fpVE0Ac-58H6zxk07rM_H3Lzbyf_/view

Alternatively, you can directly download them to the server by running:
wget --load-cookies /tmp/cookies.txt "https://docs.google.com/uc?export=download&confirm=$(wget --quiet --save-cookies /tmp/cookies.txt --keep-session-cookies --no-check-certificate 'https://docs.google.com/uc?export=download&id=1bqd2fpVE0Ac-58H6zxk07rM_H3Lzbyf_' -O- | sed -rn 's/.*confirm=([0-9A-Za-z_]+).*/\1\n/p')&id=1bqd2fpVE0Ac-58H6zxk07rM_H3Lzbyf_" -O kitti_c.zip && rm -rf /tmp/cookies.txt
Then unzip with:
unzip kitti_c.zip
Hope the above solves your problem. Please let me know if there is any other problem. Thanks a lot!

Hi @ldkong1205 , thanks for your new link for kitti_c.zip, and it works now.

Another issue is the "gt_depth.npz" used in competition/evaluator_color.py file doesn't exist:

gt_path = os.path.join(splits_dir, opt.eval_split, "gt_depths.npz")

Could you help provide such a file? Or how can we generate it by ourselves?

Thank you!

ldkong1205 commented 1 year ago

Hi @aaron-h-code, the gt_depths.npz serves as the ground-truth for the competition, and it is kept secret. For evaluation purposes, please submit your prediction file to the competition evaluation servers, and your model's performance will be automatically shown on the leaderboard.

If you have any questions regarding the preparation of competition submissions, please refer to the instructions listed here.

Please let me know if this works for you. Thanks!

hhuang-code commented 1 year ago

Hi @aaron-h-code, the gt_depths.npz serves as the ground-truth for the competition, and it is kept secret. For evaluation purposes, please submit your prediction file to the competition evaluation servers, and your model's performance will be automatically shown on the leaderboard.

If you have any questions regarding the preparation of competition submissions, please refer to the instructions listed here.

Please let me know if this works for you. Thanks!

Hi @ldkong1205 , thanks for your explanation!

Just to double check, to generate submission file, we run "evaluator_color.py" file, but have to comment out the lines below: https://github.com/ldkong1205/RoboDepth/blob/3c4766c155eaac09831d3a350fa51ed964f10d2e/competition/evaluator_color.py#L191

Thank you!

ldkong1205 commented 1 year ago

Hi @aaron-h-code, for the purpose of generating submissions for the competition:

Running evaluator_color.py is not necessary. All we need is actually the predictions in the numpy format.
You can use the code snippets from this instruction as an example for saving predictions into the .npy file (for Track 1) or .npz file (for Track 2).
Note that the order matters during evaluation. As such, please sort the sample list first before forwarding each one of them. Then you can append each prediction into pred_disps, save it, and compress it accordingly.
The last step would be submitting the compressed prediction file to the evaluation server. You will then receive an email for your evaluation status.

Please let me know if the above advice solves your questions. Feel free to comment more if you need extra help!

olmerg commented 1 year ago

Hi,

I have the model working, but I have trying different ways to send the results. the code that I am using is

    pred_disps = []
    # PREDICTING ON EACH IMAGE IN TURN
    with torch.no_grad():
        for idx, image_path in enumerate(paths):

            if image_path.endswith("_disp.jpg"):
                # don't try to predict disparity for a disparity image!
                continue

            # Load image and preprocess
            input_image = pil.open(image_path).convert('RGB')
            original_width, original_height = input_image.size
            input_image = input_image.resize((feed_width, feed_height), pil.LANCZOS)
            input_image = transforms.ToTensor()(input_image).unsqueeze(0)

            # PREDICTION
            input_image = input_image.to(device)
            features = encoder(input_image)
            outputs = depth_decoder(features)

            disp = outputs[("disp", 0)]
            disp_resized = torch.nn.functional.interpolate(
                disp, (original_height, original_width), mode="bilinear", align_corners=False)

            # Saving numpy file
            output_name = os.path.splitext(os.path.basename(image_path))[0]
            scaled_disp, depth = disp_to_depth(disp, 0.1, 100)

            #codelab
            pred_disp,_ = disp_to_depth(disp_resized,0.1,100)
            # pred_disp = scaled_disp
            pred_disp = pred_disp.cpu()[:, 0].numpy()
            pred_disps.append(pred_disp) 
    np.save("disp.npy", pred_disps)

After sent to codelab I receive: ValueError: operands could not be broadcast together with shapes (17281,) (17281,640)

ldkong1205 commented 1 year ago

Hi @olmerg, thanks for raising this question!

Could you check whether the paths contains the correct roots for evaluation images? For track 1, the total number of images is 500, which means that the saved numpy arrays should also have a length of 500.

olmerg commented 1 year ago

HI @ldkong1205 , I have a list of 500 with profundidty of array(1, 192, 640)

xhqi0109 commented 1 year ago

@olmerg ,Hello, I have the same problem:ValueError: operands could not be broadcast together with shapes (17281,) (17281,640) , could you please solve it

ldkong1205 / RoboDepth

Problem running evaluate_kittic #4