Verifying the correctness of homography matrix

[VeeranjaneyuluToka]

Hi,

This is very nice implementation and it is very useful. However i have a question, is there anyway we can verify the correctness of homography matrix?

Thanks, Veeru.

[Vincentqyw]

If you only want to qualitatively assess the correctness of the estimated homography matrix, you can look at the 'warped image', which represents the view of image1 transformed to the perspective of image0 through H. If you want to quantitatively evaluate whether the H transformation is accurate, you can use the Hpatches dataset to evaluate the algorithm you have chosen. It provides the GT of H, but at this point, you need to use the API to obtain the estimated H and compare it with the true H.

[VeeranjaneyuluToka]

@Vincentqyw , thanks for quick reply!

How can i dump the warped image from the API that you have pointed in the above thread? and the below are matched and warped images

It does not really look to me correct however do you have any comments?

Thanks, Veeru!

[Vincentqyw]

The API has already provided some intermediate results. For example, pred = api(image0, image1), where pred is a dictionary containing the homography matrix H and the original images image0_orig and image1_orig.

https://github.com/Vincentqyw/image-matching-webui/blob/a46dee87161aaeadae0789bfe0a859ac928ed09f/test_app_cli.py#L74

You can utilize cv2.warpPerspective to obtain the warped image.

h0, w0, _ = image0_orig.shape
warped_image1 = cv2.warpPerspective(image1_orig, H, (w0, h0))

The two images you've shared appear to have been taken on the water's surface, making feature matching quite challenging. With much of the scene filled with sky and water, the presence of water ripples and moving clouds complicates feature matching.

Without more context about your application, I'm unable to offer further comments.

[VeeranjaneyuluToka]

Thanks for quick reply again!

I tried what you mentioned already i.e. extracting H and using warpPerspective from OpenCV but not getting satisfactory results (i do not see warp functionality implementation in github but webui shows the warped frame, so was wondering where is that warpPerspective call). Wrt your comment, what do you mean by feature matching is difficult? do you mean as we have week features (as scene is water and sky mostly), matching feature from src to dest is difficult and might not get correct homograpy matrix as it heavily dependent on right set of feature matches?

Here is my warpPerspective call and blending calls for your reference.

    H = preds["H"]

    # Extract conners of two images: top-left, bottom-left, bottom-right, top-right
    pts1 = np.float32(
        [[0, 0], [0, height_src], [width_src, height_src], [width_src, 0]]
    ).reshape(-1, 1, 2)
    pts2 = np.float32(
        [[0, 0], [0, height_dst], [width_dst, height_dst], [width_dst, 0]]
    ).reshape(-1, 1, 2)

    try:
        # Aply homography to conners of src_img
        pts1_ = cv.perspectiveTransform(pts1, H)
        pts = np.concatenate((pts1_, pts2), axis=0)

        # Find max min of x,y coordinate
        [xmin, ymin] = np.int64(pts.min(axis=0).ravel() - 0.5)
        [_, ymax] = np.int64(pts.max(axis=0).ravel() + 0.5)
        t = [-xmin, -ymin]

        # top left point of image which apply homography matrix, which has x coordinate < 0, has side=left
        # otherwise side=right
        # source image is merged to the left side or right side of destination image
        if pts[0][0][0] < 0:
            side = "left"
            width_pano = width_dst + t[0]
        else:
            width_pano = int(pts1_[3][0][0])
            side = "right"
        height_pano = ymax - ymin

        # Translation
        # https://stackoverflow.com/a/20355545
        Ht = np.array([[1, 0, t[0]], [0, 1, t[1]], [0, 0, 1]])
        src_img_warped = cv.warpPerspective(
            src_img, Ht.dot(H), (width_pano, height_pano)
        )
        # Generating size of dst_img_rz which has the same size as src_img_warped
        dst_img_rz = np.zeros((height_pano, width_pano, 3))
        if side == "left":
            dst_img_rz[t[1] : height_src + t[1], t[0] : width_dst + t[0]] = dst_img
        else:
            dst_img_rz[t[1] : height_src + t[1], :width_dst] = dst_img

        # blending panorama
        pano, nonblend, leftside, rightside = panoramaBlending(
            dst_img_rz, src_img_warped, width_dst, side, showstep=showstep
        )

        # croping black region
        pano = crop(pano, height_dst, pts)

Somehow color conversion is not working on the fused image (need to check further on this)

Note: I have many frames but most of the cases the above implementation fails with negative panaroma size. I really think that is due to wrong homography matrix as it has negatives and it cases negative size of final panaroma.

I have three cameras with the following placements Can we still use geometry type as homography?

My end goal is: i am keep getting frames from these three cameras, i need to stitch and show them as one.

Thanks, Veeru.

[Vincentqyw]

Wrt your comment, what do you mean by feature matching is difficult? do you mean as we have week features (as scene is water and sky mostly), matching feature from src to dest is difficult and might not get correct homograpy matrix as it heavily dependent on right set of feature matches?

The challenge with image matching is that the overlapping areas between the forward view and the left or right view are too small, and the scene is filled with water and sky, which complicates the image matching process.

My end goal is: i am keep getting frames from these three cameras, i need to stitch and show them as one.

I have a general understanding of what you want to do. Just to confirm, your three cameras are rigged and facing forward, left, and right, with some overlapping fields of view. If that's the case, it seems somewhat similar to the 3D Surround View used in vehicles. You might want to research the literature in this area and look for some solutions based on that. For example: 3-D Surround View for Advanced Driver Assistance Systems

[VeeranjaneyuluToka]

Thanks for the discussion and it really make sense your comments. I have had a look into the following discussion https://stackoverflow.com/questions/14954220/how-to-check-if-obtained-homography-matrix-is-good And plotted the some of the intermediate results and they do not make sense to me. So closing this here!