How should I read and use optical flow from '.npz' files?

[MiaoJieF]

Here's my code. I am trying to read the optical flow to warp frame 2 back to frame 1. But the results appear to be wrong.

        flo = np.load(r'E:\Dataset\SHIFT\flow\0003-17fb\00000040_flow_front.npz')
        flo = flo['flow']
        img_1 = cv2.imread('video/00000040.jpg')
        img_2 = cv2.imread('video/00000041.jpg')

        flow_vis = flow_viz.flow_to_image(flo)
        cv2.imshow('flow_vis', flow_vis)
        cv2.imshow('img', img_1)
        cv2.imshow('img2', img_2)

        img_ten = torch.tensor(img_2).unsqueeze(dim=0).permute(0, 3, 1, 2).cuda().float()
        flow = torch.tensor(flo).unsqueeze(dim=0).permute(0, 3, 1, 2).cuda() * 1280

        img_ten_warp = warp(img_ten, flow)
        img_ten_warp = img_ten_warp.squeeze(dim=0).permute(1, 2, 0).cpu().numpy().astype(np.uint8)

        cv2.imshow('img_ten_warp', img_ten_warp)
        cv2.waitKey()

[suniique]

Hey @MiaoJieF, thanks for the question! It seems that in your code, you didn't map the optical flow range correctly. In the Carla simulator, the optical flow range is in [-2, 2] (you cloud check its official document https://carla.readthedocs.io/en/latest/ref_sensors/#optical-flow-camera). Also, the codes in https://github.com/carla-simulator/carla/issues/4795 might be helpful. Please come back to me if the problem persists!

[MiaoJieF]

Hey @MiaoJieF, thanks for the question! It seems that in your code, you didn't map the optical flow range correctly. In the Carla simulator, the optical flow range is in [-2, 2] (you cloud check its official document https://carla.readthedocs.io/en/latest/ref_sensors/#optical-flow-camera). Also, the codes in carla-simulator/carla#4795 might be helpful. Please come back to me if the problem persists!

Thank you for your reply. I still have a little doubt, is the optical flow provided in SHIFT from t->t-1 or t->t+1?