shift of xy coordinates on the labeled_videos

[Di-Wang-AIND]

Hi, thanks for releasing lightning-pose. 

I found a bug when predicting a folder of videos using the script lightning-pose/scripts/train_hydra.py. The x and y coordinates of keypoints on the labeled videos will be shifted when the testing videos have different dimensions.

To avoid the shift of xy coordinates, I updated the image's original dimension for the testing video with the following codes before calling export_predictions_and_labeled_video():

        clip = VideoFileClip(video_file)
        cfg.data.image_orig_dims.width  = clip.w
        cfg.data.image_orig_dims.height = clip.h

Thanks, Di

[themattinthehatt]

good find! all of our videos have been the same size thus far, so this is a valuable suggestion. will fix this today

[Di-Wang-AIND]

Hi Matt, 

Thanks for the great work. I wonder if the lightning-pose could train the model on a labeled dataset that contains multiple dimensions. I found the predicted keypoints shifted when making a prediction on the frame with a different dimension from the first labeled image, and there was a big pixel error in the predictions_pixel_error.csv. The following code from the script lightning-pose/scripts/train_hydra.py was used.

    preds_file = os.path.join(hydra_output_directory, "predictions.csv")
    predict_dataset(
        cfg=cfg,
        trainer=trainer,
        model=model,
        data_module=data_module_pred,
        ckpt_file=best_ckpt,
        preds_file=preds_file,
    )
    # compute and save various metrics
    try:
        compute_metrics(cfg=cfg, preds_file=preds_file, data_module=data_module_pred)
    except Exception as e:
        print(f"Error computing metrics\n{e}")

Please let me know if you need more information. Thanks.

Best, Di

[themattinthehatt]

Hi @Di-Wang-AIND, good question.

We are currently working to resolve this issue in pursuit of adding some other features.

One option is to resize the frames and labels to all be the same size, though I understand this is not a great long term solution.

If you're feeling adventurous you can try out this branch: https://github.com/danbider/lightning-pose/tree/dynamic_crop We have removed data.image_orig_dims from the code there (except the FiftyOne module), but we have not yet tried training on labeled frames with different dimensions. If you go this route please keep me updated! (this branch is under active development, so if anything doesn't work as expected we can patch it quickly).

[Selmaan]

I think this should be supported in the latest code, I trained on a dataset with images of different heights (but same widths) and seemed to work!

On Mon, Nov 20, 2023 at 3:21 PM Matt Whiteway @.***> wrote:

Reopened #109 https://github.com/danbider/lightning-pose/issues/109.

— Reply to this email directly, view it on GitHub https://github.com/danbider/lightning-pose/issues/109#event-11016286869, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABJNQGSLAXAKWQYXC5Z2I3TYFO3WDAVCNFSM6AAAAAA3DPJVPOVHI2DSMVQWIX3LMV45UABCJFZXG5LFIV3GK3TUJZXXI2LGNFRWC5DJN5XDWMJRGAYTMMRYGY4DMOI . You are receiving this because you are subscribed to this thread.Message ID: @.***>

[themattinthehatt]

Hi @Di-Wang-AIND, just checking in to see if you've tried out the dynamic_crop branch - if you give it a shot please do let us know if you run into any issues.

[Di-Wang-AIND]

Hi Matt, I tried dynamic_crop branch and it looks good. I did not run into any issues.

Hi @Di-Wang-AIND, just checking in to see if you've tried out the dynamic_crop branch - if you give it a shot please do let us know if you run into any issues.

[themattinthehatt]

Glad to hear! Will close this issue now.