Scale new instances to new frame size

ssrinath22 commented 1 year ago

Description

Fixed the updated x and y values when copying an instance across frames so that the new x and y values are still within the boundaries of the new frame.

Types of changes

[x] Bugfix
[ ] New feature
[ ] Refactor / Code style update (no logical changes)
[ ] Build / CI changes
[ ] Documentation Update
[ ] Other (explain)

Does this address any currently open issues?

1551

Outside contributors checklist

[ ] Review the guidelines for contributing to this repository
[ ] Read and sign the CLA and add yourself to the authors list
[ ] Make sure you are making a pull request against the develop branch (not main). Also you should start your branch off develop
[ ] Add tests that prove your fix is effective or that your feature works
[ ] Add necessary documentation (if appropriate)

Thank you for contributing to SLEAP!

:heart:

Summary by CodeRabbit

New Feature:

Enhanced video frame resizing in the SLEAP GUI. The update ensures that when the size of a video frame changes, the coordinates of visible nodes are accurately scaled to fit within the new frame size. This improvement maintains the correct positioning of nodes, providing a more accurate and consistent user experience during video analysis.

Documentation:

Added a new entry for Lili Karashchuk with email lili.karashchuk@alleninstitute.org and affiliation with the Allen Institute of Neural Dynamics.

Refactor:

Modified the read_frames function in deeplabcut.py to improve efficiency and track information.

Bug Fix:

Fixed a spelling error in the summary() function of nn/system.py.

Style:

Updated the encoding process in the to_hdf5 function of video.py to optimize storage of video frames in the HDF5 file.

Test:

Modified the test case test_import_labels_from_dlc_folder() in test_commands.py to assert the correct number of tracks in the labels object.

Chores:

Updated dependencies to the latest versions.

Note: The changes in tests/io/test_formats.py and sleap/gui/commands.py are not categorized as they do not fall under any of the specified categories.

coderabbitai[bot] commented 1 year ago

Walkthrough

The introduced changes focus on improving the set_visible_nodes function in sleap/gui/commands.py. The new logic ensures that the coordinates of nodes are correctly scaled when the video frame size changes, maintaining the relative positions of the nodes within the frame.

Changes

File	Summary
`sleap/gui/commands.py`	The `set_visible_nodes` function now calculates new `x` and `y` coordinates based on the ratio of old and new frame sizes. This ensures that the nodes' positions are scaled correctly when the frame size changes.
`AUTHORS`	Added a new entry for Lili Karashchuk with email lili.karashchuk@alleninstitute.org and affiliation with the Allen Institute of Neural Dynamics.
`sleap/io/format/deeplabcut.py`	Various changes including import statements, modifications to the `read_frames` function, and updates to video file searching logic. These changes enhance the functionality and efficiency of reading frames and track information.
`sleap/io/video.py`	Changes to the `to_hdf5` function optimize the encoding and storage of video frames in the HDF5 file. The frames are now encoded as `int8` type and padded with zeroes. The dataset creation has been modified accordingly.
`sleap/nn/system.py`	A spelling correction in the `summary()` function, changing "Initalized" to "Initialized".
`tests/gui/test_commands.py`	Modification in the test case `test_import_labels_from_dlc_folder()`, changing the assertion for the number of tracks in the `labels` object.
`tests/io/test_formats.py`	Removal of a parameterization line and addition of a new test function `test_maudlc` with parameterization for different test data. The function performs various assertions on the `labels` object and checks the correctness of tracks.

🐇

"In the land of code, where the shadows lie,

A rabbit hops, under the binary sky.

With each key press, a new world unfurls,

In the dance of brackets, and the swirl of curls.

A change is made, a node takes flight,

Scaling with grace, in the video's light.

AUTHORS grow, new names appear,

Lili Karashchuk joins, spreading cheer.

DeepLabCut, a format refined,

Tracks and frames, in harmony aligned.

Video encoding, a touch so fine,

Frames stored with care, in HDF5's shrine.

System summary, a word corrected,

Attention to detail, never neglected.

Tests evolve, assertions renewed,

Labels and tracks, correctness pursued.

In this coder's realm, changes take flight,

A dance of progress, in the day and night." 🎉

Tips

### Chat with CodeRabbit Bot (`@coderabbitai`) - If you reply to a *review comment* from CodeRabbit, the bot will automatically respond. - To engage with CodeRabbit bot directly around the specific lines of code in the PR, mention @coderabbitai in your review comment - Note: Review comments are made on code diffs or files, not on the PR overview. - Add `@coderabbitai ignore` anywhere in the PR description to prevent this PR from being reviewed. ### CodeRabbit Commands (invoked as PR comments) - `@coderabbitai pause` to pause the reviews on a PR. - `@coderabbitai resume` to resume the paused reviews. - `@coderabbitai review` to trigger a review. This is useful when automatic reviews are disabled for the repository. - `@coderabbitai help` to get help. - `@coderabbitai resolve` to resolve all the CodeRabbit review comments. Note: For conversation with the bot, please use the review comments on code diffs or files. ### CodeRabbit Configration File (`.coderabbit.yaml`) - You can programmatically configure CodeRabbit by adding a `.coderabbit.yaml` file to the root of your repository. - The JSON schema for the configuration file is available [here](https://coderabbit.ai/integrations/coderabbit-overrides.json). - If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: `# yaml-language-server: $schema=https://coderabbit.ai/integrations/coderabbit-overrides.json`

codecov[bot] commented 1 year ago

Codecov Report

Merging #1568 (4367467) into develop (dbe14a8) will increase coverage by 0.01%. The diff coverage is 100.00%.

@@             Coverage Diff             @@
##           develop    #1568      +/-   ##
===========================================
+ Coverage    73.36%   73.38%   +0.01%     
===========================================
  Files          134      134              
  Lines        24012    24024      +12     
===========================================
+ Hits         17616    17629      +13     
+ Misses        6396     6395       -1

Files	Coverage Δ
sleap/gui/commands.py	`61.96% <100.00%> (+0.28%)`	:arrow_up:

... and 1 file with indirect coverage changes

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

roomrys commented 1 year ago

@coderabbitai review

roomrys commented 9 months ago

I've run into a bug that originates from this PR - but might actually just be a side-effect of another issue.

It looks like copy_instance has a frame attribute of None.

Is this the case for all PredictedInstances? No, just for the PredictedInstances that were created when I ran inference for my bottom-up id model with max-instances of 1:

[p_inst for p_inst in labels.predicted_instances if p_inst.frame is None]
[PredictedInstance(vi...core=0.00), PredictedInstance(vi...core=0.00), PredictedInstance(vi...core=0.00)]
special variables
function variables
0:
PredictedInstance(video=None, frame_idx=None, points=[head: (780.3, 676.1, 0.89), thorax: (795.8, 643.9, 0.71), abdomen: (808.2, 608.2, 0.54), wingL: (816.1, 595.6, 0.44), wingR: (808.2, 592.4, 0.42), forelegL4: (795.7, 683.9, 0.48), forelegR4: (771.7, 676.0, 0.51), midlegL4: (824.2, 660.0, 0.45), midlegR4: (771.7, 627.9, 0.39), hindlegL4: (827.7, 623.9, 0.41), hindlegR4: (787.6, 611.6, 0.40), eyeL: (792.5, 675.7, 0.87), eyeR: (776.1, 667.9, 0.85)], score=8.61, track=None, tracking_score=0.00)
1:
PredictedInstance(video=None, frame_idx=None, points=[head: (479.9, 663.9, 0.99), thorax: (492.3, 631.7, 0.70), abdomen: (508.0, 603.6, 0.42), wingL: (516.2, 583.8, 0.35), wingR: (511.9, 580.2, 0.31), forelegL4: (492.0, 672.0, 0.52), forelegR4: (468.1, 667.5, 0.59), midlegL4: (524.0, 644.3, 0.42), midlegR4: (467.6, 619.7, 0.36), hindlegL4: (524.1, 615.8, 0.39), hindlegR4: (480.2, 596.1, 0.36), eyeL: (492.0, 663.5, 0.95), eyeR: (472.8, 652.4, 0.89)], score=8.26, track=None, tracking_score=0.00)
2:
PredictedInstance(video=None, frame_idx=None, points=[head: (536.1, 472.4, 0.80), thorax: (556.3, 443.9, 0.67), abdomen: (584.3, 411.7, 0.58), wingL: (596.1, 396.2, 0.55), wingR: (592.1, 396.1, 0.54), forelegL4: (547.9, 484.0, 0.48), forelegR4: (524.3, 475.8, 0.43), midlegL4: (584.2, 464.3, 0.40), midlegR4: (531.8, 427.7, 0.35), hindlegL4: (599.6, 427.9, 0.41), hindlegR4: (563.9, 400.1, 0.34), eyeL: (551.8, 472.3, 0.76), eyeR: (532.3, 464.0, 0.71)], score=8.06, track=None, tracking_score=0.00)
len():
3

In BottomUpPredictor._make_labeled_frames_from_generator, we seem to make the LabeledFrame correctly (specifying the frame_ind): https://github.com/talmolab/sleap/blob/5fa10795f51fd25568c444dd95d370b86054acdc/sleap/nn/inference.py#L3251-L3257

Passing the list of PredictedInstances to the instances argument for LabeledFrame should then run through this LabeledFrame.__attrs_post_init__ method which adds the frame reference to each PredictedInstance: https://github.com/talmolab/sleap/blob/5fa10795f51fd25568c444dd95d370b86054acdc/sleap/instance.py#L1304-L1306

So what is going on? It seemed to be an issue that we faces when predictions were merged into an existing LabeledFrame. A solution was suggested in

1807

Steps

The steps I took to create the bug were:

Predicted on user-labeled frames
Double-clicked prediction to create a user Instance
Received error

Traceback

Traceback (most recent call last):
  File "d:\social-leap-estimates-animal-poses\source\sleap\sleap\gui\app.py", line 1592, in _handle_instance_double_click
    copy_instance=instance, mark_complete=mark_complete
  File "d:\social-leap-estimates-animal-poses\source\sleap\sleap\gui\commands.py", line 522, in newInstance
    mark_complete=mark_complete,
  File "d:\social-leap-estimates-animal-poses\source\sleap\sleap\gui\commands.py", line 241, in execute
    command().execute(context=self, params=kwargs)
  File "d:\social-leap-estimates-animal-poses\source\sleap\sleap\gui\commands.py", line 137, in execute
    self.do_with_signal(context, params)
  File "d:\social-leap-estimates-animal-poses\source\sleap\sleap\gui\commands.py", line 161, in do_with_signal
    cls.do_action(context, params)
  File "d:\social-leap-estimates-animal-poses\source\sleap\sleap\gui\commands.py", line 2870, in do_action
    from_prev_frame=from_prev_frame,
  File "d:\social-leap-estimates-animal-poses\source\sleap\sleap\gui\commands.py", line 2902, in create_new_instance
    mark_complete=mark_complete,
  File "d:\social-leap-estimates-animal-poses\source\sleap\sleap\gui\commands.py", line 2993, in set_visible_nodes
    old_size_width = copy_instance.frame.video.shape[2]
AttributeError: 'NoneType' object has no attribute 'video'
Traceback (most recent call last):
  File "d:\social-leap-estimates-animal-poses\source\sleap\sleap\gui\app.py", line 1592, in _handle_instance_double_click
    copy_instance=instance, mark_complete=mark_complete
  File "d:\social-leap-estimates-animal-poses\source\sleap\sleap\gui\commands.py", line 522, in newInstance
    mark_complete=mark_complete,
  File "d:\social-leap-estimates-animal-poses\source\sleap\sleap\gui\commands.py", line 241, in execute
    command().execute(context=self, params=kwargs)
  File "d:\social-leap-estimates-animal-poses\source\sleap\sleap\gui\commands.py", line 137, in execute
    self.do_with_signal(context, params)
  File "d:\social-leap-estimates-animal-poses\source\sleap\sleap\gui\commands.py", line 161, in do_with_signal
    cls.do_action(context, params)
  File "d:\social-leap-estimates-animal-poses\source\sleap\sleap\gui\commands.py", line 2870, in do_action
    from_prev_frame=from_prev_frame,
  File "d:\social-leap-estimates-animal-poses\source\sleap\sleap\gui\commands.py", line 2902, in create_new_instance
    mark_complete=mark_complete,
  File "d:\social-leap-estimates-animal-poses\source\sleap\sleap\gui\commands.py", line 2993, in set_visible_nodes
    old_size_width = copy_instance.frame.video.shape[2]
AttributeError: 'NoneType' object has no attribute 'video'

talmolab / sleap