search

talmolab / sleap

A deep learning framework for multi-animal pose tracking.
https://sleap.ai
Other
432 stars 96 forks source link

Error in OKS score calculation #1346

Closed sean-afshar closed 1 year ago

sean-afshar commented 1 year ago

Bug description

You multiply by an additional factor of 2 in the sleap.nn.evals compute_oks function on line 206. You can fix this by either deleting the 2 on line 206 https://github.com/talmolab/sleap/blob/c94c75517b34d6c14f4e3bd80695daab372fdf0a/sleap/nn/evals.py#L206 or getting rid of the factor of 2 in both the spread and scale factors and putting the factor of 2 in the normalization_factor variable.

Expected behaviour

This underestimates what the actual oks scores should be.

Actual behaviour

See expected behavior.

Your personal set up

Environment packages ``` # paste output of `pip freeze` or `conda list` here ```
Logs ``` # paste relevant logs here, if any ```

Screenshots

How to reproduce

  1. Go to '...'
  2. Click on '....'
  3. Scroll down to '....'
  4. See error
talmo commented 1 year ago

TODO (@shrivaths16): Even though we already merged, can we double check against this reference code?

https://github.com/cocodataset/cocoapi/blob/8c9bcc3cf640524c4c20a9c40e89cb6a2f2fa0e9/PythonAPI/pycocotools/cocoeval.py#L192-L233

If our new code is exactly equivalent (algorithmically AND mathematically) to this one, then close this issue. A little gist/Colab running both functions standalone would be preferred so we can add tests.

roomrys commented 1 year ago

This should be addressed in SLEAP 1.3.2.