search

talmolab / sleap

A deep learning framework for multi-animal pose tracking.
https://sleap.ai
Other
427 stars 97 forks source link

Confidence Maps Offset #1611

Open roomrys opened 10 months ago

roomrys commented 10 months ago

Discussed in https://github.com/talmolab/sleap/discussions/1608

Originally posted by **daskandalis** November 25, 2023 I've often observed that the validation confidence maps are highly offset. Is this just a display issue? It's highly directional and the green (MAP?) estimate lies outside the confidence bound. ![image](https://github.com/talmolab/sleap/assets/1887822/1a23bff5-43d0-429b-a051-fc6c703f112b) I'm wondering though if it's related to this printout: >INFO:sleap.nn.training: Input shape: (1088, 1920, 1). The width is 1088 but I think it should be 1080, the resolution of the image (unless the convolution kernels are padded in width but not height?). Incidentally, is it possible to access the variance of the estimated position from the outputs? The measurement error would be useful in filtering. Model parameters: "unet": { "stem_stride": null, "max_stride": 64, "output_stride": 4, "filters": 32, "filters_rate": 1.5, "middle_block": true, "up_interpolate": true, "stacks": 1 } EDIT: or maybe comes from translation in the augmentation step?

This looks to be a visualization issue. The red predicted points should be centered on the peak value of the green 2D Gaussian (which depicts confidence values at pixel locations). While it seems that the red predicted and green ground truth points are correctly overlaid on the image, the confidence maps (from which the predicted points are derived from) are not being correctly displayed. We must be missing some offset for the visualization of the confidence maps.

Note: it seems you have a green point labeled as visible that should be invisible (out above the animal).

Thanks for reporting this! - I'll convert this discussion to an issue. Liezl

talmo commented 8 months ago

Right, I think we add padding to the image to avoid truncation when downsampling --> upsampling, but not to the confidence maps, which might result in weird aspect ratio issues?

Relevant plotting code:

https://github.com/talmolab/sleap/blob/16241e07fb64520c8ff22d8cf180c6bf811e698a/sleap/nn/viz.py#L81-L96

Called from here during training (for single instance -- other model types plot a bit differently):

https://github.com/talmolab/sleap/blob/16241e07fb64520c8ff22d8cf180c6bf811e698a/sleap/nn/training.py#L1068-L1122