Specular reflective surfaces

[bhack]

The performance on the well known ambiguous specular reflective surfaces are really not good also on light reflective surfaces. Often it is tracked the reflected content and the occlusion logic seems to be off. Do you plan to improve this with synthetic data like pointodyssey?

[nikitakaraevv]

Hi @bhack, yes, we also plan to release a model that will be trained on all available datasets.

[bhack]

Probably a light segmentation component it could help in the model to understand what we are tracking with the grid allocation in cluster tracks.

[nnop]

Is the released cotracker2 checkpoint trained on all available datasets? @nikitakaraevv

[nikitakaraevv]

Hi @nnop, we found that performance does not always improve when training on different datasets. This is because the model already exhibits strong synthetic-to-real generalization, and additional synthetic datasets do not necessarily make it better. We're trying to figure out a way to improve the model with more data.

[bhack]

@nikitakaraevv probably you could explore any recent active learning approach to select new synthetic or real useful data samples.