cvisionai / tator

Video analytics web platform
https://tator.io
GNU Affero General Public License v3.0
98 stars 32 forks source link

Add support for displaying stereo images in multiview #1669

Open danellecline opened 1 month ago

danellecline commented 1 month ago

Viewing stereo images would be very helpful for comparing model output side-by-side for robotics research applications.

Do you have an estimate of the level of effort this would require? I'm willing to consider doing this work but must balance it with several other projects. A rough estimate in days/weeks is helpful.

jrtcppv commented 1 month ago

Hello, it would be really great if you added this feature. I would estimate around 120-160 hours of work to implement, using the image annotation view and multi-stream video annotation view as a guide. There may be some edits required to tator-js, which contains some of the lower level functionality. Specifically you may need to modify the image canvas element, as some modifications were needed for the video canvas to support grid views etc. in the multi-stream video annotation view.

In addition to the front end, there may be some modifications to the REST service required to represent multi-view images in the media multi dtype. This will likely only include safeguards, such as preventing combination of video and image. We believe the existing schema should have everything you need so do not expect any database migrations (changes to models.py). There may be a need to modify the tator-py utility for creating multi-view images as it currently expects videos only. Finally we would want to implement unit tests for the multi-image view.

I think we can take care of most of the items laid out in the second paragraph above; none of those are required for an initial capability. You could start by just making a Media object manually with the Django shell, set its component IDs to two images, and then start building the multi image view around that. If you can get there, we can help with the rest (REST safeguards, tator-py utility, and unit tests). And feel free to ask any questions here, or if you'd like we can set up a shared Slack channel.

marktaipan-cvisionai commented 1 month ago

@danellecline - the [dev/1669-multi-images](https://github.com/cvisionai/tator/tree/dev/1669-multi-images) branch has a skeleton of the proposed changes. It includes a lot of the REST service multi dtype changes (the inclusion of the multi-image dtype) as well as an annotation-multi-image.js web element that mimics the annotation-multi.js video version with all the video-specific stuff removed. I'm hoping the branch helps with jumping into this feature development. Let us know if there are any questions, and thanks for taking a look!

danellecline commented 1 month ago

Terrific. I will inspect this more closely and setup a dev environment next week. Many thanks @marktaipan-cvisionai.