search

gustavz / realtime_object_detection

Plug and Play Real-Time Object Detection App with Tensorflow and OpenCV
MIT License
281 stars 148 forks source link

Real-time segmentation #6

Open Kowasaki opened 6 years ago

Kowasaki commented 6 years ago

Hey @GustavZ, as mentioned is the previous threads I am looking into efficient segmentation networks that can be run on the TX2. Would you mind looping me in to the conversation regarding the mask implementation? Me and rest of the team at my university are all very interested in this subject and would like to figure out the best approach for doing real-time segmentation.

gustavz commented 6 years ago

@Kowasaki I am currently working my way through the following projects / papers to come up with a ssd modification that allows segmentation in parallel to bounding box detection. Please feel free to add code proposals or other sources if you find some. I will keep you up to date as i proceed.

Kowasaki commented 6 years ago

Thanks for the links--I will be looking through them as well!

I'm also wondering if you are open to plain old segmentation models? I've found a model that is meant to achieve 7fps on the TX1: https://github.com/Eromera/erfnet_pytorch

I'll need to test it out after reading up on PyTorch but assuming the GPU RAM usage isn't overwhelming I'm considering running both the detector and the segmentation model at once.

gustavz commented 6 years ago

yeah you can do this ofcourse! its a valid workaround, but i want to achieve a network that can do both simultaniously :)

Kowasaki commented 6 years ago

Gotcha--in this case you might want to take at look at the differences between the Box predictors used for the SSD and the Mask-RCNN networks here: https://github.com/tensorflow/models/blob/master/research/object_detection/core/box_predictor.py.

I suspect you can add segmentation for the SSD boxes to use some sort of encoder-decoder network like you mentioned before. The FCN on the Mask-RCNN stuff is way too heavy for real-time. I'll have to try this later as well but unfortunately time constraints dictate I stick with what works for now!

gustavz commented 6 years ago

@Kowasaki any news about this topic from your side?

Kowasaki commented 6 years ago

@GustavZ Unfortunately no. The SSDlite-Deeplab model described in the mobilenet v2 paper is the best bet I think, but I haven't had time to look over how to implement that whole thing.

Have you found anything that works well with SSD? The SSD-Lite part described in the paper may be worth a shot.