search

webrtcHacks / tfObjWebrtc

Tensorflow Object Detection API Web Service wrapper that works on any <video> tag and WebRTC streams
MIT License
179 stars 52 forks source link

Move away from POST and use sockets ? #1

Open wardhanster opened 6 years ago

wardhanster commented 6 years ago

I was wondering if this was possible and if this is a possible route to go in order to eliminate the poor frame rate coming in because of the polling method employed ? Also is I was wondering if there was any specific reason you used POST method over sockets ?

Any Insights would be amazing. Thanks for creating this !

chadwallacehart commented 6 years ago

I agree websockets is mostly better for this application, but I do not think that will have a major impact on the response time.

I did not do websockets originally because the framerate bottleneck in my case is actually running the inference on the image. On my MacBook Pro this takes 700 to 1300 ms per image with a 1280 px uploadWidth. The several ms saved by moving to websockets was not worth the added troubleshooting overhead - it is very easy to inspect the POST message & response with the built in debugging tools in all browsers.

The easiest way to increase the responsiveness is to drop the uploadWidth parameter to 320. This will decrease the image recognition accuracy, but that may be ok depending on your application. I quick test I just did dropped the total upload, inference, and response time to 120-130 ms. For added cost, you can also get a GPU or something like Intel's Movidius to speed processing.

I am working on applying this codebase to a home robotics project that uses websockets. It is likely I will provide an update that uses websockets in the coming weeks.

chadwallacehart commented 5 years ago

I'm not sure when/if I will add WebSockets for this project, but you can see an example WebSocket implementation I did on another repo here: https://github.com/chadwallacehart/sense_hat_server

In that project, I used flask_socketio. This line here shows how you could respond to an incoming image from the web client: https://github.com/chadwallacehart/sense_hat_server/blob/3c6e4dfc849772643112d6f21c688b80c9c2cce1/server.py#L167