[Discussion] General performance when sending image frames

[binarykitchen]

Hello guys

I'm looking for a solution within my own app to send image frames to my server as fast as possible. At the moment I use socket.io for that and have a discussion here https://github.com/LearnBoost/socket.io/issues/1175

I see, you're using a different approach here, you're piping image frames over UDP. Is this correct? What's the maximum FPS and do you think the videos are smooth? Do you think this is better than using Socket.IO?

Thanks for any clues, advices, etc.

Cheers Michael

[karlwestin]

Hey, thanks for writing in!

I understand your question as "could the tech in node-dronestream be used to send images"

I think the issue filed on socket.io was slightly out of the scope of that project. This problem is way more complex than the one they're trying to solve. Let's look at it step by step how node-dronestream works:

1. We're getting raw, compressed h.264 data from the camera. This is transfered to the relay server (the http.createServer in examples/createServer) over UDP
2. Now, a proprietary header from the drone is parsed out of the video stream, and the raw h.264 data is sent over websockets (tcp).
3. The data is decompressed on the client. This is the key to node-dronestreams performance

Unfortunately, i don't really have any benchmarks on data sent/frame or FPS. I guess what i wanted to say is that the problem is bigger than just socket.io, it's also about data formats, where to do compression and decompression, and to make sure that those aren't adding lag. What's your image source right now? Do you read png's with file headers, etc? Are you able to get raw h.264 data from your data source?

Here's a link to the broadway parser project: https://github.com/mbebenita/Broadway

Hope this help you see the problem in a different light :)

[binarykitchen]

Hallo Karl

Danke danke für deine Zeilen, das hilft mir sehr. Umm, I better switch from German to English so that anyone can follow.

I see, the drone has a real h.264 cam and is sending raw data via UDP. Not surprised it's much faster and better.

Let me explain you my case: I want to emit image frames from a normal webcam to the server. With getUserMedia I can extract them and send easily via socket.io. Works well. And on the server I encode the frames into mp4 and webm videos for all browsers with avconv.

My problem is performance. Sending all these image frames via socket.io at fps = 15 becomes slow. My data source is a normal Logitech webcam without the capability to send raw h.264 data. And anyways, I'm bound to the limitations of getUserMedia where you can extract images but not have the real video stream (WebRTC is a whole different story because it's Peer 2 Peer, not Peer 2 Server).

I already have checked https://github.com/mbebenita/Broadway but it's a decoder only. All I need is a video encoder on the client side! That would be great. Then I wouldn't have to emit all the image frames to the server ...

Should I try porting avconv to JavaScript with emscripten? Or is this too crazy? Grins ...

Viele Grüsse aus Auckland, Michael

[binarykitchen]

PS: I also checked Whammy (https://github.com/antimatter15/whammy) but that's out of question because only one browser supports WebP + WebM.

[bkw]

I don't understand, why you want to do this on the client side. This sounds more like a case for ffserver for me: Define the webcam as a feed in ffserver and setup multiple streams for the clients and an archive on the server. Should be pretty straightforward.

[binarykitchen]

@bkw, maybe I didn't tell you that I am Deaf. I am trying to build something where we Deaf people can record a message in Sign Language and send it away as if it were a video email. Hence I am not interested in streaming solutions.

And anyways, with ffserver you'd need Flash on the client side and that's a no-go for me. I am searching for a pure HTML5 solution without Flash.

[bkw]

@binarykitchen Sorry, I had now idea about the actual use case. Btw, there's no need to use flash with ffserver, it supports all kinds of outputs, like mjpeg, avi, even webm with newer releases. But still, this probably does not help with your application. If I were to to it, I'd try to direct the stream to a server using the technologies used by webrtc (srtp websockets), and encode and store it on the server. Users would just exchange a link to the encoded video. Sorry, this is just an untested idea, I haven't tried this.

[binarykitchen]

Right, I want to direct the webcam output with plain HTML5 + JavaScript to the server.

As I mentioned above, WebRTC is only a Peer 2 Peer solution. You cannot redirect that video traffic to a server. Maybe in the future, but not yet. It's against the SIP protocol to store the video on the server. But for my application I want to do that. Any suggestions are very welcome.

[bkw]

I fail to see how a proper implementation on a server that acts just like a client would be distinguishable from a 'real' client. I'm not saying it is going to be easy, but I can't find anything in the spec that says "a peer has to be a humanly operated browser with a graphical user interface and a physical camera" or anything like that. There's also WebRTC support for Asterisk servers. Anyway. I'm afraid I cannot help you here.

[binarykitchen]

No problem. I already did lots of research and doubt any WebRTC or SIP based solution is able to store a video on the server.

What servers mostly do for WebRTC is publishing and removing callers or listeners (subscription, presence status etc) but nothing else like encoding, decoding or saving audio nor videos.

All I need is a video encoder in pure JavaScript!

[binarykitchen]

Any comments, ideas, suggestions? :)

[karlwestin]

Closing this issue now, hope you found a solution for this :)

[binarykitchen]

Not yet, this ticket for porting avconv/ffmpeg to javascript will help me https://github.com/kripken/emscripten/issues/1042