WebRTC recording with OpenCV integration?

[fohara]

Is there an example of using OpenCV to process frames from a live WebRTC video stream? As an example, say you wanted to perform some basic feature detection (e.g. canny edge detection) on frames as they streamed in from a WebRTC peer; how would you accomplish this with libsourcey?

I noticed the WebRTC Native Video Recorder sample and this bit of code:

        Signaler app(options);

        Idler rtc(app.loop,
                  [](void* arg) {
                      // TraceL << "Running WebRTC loop" << endl;
                      auto thread = reinterpret_cast<rtc::Thread*>(arg);
                      thread->ProcessMessages(10);
                  },
                  rtc::Thread::Current());

It's not clear to me though how I would integrate OpenCV to process frames as they were received. Is there something like a "frame received" event/object? Recording the stream is also desirable, so it seems like this example is a good place to start out, but I would also like to access the frames as images before they are recorded. Thanks.

[auscaster]

The correct way to handle the incoming video would be to modify the StreamRecorder, namely the void StreamRecorder::OnFrame(const cricket::VideoFrame& yuvframe) method and pass the video to OpenCV instead of handling with FFmpeg.

[auscaster]

Were you able to find a solution? Please close this if fixed.

[fohara]

@auscaster yes that makes sense, thanks!

[fohara]

@auscaster I'm revisiting this; do you have an example of passing the video to OpenCV in the OnFrame method you mention above? I'm guessing there's some conversion of a yuvframe to a Mat, etc, and wanted to check if there was an example of this in the codebase. Thanks!

[lutsykigor]

I am interested in this as well, it would be nice to have some example.

[fohara]

I think an example of this would make a nice contribution to the repo, so I've posted a bounty to help get this done.

[auscaster]

Hey @fohara nice work on the bounty. So to be clear you're just looking for a bi-directional webrtc cricket::VideoFrame to cv::Mat converter? Not looking to use OpenCV capture sources or anything like that?

[lutsykigor]

Please check this pull request for sample app: https://github.com/sourcey/libsourcey/pull/206

BTW: Didn’t detect that there is a 100 bucks bounty for this :) @fohara take it for free, just a Christmas gift ;)

[fohara]

@auscaster using an OpenCV capture source sounds reasonable, but I'm still getting familiar with these APIs. Ultimately, whatever fits the use case best, which is to:

• Analyze a live video stream frame-by-frame with OpenCV (which I assume entails a cv::Mat conversion)
• Record the video stream, in addition to analyzing it
• Optionally stream a modified video back to client, i.e. video that has bounding box annotations, or has been processed with a canny edge detector

An example that accomplishes the above points, along with a README and self-contained project structure similar to the other WebRTC samples, is what I'm looking for.

Let me know if that makes sense, thanks.

[fohara]

@lutsykigor nice! It looks like you're delivering on some of the points I mentioned in my previous comment. I'll take a look this evening (PST) and see if I can get it running. Thanks!

[lutsykigor]

@fohara the easiest way for you to run a sample client is to use Firefox(it should work out of the box), I have just added extra comments to my pull request a few seconds ago regarding running it on a Chrome, please check updated Readme.md file. I do not have access to my work PC right now, so I have just edited these files online :-) Chrome does not allow you to access media capabilities on HTTP host, you need to be on HTTPS. You will need to make simple tweaks to change sample Node.js web server to serve requests on HTTPS.

I hope it helps. Feel free to ask any questions if you have problems running this sample.

According to your points in previous post:

Analyze a live video stream frame-by-frame with OpenCV (which I assume entails a cv::Mat conversion)

You do not need to run heavy neural network computations on each frame, as the situation won't change every frame. Please note that in my sample I am using single shot detector plus mobile nets which is very light weight, if you plan to use some heavy neural networks you must have few CUDAs to process analytics realtime fore every frame (I am just guessing here).

Record the video stream, in addition to analyzing it

You can use webrtcrecorder sample for this

Optionally stream a modified video back to client, i.e. video that has bounding box annotations, or has been processed with a canny edge detector

I do not think that you want that :-) To draw some analytics data on server side, you need to decode every single frame, run analytics on it (you may skip analyzing some frames actually), draw analytics output on every frame and encode it before sending back to clients. Thus you will have huge delays in video delivery and a substantial server resources exhaust. Check my comments in the pull request description. You can just decode frame, run neural network computations only on some frames, and send analytics data back to web clients by a websocket connection, then you can use JavaScript do draw it on a client side.

Regards

[auscaster]

@fohara I've merged the OpenCV sample created by @lutsykigor into master. I look forward to hearing how it goes for you!

[fohara]

@lutsykigor @auscaster ok I was able to get libsourcey to compile with the opencvanalyzer sample app, but I got a few errors when I attempted to run it. My eventual goal is to run this within a Docker container, so I started putting a Dockerfile together to build an image with all the dependencies and configurations required to run opencvanalyzer.

In the meantime, this Dockerfile should be useful for debugging the errors I encountered since it will provide a consistent build environment. With that in mind, here are my repro steps:

• Download the work-in-progress Dockerfile I have posted in this gist
• Build the Dockerfile with (this will take a while): docker build -t libsourcey-webrtc-opencvanalyzer . | tee docker_build.log
• Start an interactive bash session in the Docker container with: docker run -i -t libsourcey-webrtc-opencvanalyzer /bin/bash
• Finally, attempt to run the sample help with: ./webrtc/samples/opencvanalyzer/opencvanalyzer -h

That last command prints out a bunch of config errors:

[000:000] [17] (webrtcvoiceengine.cc:974): webrtc:   failed to connect context, error=-1
[000:000] [17] (audio_device_pulse_linux.cc:171): failed to initialize PulseAudio
[000:003] [17] (webrtcvoiceengine.cc:974): webrtc: GetDevicesInfo - Could not find device name or numbers
ALSA lib control.c:954:(snd_ctl_open_noupdate) Invalid CTL
[000:005] [17] (webrtcvoiceengine.cc:974): webrtc:      snd_mixer_attach(_outputMixerHandle, ) error: No such file or directory
[000:006] [17] (webrtcvoiceengine.cc:974): webrtc: GetDevicesInfo - Could not find device name or numbers
ALSA lib control.c:954:(snd_ctl_open_noupdate) Invalid CTL
[000:006] [17] (webrtcvoiceengine.cc:974): webrtc:      snd_mixer_attach(_inputMixerHandle, ) error: No such file or directory
[000:006] [17] (webrtcvoiceengine.cc:974): webrtc: GetDevicesInfo - Could not find device name or numbers
ALSA lib control.c:954:(snd_ctl_open_noupdate) Invalid CTL
[000:006] [17] (webrtcvoiceengine.cc:974): webrtc:      snd_mixer_attach(_outputMixerHandle, ) error: No such file or directory
[000:006] [17] (webrtcvoiceengine.cc:974): webrtc: GetDevicesInfo - Could not find device name or numbers
ALSA lib pcm.c:2266:(snd_pcm_open_noupdate) Unknown PCM
[000:006] [17] (webrtcvoiceengine.cc:974): webrtc:      unable to open playback device: No such file or directory (-2)
[000:006] [17] (webrtcvoiceengine.cc:974): webrtc: GetDevicesInfo - Could not find device name or numbers
ALSA lib control.c:954:(snd_ctl_open_noupdate) Invalid CTL
[000:006] [17] (webrtcvoiceengine.cc:974): webrtc:      snd_mixer_attach(_inputMixerHandle, ) error: No such file or directory
[000:007] [17] (webrtcvoiceengine.cc:974): webrtc: GetDevicesInfo - Could not find device name or numbers
ALSA lib pcm.c:2266:(snd_pcm_open_noupdate) Unknown PCM
[000:007] [17] (webrtcvoiceengine.cc:974): webrtc:     unable to open record device: No such file or directory
[000:007] [17] (audio_device_generic.cc:51): BuiltInAECIsAvailable: Not supported on this platform
[000:007] [17] (audio_device_generic.cc:61): BuiltInAGCIsAvailable: Not supported on this platform
[000:007] [17] (audio_device_generic.cc:71): BuiltInNSIsAvailable: Not supported on this platform
[000:007] [17] (adm_helpers.cc:47): Unable to set recording channel to kChannelBoth.
[000:007] [17] (webrtcvoiceengine.cc:974): webrtc: GetDevicesInfo - Could not find device name or numbers
ALSA lib control.c:954:(snd_ctl_open_noupdate) Invalid CTL
[000:007] [17] (webrtcvoiceengine.cc:974): webrtc:      snd_mixer_attach(_inputMixerHandle, ) error: No such file or directory
[000:007] [17] (adm_helpers.cc:56): Unable to access microphone.
[000:007] [17] (webrtcvoiceengine.cc:974): webrtc: GetDevicesInfo - Could not find device name or numbers
ALSA lib control.c:954:(snd_ctl_open_noupdate) Invalid CTL
[000:007] [17] (webrtcvoiceengine.cc:974): webrtc:      snd_mixer_attach(_inputMixerHandle, ) error: No such file or directory
[000:008] [17] (webrtcvoiceengine.cc:974): webrtc: GetDevicesInfo - Could not find device name or numbers
ALSA lib pcm.c:2266:(snd_pcm_open_noupdate) Unknown PCM
[000:008] [17] (webrtcvoiceengine.cc:974): webrtc:     unable to open record device: No such file or directory
[000:008] [17] (webrtcvoiceengine.cc:974): webrtc: GetDevicesInfo - Could not find device name or numbers
ALSA lib control.c:954:(snd_ctl_open_noupdate) Invalid CTL
[000:008] [17] (webrtcvoiceengine.cc:974): webrtc:      snd_mixer_attach(_outputMixerHandle, ) error: No such file or directory
[000:008] [17] (adm_helpers.cc:100): Unable to access speaker.
[000:008] [17] (webrtcvoiceengine.cc:974): webrtc: GetDevicesInfo - Could not find device name or numbers
ALSA lib control.c:954:(snd_ctl_open_noupdate) Invalid CTL
[000:008] [17] (webrtcvoiceengine.cc:974): webrtc:      snd_mixer_attach(_outputMixerHandle, ) error: No such file or directory
[000:008] [17] (webrtcvoiceengine.cc:974): webrtc: GetDevicesInfo - Could not find device name or numbers
ALSA lib pcm.c:2266:(snd_pcm_open_noupdate) Unknown PCM
[000:008] [17] (webrtcvoiceengine.cc:974): webrtc:      unable to open playback device: No such file or directory (-2)
terminate called after throwing an instance of 'std::runtime_error'
  what():  Cannot connect to Symple server. Did you start the demo app and the Symple server is running on port 4500?
Aborted

These errors indicate that I've likely missed some configuration steps, which should be evident by what's in the Dockerfile. Once these errors are resolved, I can proceed to installing the node.js client app, but I wanted to make sure this part was solid first. Thanks!

[lutsykigor]

@fohara great job on a containerization, I am just scratching a surface here but this is what I found:

1. You are running an opencvanalyzer binarry inside a container only, but you need to run singlaling(symple server - last error highlights that) and web server as well, please refer to sample readme file, these both servers should be started with single Node.js app.
2. All these exceptions from WebRTC stack, probably, are thrown because of a lack of actual playback and capture devices inside a container. I do not know much about running Google’s WebRTC native stack inside a container, maybe this will help: https://github.com/DamonOehlman/webrtc-testbot
3. Seems that you are installing develop version of OpenCV, from the master branch. I do not recommend doing that, try to use tagged 3.3 version instead.

In general, I suggest you to run sample app without containeriazation. It can help you to troubleshoot future problems related to inserting that app to a container.

I hope that my answers will help you, but probably there are still a lot of other issues to resolve.