raspivid support

[lumenier1]

Summary

Add one more example with raspivid support instead of gstreamer to provide hardware encoding for Raspberry Pi.

Motivation

Since Raspberry Pi is wildly used in many projects it would be nice to add webrtc implementation with Go for it.

Alternatives

No such brilliant alternatives like Go & webrtc yet.

Additional context

Just to send video from Raspberry Pi to browser with raspivid for hardware encoding.

[alexey-kravtsov]

@lumenier1 Hi! In my project I'm using hardware encoding on Raspberry Pi 3 with GStreamer - just try x264enc. For gstreamer-send example you can change webrtc.DefaultPayloadTypeVP8 to webrtc.DefaultPayloadTypeH264 and webrtc.VP8 to webrtc.H264 in main.go. However there are ~3 sec latency in transmitted video, I've fixed it in https://github.com/pion/example-webrtc-applications/pull/45

[djmaze]

@alexey-kravtsov Does x264enc really support hardware acceleration on the Raspberry Pi? It works for me but leads to a CPU load of about 1.5 to 2 (in my test setup).

I switched to using omxh264enc in my test app. That seems to have decreased the load down to less than 0.5.

[alexey-kravtsov]

@djmaze actually I'm not 100% sure about x264enc. I've tried omxh264enc - it really loads CPU far less, but I have ~10s startup delay with it. Could you please share your pipeline with omxh264enc or test app?

[djmaze]

There is almost no startup delay for me.

The complete video pipeline in the application on the Pi looks like this:

v4l2src ! video/x-raw, width=640, height=480, framerate=15/1 ! queue ! video/x-raw,format=I420 ! omxh264enc control-rate=1 target-bitrate=600000 ! h264parse config-interval=3 ! video/x-h264,stream-format=byte-stream | appsink name=appsink

In order to see the difference, here is the pipeline when running on x64 machines:

v4l2src ! video/x-raw, width=640, height=480, framerate=15/1 ! queue ! video/x-raw,format=I420 ! x264enc speed-preset=ultrafast tune=zerolatency key-int-max=20 ! video/x-h264,stream-format=byte-stream ! appsink name=appsink

EDIT: Maybe the SPS insertion every 3 seconds (using the h264parse element) is the crucial missing part in your pipeline? I read about that somewhere, so I added it, which made my pipeline work at all.

[alexey-kravtsov]

Thank you! I've missed h264parse in my pipeline, so I don't know why it was working at all before :) I've slightly modified your example for my project (added videoconvert before encoder), and now it works fine for me v4l2src ! video/x-raw, width=640, height=480 ! videoconvert ! video/x-raw,format=I420 ! omxh264enc control-rate=1 target-bitrate=600000 ! h264parse config-interval=3 ! video/x-h264,stream-format=byte-stream ! appsink name=appsink

[djmaze]

Great to hear it works for you. Was quite some trial & error and searching around the nets, because no one explains this kind of stuff to you..

Btw, I also used the videoconvert when running on x64 but it at least seemed not necessary when running on a Pi 3 for me.

[lherman-cs]

@lumenier1 now you can use mediadevices to do hardware encoding on a raspberry pi 3, https://github.com/pion/mediadevices/tree/master/examples/webrtc.

Here's the snippet:

package main

import (
    "fmt"

    "github.com/pion/mediadevices"
    "github.com/pion/mediadevices/examples/internal/signal"
    "github.com/pion/mediadevices/pkg/frame"
    "github.com/pion/mediadevices/pkg/prop"
    "github.com/pion/webrtc/v3"

    "github.com/pion/mediadevices/pkg/codec/mmal"
    _ "github.com/pion/mediadevices/pkg/driver/camera"     // This is required to register camera adapter
)

func main() {
    config := webrtc.Configuration{
        ICEServers: []webrtc.ICEServer{
            {
                URLs: []string{"stun:stun.l.google.com:19302"},
            },
        },
    }

    // Wait for the offer to be pasted
    offer := webrtc.SessionDescription{}
    signal.Decode(signal.MustReadStdin(), &offer)

         // mmal package uses Raspberry Pi's hardware encoder.
         // Reference: https://github.com/raspberrypi/userland/tree/master/interface/mmal 
    mmalParams, err := mmal.NewParams()
    if err != nil {
        panic(err)
    }
    mmalParams.BitRate = 500_000 // 500kbps

    codecSelector := mediadevices.NewCodecSelector(
        mediadevices.WithVideoEncoders(&mmalParams),
    )

    mediaEngine := webrtc.MediaEngine{}
    codecSelector.Populate(&mediaEngine)
    api := webrtc.NewAPI(webrtc.WithMediaEngine(&mediaEngine))
    peerConnection, err := api.NewPeerConnection(config)
    if err != nil {
        panic(err)
    }

    // Set the handler for ICE connection state
    // This will notify you when the peer has connected/disconnected
    peerConnection.OnICEConnectionStateChange(func(connectionState webrtc.ICEConnectionState) {
        fmt.Printf("Connection State has changed %s \n", connectionState.String())
    })

    s, err := mediadevices.GetUserMedia(mediadevices.MediaStreamConstraints{
        Video: func(c *mediadevices.MediaTrackConstraints) {
            c.FrameFormat = prop.FrameFormat(frame.FormatI420)
            c.Width = prop.Int(640)
            c.Height = prop.Int(480)
        },
        Codec: codecSelector,
    })
    if err != nil {
        panic(err)
    }

    for _, track := range s.GetTracks() {
        track.OnEnded(func(err error) {
            fmt.Printf("Track (ID: %s) ended with error: %v\n",
                track.ID(), err)
        })

        _, err = peerConnection.AddTransceiverFromTrack(track,
            webrtc.RtpTransceiverInit{
                Direction: webrtc.RTPTransceiverDirectionSendonly,
            },
        )
        if err != nil {
            panic(err)
        }
    }

    // Set the remote SessionDescription
    err = peerConnection.SetRemoteDescription(offer)
    if err != nil {
        panic(err)
    }

    // Create an answer
    answer, err := peerConnection.CreateAnswer(nil)
    if err != nil {
        panic(err)
    }

    // Sets the LocalDescription, and starts our UDP listeners
    err = peerConnection.SetLocalDescription(answer)
    if err != nil {
        panic(err)
    }

    // Output the answer in base64 so we can paste it in browser
    fmt.Println(signal.Encode(answer))
    select {}
}

[G2G2G2G]

@lherman-cs why is it using google's servers?