MJPEG Stream only has 5 FPS, H.264 has no problem with 30 FPS

benni-tec commented 2 years ago

So I'm trying to build an application to stream the Raspberry Pi Camera from my custom Go application. Your package seems to do exactly what I need. After a lot of testing I'm continually running into the same issue.

I am running a Raspberry Pi 3B+ with Raspbian Bullseye 32bit with kernel 5.10.103-v7+

The device-struct seems to only be able to return 5 fps in MJPEG mode, however H.264 works at 30 fps.

Bitrate does not seem to be the problem, I have 256mb assigned to the gpu and changing the bitrate does not make a diffrence. From what I've read these should totaly be sufficient for 720p.

The v4l2-ctl utility does show that the camera can do 30fps in MJPEG mode

v4l2-ctl --set-fmt-video=width=1280,height=720,pixelformat=MJPG
v4l2-ctl --stream-mmap=2 --stream-count=500

KKKKKKKKKKKKKKKKKKKKKKKKKKKKKK 30.02 fps

However without the setup call I'm only getting <<<<<< 5.52 fps, so it seems to me it could be a setup issue, however as these are not Keyframes this would also suggest it's not MJPEG format, which your device-struct seems to return. So maybe parts of the setup fail silently?

The delay does NOT occur in my processing, I'm using the following code to pump the stream from the camera into the stream-struct (saljam/mjpeg), this does not produce latency greater 1ms almost always with spikes up to 8ms, which still wouldn't be an issue. Rather the channel only returns a value every ~180ms which equals ~5fps

func streamPump5() error {
cam, err := v4l2.Open("/dev/video0")
if err != nil {
    return err
}

println("opened cam")

if err := cam.SetPixelFormat(1280, 720, v4l2.V4L2_PIX_FMT_JPEG); err != nil {
    return err
}
if err := cam.SetBitrate(1500000); err != nil {
    return err
}

println("setup cam")

if err := cam.Start(); err != nil {
    println(err)
    return err
}

go func() {
    defer func() {
        cam.Stop()
        running5 = 0
    }()

    lastBefore := time.Now()
    lastAfter := time.Now()
    for frame := range cam.C {
        before := time.Now()
        if running5 == 0 {
            return
        }

        basestream.UpdateJPEG(frame.Data)
        frame.Release()

        after := time.Now()

        deltaFrame := after.Sub(lastBefore)
        fps := 1 / deltaFrame.Seconds()
        deltaProcessing := after.Sub(before)
        deltaBetween := before.Sub(lastAfter)

        fmt.Printf(
            "Frame-Delta: %dms Processing-Delta: %dms Between-Delta: %dms FPS: %s                      \r",
            int(deltaFrame.Milliseconds()),
            int(deltaProcessing.Milliseconds()),
            int(deltaBetween.Milliseconds()),
            strconv.FormatFloat(fps, 'f', 2, 64),
        )

        lastBefore = before
        lastAfter = after
    }
}()

return nil
}

I have been digging through your code but could not find the issue up until now, any help would be greatly appreciated. Have a nice weekend, Benjamin

thinkski commented 2 years ago

Were you able to figure this out? Profiling the process may provide some insight.

benni-tec commented 2 years ago

Yes profiling might help and reveal the issue. However I was already time constraint since uni started back up, so I switched to blackjack/webcam wich works.

where you able to replicate the issue? Maybe comparing the code to webcam might give additional insight

thinkski / go-v4l2

MJPEG Stream only has 5 FPS, H.264 has no problem with 30 FPS #1