Stream h264 from webcam with hardware acceleration

KapJI commented 2 years ago

I want to stream video from my webcam to internet. This sounds like a basic task but I haven't found a good solution yet.

Requirements

Webcam which outputs MJPEG stream.
Raspberry Pi 4.
Stream 1080p in 30 fps.
Low bandwidth, around 8 mbit/s for such stream. This basically means using H264.
Low CPU usage. This means both MJPEG decoding and H264 encoding should be hardware accelerated.
Low latency, below 500ms. This means using WebRTC. My experiments show that 200ms is achievable.
Multiple clients watching the same stream simultaneously. WebRTC should help with that as well.
Modern OS, 64-bit Bullseye. OpenMAX IL is not available there, basically V4L2 M2M should be used.
Some authentication on top of it to make it available from the internet. I think Janus should be able to handle that.

What I've tried

I tried using aiortc for WebRTC but I found that Janus is more mature, reliable and faster because it's written in native code. So I want to use Janus.

I tried using ffmpeg for decoding and encoding. It works fine but only if client joined before the stream has started. And the second client can't decode stream.

Example command (ffmpeg doesn't have mjpeg_v4l2m2m decoder out of the box, but it can be enabled with this patch):

ffmpeg -f v4l2 -input_format mjpeg -framerate 30 -video_size 1920x1080 -c:v mjpeg_v4l2m2m -i /dev/video0 -codec:v h264_v4l2m2m -num_capture_buffers 32 -pix_fmt yuv420p -b:v 8M -f rtp -f rtp rtp://127.0.0.1:8004

It works if libx264 encoder is used but it's not hardware accelerated, so CPU usage is very high. Not sure what is missing for M2M encoder. Maybe it inserts key frames differently.

Questions

Can ustreamer help me with that? PiKVM seems to be doing something very similar to what I need.

I noticed that you have some M2M support on m2m branch but it doesn't have support for MJPEG decoder. How difficult will it be to add? What else is missing there and when are you going to merge it?

Also do you have any documentation about how to use ustreamer janus plugin?

I'd appreciate any help.

PS: I can speak russian btw, if you have some time to chat.

mdevaev commented 2 years ago

Hello.

I will soon finish native H264 hardware support for the M2M. MJPEG is also almost finished, but requires some assistance from Raspberry. We are solving this issue.
Integration with Janus works great in PiKVM, but I don't have time to write documentation for it right now. It will be done after M2M.

KapJI commented 2 years ago

Thanks, looking forward to it! I'm happy to help you with testing if you need some.

noahwilliamsson commented 2 years ago

Hi, what's the status on the M2M and MJPEG support in ustreamer? I noticed a few fixes went into a release of the Raspberry Pi 5.15 kernel (installable via sudo rpi-update) last week.

mdevaev commented 2 years ago

You can try m2m branch from git

mdevaev commented 2 years ago

All done. See here: https://github.com/pikvm/ustreamer/blob/master/docs/h264.md

thbl commented 2 years ago

All done. See here: https://github.com/pikvm/ustreamer/blob/master/docs/h264.md

@mdevaev

Is there any doc about what m2m is ? Also if you use a webcam with MJPEG and rpi then follow the Janus guide, will it use the rpi GPU for encoding the MJPEG to h264 ?

mdevaev commented 2 years ago

M2M is an gpu encoder on raspberry pi

will it use the rpi GPU for encoding the MJPEG to h264?

Yes, but it will be slow, because it's CPU-based reencoding: mjpeg->unjpeg(cpu)->raw->h264(m2m)

thbl commented 2 years ago

M2M is an gpu encoder on raspberry pi

will it use the rpi GPU for encoding the MJPEG to h264?

Yes, but it will be slow, because it's CPU-based reencoding: mjpeg->unjpeg(cpu)->raw->h264(m2m)

Would it be better with a Nvidia Jetson, and is it supported ?

mdevaev commented 2 years ago

I don't know, sorry, I never used Jetson.

pikvm / ustreamer