search

pikvm / ustreamer

µStreamer - Lightweight and fast MJPEG-HTTP streamer
https://pikvm.org
GNU General Public License v3.0
1.67k stars 226 forks source link

Consider supporting h264_v4l2m2m instead of OpenMax IL #83

Closed LalitMaganti closed 2 years ago

LalitMaganti commented 3 years ago

I wanted to test out ustreamer on Ubuntu 20.10 64-bit including with OMX enabled. I quickly ran into the issue that OpenMax IL is considered deprecated and will never work on 64 bit OSes. h264_v4l2m2m is considered the replacement [1].

Considering that 64-bit will end up being the future on the Pi (both 4 and 5+), focusing effort on using h264_v4l2m2m instead may be worthwhile - especially as H264 support is still in beta stages and fixing this now will be easier than when the feature is stabilised.

[1] https://github.com/raspberrypi/Raspberry-Pi-OS-64bit/issues/98

mdevaev commented 3 years ago

I don't think "instead" is a good solution. Can be implemented additionally as another encoding option. Now I don't have time to do this, because first of all ustreamer is necessary to solve local problems of the pikvm. This will be implemented in the future.

LalitMaganti commented 3 years ago

Hm if you think it could be implemented on the side as another option, seems like a self contained enough project. No promises but may take a look into this myself :)

mdevaev commented 3 years ago

The components are loosely connected to each other and have a clear interface. jpeg encoders are interchangeable, h264 can be implemented in the same way. I don't want you to misunderstand me, I don't think your proposal is a bad idea, it's just that now I am the only developer of it and have to prioritize the start of sales of pikvm.

LalitMaganti commented 3 years ago

Yeah no worries I totally understand.

Video encoding has long been a pet interest of mine so that's why I might I look into this for my own curiosity.

LalitMaganti commented 3 years ago

Wanted to provide an update on this: managed to get a small local program which encodes by directly interfacing with the v4l2 API using ioctls. Even with my very inefficient implementation, performance seems to be reasonable.

Next step is to integrate this test program into ustreamer.

LalitMaganti commented 3 years ago

Pushed my change to a fork of this repo: https://github.com/LalitMaganti/ustreamer/commit/a14822a03da74c429928060d99a06e7a0a39d030

I implemented as a rewrite for simplicity sake of testing. I can max out my encoding with my USB adapter at 1080p@30fps using ustreamer + ustreamer-dump + ffmpeg for encapsulating into mp4. Would be curious to see if with the CSI-2 interface if we can push up to beyond the limits of the OpenMax implementation.

mdevaev commented 3 years ago

I haven't looked at it yet, but I'll look into it. I'm catastrophically busy right now. With a quick glance, I see a lot of changes, I need to learn how it works.

LalitMaganti commented 3 years ago

Since I found this whole experience very interesting, I wrote a blog post about it. You might find it useful when looking through the changes I made.

Link: here

mdevaev commented 3 years ago

Wow, thank you! This will definitely make my life easier when I want to implement this in ustreamer.

mdevaev commented 3 years ago

Some important info about this: https://github.com/raspberrypi/linux/issues/3974

mdevaev commented 2 years ago

https://github.com/raspberrypi/libcamera-apps/blob/main/encoder/h264_encoder.cpp

mdevaev commented 2 years ago

Since MMAL and OMX is deprecated now, I'll move this encoders to V4L2 API. See https://github.com/pikvm/ustreamer/tree/m2m branch. Now H.264 with DMA is implemented. JPEG - soon.

mdevaev commented 2 years ago

Closing it since H.264 is ready.