HLS on arm64

[chudsaviet]

HLS does not work on arm64 build. This is the bug to track these issues.

[chudsaviet]

On 0.18.0 RC1 arm64 version, it was found that HLS stream is not working.

I have investigated these issues.

So far we were relying on RaspberryPi hardware h264 encoding capabilities. Unfortunately, now there is no hardware encoding library /opt/vc/lib/libopenmaxil.so for arm64. We can track next repositories on arm64 hardware encoding implementation: 1) https://github.com/raspberrypi/firmware 2) https://github.com/raspberrypi/userland

Basically, libopenmaxil.so is the part of userland repo. It's built and the binaries are put to the firmware repo. So far, it does not build for arm64.

Also, we will have to use FFmpeg built with --enable-omx-rpi flag, which is on by default for armhf and off for arm64.

[chudsaviet]

As a solution, we can use software h264 encoding. Actually, Raspberry Pi 4 is powerful enough to do this without interfering with OctoPrint wannabe-realtime core.

Good news is that even with veryfast preset, libx264 is producing much lower bitrate for given visual quality.

I think I will work on webcamd analog script written in Python3, which will be flexible enough to choose between software/hardware encoder.

[chudsaviet]

For now, users can manually set encoder to h264 instead of h264_omx, and set --preset:v veryfast in ffmpeg-hls Systemd unit.

[DrDrache]

not to add unuseful information, but from my understanding OMX (openmax) will "never" come to 64bit userland. it would be a nightmare to map the VPU. the current (supported?) h264 hardware-assisted method is h264_v4l2m2m via ffmpeg >=4.3

[chudsaviet]

Thanks, @DrDrache , it’s actually useful. Yeah, I was trying to use V4L2M2M, but it had not supported all options I needed. Let me try to investigate it more.

[chudsaviet]

Good news - h264_v4l2m2m works on Rpi4 on arm64, and pretty fast. :-D I will work on a patch, and maybe we will switch armhf version to it too.

[chudsaviet]

Bad news: ‘h264_v4l2m2m’ encoder in ffmpeg is pretty buggy. It does not work correctly when I’m trying to split streams from webcam to two h264 and one jpeg stream. I will investigate if we can use software encoder without overloading CPU.

[fastcompjason]

Any updates on this? I am trying to use an older Logitech Pro 9000 webcam with OctoPi running on a new RPi 4B+ 8GB and when I specifically enable HLS, I get nothing at all. I have even tried to specifically change the config file to specify a resolution and refresh rate. I would very much like to run a H.264 stream out of OctoPi to import into OBS or into other software hardware platforms for monitoring if possible.

[guysoft]

None, also found this and there seems to be no news there neither: https://github.com/jellyfin/jellyfin/issues/4023#issue-689108282

[chudsaviet]

Yeah, we can only run the software encoder on arm64 for now.

[cryptoAlgorithm]

Bad news: ‘h264_v4l2m2m’ encoder in ffmpeg is pretty buggy. It does not work correctly when I’m trying to split streams from webcam to two h264 and one jpeg stream.

@chudsaviet yeah I observed that too, using h264_v4l2m2m in place of h264 throws a few errors, firstly the -preset option isn't supported, next the stream simply doesn't work with the preset removed. It works perfectly with h264 though, using around 110% CPU on my pi4b overclocked to 1.8GHz.

[cryptoAlgorithm]

Is there a way to get h264_v4l2m2m to work with just one stream? That would be great!

[chudsaviet]

Yep, just remove JPEG stream parameters from ffmpeg config,

[cryptoAlgorithm]

@chudsaviet So I could just get rid of the jpeg stream part, and it would work? I can still have 2 streams?

[cryptoAlgorithm]

@chudsaviet No luck, substituting h264 for h264_v4l2m2m produces an unplayable stream (notice the "Plug-in handled load" error, ignore all the AdGuard messages)

Here's the full ffmpeg command that I'm running:

/usr/bin/ffmpeg \
    \
    -framerate 30 -video_size 1280x960 \
    -i /dev/video0 \
    -pix_fmt yuv420p \
    \
    -c:v h264_v4l2m2m \
    -b:v 800k -flags +cgop \
    -g 30 -keyint_min 30 \
    \
    -f hls -hls_time 1 \
    -hls_flags delete_segments+program_date_time+temp_file+independent_segments \
    -hls_allow_cache 0 -hls_segment_type fmp4 \
    -hls_list_size 2 -hls_delete_threshold 3 \
    /run/webcam/hls/720p/stream.m3u8

[chudsaviet]

v4l2m2m is buggy :( A good way to reduce CPU usage of software h264 encoder is to reduce resolution, fps and use -preset veryfast parameter.

[cp2004]

HLS is also now broken on the 32 bit RPi OS images since the OMX encoders were removed - I tried to get it working myself but ran into very similar issues when actually debugging it as well (https://github.com/guysoft/OctoPi/issues/770#issuecomment-1120459794).

[chudsaviet]

That’s bad news, and definitely a regression. I will go look into if I can debug v4l2m2m.

[chudsaviet]

It looks like h264_v4l2m2m works on the latest RaspberryPi OS version with kernel 5.15.32-v8+, but only for fixed-duration video files, not HLS.
There is a problem with 'dts` timestamp, which is required by HLS.
I will look into fixing it - https://github.com/raspberrypi/linux/blob/6b945e6f05d5cb0a299dbbb9bdd285aff330a204/drivers/staging/vc04_services/bcm2835-codec/bcm2835-v4l2-codec.c#L1289

[chudsaviet]

Example for my Microsoft LifeCam Studio:

sudo fuser -k /dev/video0 ; \
sudo rm /var/www/html/test.mp4 ; \
sudo v4l2-ctl -d /dev/video0 -c exposure_auto=1 && \
sudo v4l2-ctl -d /dev/video0 -c exposure_absolute=20 && \
ffmpeg \
    -loglevel info \
    \
    -t 00:00:10 \
    -input_format mjpeg \
    -framerate 60 -video_size 1280x720 \
    -i /dev/video0 \
    \
    -filter:v "format=yuv420p" \
    \
    -c:v h264_v4l2m2m \
    -b:v 2048k \
    \
    /var/www/html/test.mp4

[cryptoAlgorithm]

Yeah I experienced the same issues when using h264_v4l2m2m, which made me suspect some meta is missing in the m3u8 stream produced by it

[chudsaviet]

Ok, good news everyone!
I made it work.
I used:

• FFmpeg built from master, commit 59018254c7a178e4650ef63016493c75383a2076
• Latest stable RaspberryPi OS on 2022-07-09, kernel 5.15.32-v8+
• Rpi 4 8GiB RAM

These FFmpeg parameters (MS LifeCam Studio produces 720p@30fps only in MJPEG):

sudo fuser -k /dev/video0 ; \
sudo rm /var/www/html/stream.m3u8 ; \
sudo v4l2-ctl -d /dev/video0 -c exposure_auto=1 && \
sudo v4l2-ctl -d /dev/video0 -c exposure_absolute=20 && \
./ffmpeg \
     -loglevel info \
     -input_format mjpeg \
     -framerate 30 -video_size 1280x720 \
     -i /dev/video0 \
     \
     -filter:v "format=yuv420p" \
     -c:v h264_v4l2m2m \
     -b:v 3792k \
     -level:v 4.0 \
     \
     -f hls -hls_time 1 \
     -hls_flags delete_segments+program_date_time+temp_file+independent_segments \
     -hls_allow_cache 0 -hls_segment_type fmp4 \
     -hls_list_size 2 -hls_delete_threshold 3 \
     /var/www/html/stream.m3u8

CPU load was around 10% on all cores.

I will now work on putting it into OctoPi.

[chudsaviet]

Looks like I will need latest FFmpeg from ‘master’. It can be done, but it will increase image build time.
@guysoft , would you merge such a change?

[cryptoAlgorithm]

@chudsaviet is the raspberry pi os 64-bit? octopi seems to be using Ubuntu instead for the 64 bit images though, don't know if this solution will work for it too.

[cryptoAlgorithm]

@chudsaviet is the raspberry pi os 64-bit? octopi seems to be using Ubuntu instead for the 64 bit images though, don't know if this solution will work for it too.

[chudsaviet]

Ubuntu was chosen for 64bit only because there were no stable 64bit RaspberryPi OS builds. Now there are. Its time to switch back.

V4L2 M2M It shall work on 32bit too, actually. I just haven't tested it yet.

[cryptoAlgorithm]

Time to reinstall I guess...

[guysoft]

I am not sure if we should switch from Ubuntu or not. Quick vote of hands - is there any preference to one or the other to you?

[cryptoAlgorithm]

If we can move over flawlessly and everything remains the same, I'm in vote of moving to raspbian since its easier to use tools everyone is familiar with like raspi-config etc. Ubuntu is lacking in that aspect.

Message ID: @.***>

[chudsaviet]

As we see, RaspberryPi OS have better hardware support - correctly working V4L2 M2M is an example.
And now its stable in 64bit. Anyway, its a question for another topic.

My question was about building a copy of FFmpeg just for HLS on OS image build time. Will it be OK?

[cp2004]

I am not sure if we should switch from Ubuntu or not. Quick vote of hands - is there any preference to one or the other to you?

It would make user-support much easier if the two distros were the same. No need for different guides or constant questions of which OS you installed before helping people.

[chudsaviet]

Created pull request https://github.com/guysoft/OctoPi/pull/784

[guysoft]

Ok, based on this and a few other similar answers I will move the 64bit build to use the rpi image. Also got a an answer from rpi I am referencing here: https://github.com/RPi-Distro/pi-gen/issues/481#issuecomment-1160074790

[cryptoAlgorithm]

Ok! Does the 64-bit image work on 32-bit Pis? if it does then you could just use one image, no need for 2.

[chudsaviet]

No, 64bit image works on RPi 3+ and Zero 2+.

[guysoft]

64bit can only work on devices that can execute arm64bit instruction set. That means it will not run on Rpi1, zero 2, and 3, 1A+ etc

[cryptoAlgorithm]

Ok that's a small bummer. IMO tho, you shouldn't be using OctoPrint on a pi 1 or similar, the performance would be basically unusable. Yeah, you could just keep 2 images around then.

[cryptoAlgorithm]

@chudsaviet Does compiling ffmpeg from source work on Ubuntu too? I currently still have that installed.

[cryptoAlgorithm]

@chudsaviet Could you confirm if your ffmpeg version is above 5.0? I'm going to try compiling from source on my ubuntu image, hopefully I can get hardware encoding working too. If so, it could be possible to just stick with using ubuntu for the 64-bit image.

[chudsaviet]

Yes, of course, you can easily build FFmpeg from ‘master’ branch on Ubuntu. Its newer than any released version. Just use https://github.com/FFmpeg/FFmpeg .

[chudsaviet]

But on Ubuntu, I'm not sure you have the same latest kernel as on latest RpiOS. Kernel plays a major role in V4L2 M2M too.