search

raspberrypi / linux

Kernel source tree for Raspberry Pi-provided kernel builds. Issues unrelated to the linux kernel should be posted on the community forum at https://forums.raspberrypi.com/
Other
11.08k stars 4.96k forks source link

Kernel 5.10 - RPI3 - Kodi 18.x does not play HEVC videos anymore #4112

Open mkreisl opened 3 years ago

mkreisl commented 3 years ago

Now that the kernel 5.10 has been set to default, I would like to point out that with Kodi v18 playback of HEVC videos with Raspberry Pi3 is no longer possible. Used a kernel 5.10.10 with default settings (bcm2709_defconfig). A Raspberry Pi4 on the other hand plays them and the 5.4.x kernel also shows no problems. Most of the time the playback starts, but the picture is much darker than usual, the sound is normal. but after a few seconds the playback stops and kodi restarts

The playback problem has already started with 5.7 or 5.8 kernel and has not improved until now.

This can be found in the log. Seems that there is some memory allocation problem with MMAL

Feb  2 15:51:20 kmxbilr2 2 15:51:20.088 T:1415569632   DEBUG: VideoPlayer::Sync - Audio - pts: 288000.000000, cache: 509577.709119, totalcache: 800000.011921
Feb  2 15:51:20 kmxbilr2 2 15:51:20.088 T:1415569632   DEBUG: VideoPlayer::Sync - Video - pts: 5000.000000, cache: 50000.000000, totalcache: 100000.000000
Feb  2 15:51:20 kmxbilr2 2 15:51:20.088 T:1423962336   DEBUG: CVideoPlayerVideo - CDVDMsg::GENERAL_RESYNC(-221577.709119)
Feb  2 15:51:20 kmxbilr2 2 15:51:20.089 T:1753329888   DEBUG: OnAVStarted: CApplication::OnAVStarted
Feb  2 15:51:20 kmxbilr2 2 15:51:20.089 T:1283715296   DEBUG: CVideoPlayerAudio - CDVDMsg::GENERAL_RESYNC(-221577.709119), level: 96, cache: 501168.959119
Feb  2 15:51:20 kmxbilr2 2 15:51:20.089 T:1283715296   DEBUG: CDVDAudio::Resume - resume audio stream
Feb  2 15:51:20 kmxbilr2 2 15:51:20.090 T:1423962336   ERROR: CMMALPool::GetBuffer - failed pool:0x56b18ca0 omvb:(nil) mmal:0x56bf9650 timeout:500
Feb  2 15:51:20 kmxbilr2 2 15:51:20.090 T:1423962336   ERROR: CDecoder::FFGetBuffer Failed to allocated buffer in time
Feb  2 15:51:20 kmxbilr2 2 15:51:20.090 T:1423962336   ERROR: ffmpeg[54DFF0E0]: [hevc_rpi] get_buffer() failed
Feb  2 15:51:20 kmxbilr2 2 15:51:20.091 T:1874850016   DEBUG: CRBP::AdjustHDMIClock(1.00000) = 1.00000
Feb  2 15:51:20 kmxbilr2 2 15:51:20.091 T:1874850016   DEBUG: ActiveAE - start sync of audio stream
Feb  2 15:51:20 kmxbilr2 2 15:51:20.091 T:1342464224   ERROR: ffmpeg[500460E0]: [hevc_rpi] thread_get_buffer() failed
Feb  2 15:51:20 kmxbilr2 2 15:51:20.091 T:1342464224   DEBUG: ffmpeg[500460E0]: [hevc_rpi] Error parsing NAL unit #0.
Feb  2 15:51:20 kmxbilr2 2 15:51:20.095 T:1423962336   ERROR: CMMALPool::GetBuffer - failed pool:0x56b18ca0 omvb:(nil) mmal:0x56bf9828 timeout:500
Feb  2 15:51:20 kmxbilr2 2 15:51:20.095 T:1423962336   ERROR: CDecoder::FFGetBuffer Failed to allocated buffer in time
Feb  2 15:51:20 kmxbilr2 2 15:51:20.095 T:1423962336   ERROR: ffmpeg[54DFF0E0]: [hevc_rpi] get_buffer() failed
Feb  2 15:51:20 kmxbilr2 2 15:51:20.096 T:1334071520   ERROR: ffmpeg[4F8450E0]: [hevc_rpi] thread_get_buffer() failed
Feb  2 15:51:20 kmxbilr2 2 15:51:20.096 T:1334071520   DEBUG: ffmpeg[4F8450E0]: [hevc_rpi] Error parsing NAL unit #0.
Feb  2 15:51:20 kmxbilr2 2 15:51:20.098 T:1423962336   ERROR: CMMALPool::GetBuffer - failed pool:0x56b18ca0 omvb:(nil) mmal:0x56bf9a00 timeout:500
Feb  2 15:51:20 kmxbilr2 2 15:51:20.098 T:1423962336   ERROR: CDecoder::FFGetBuffer Failed to allocated buffer in time
Feb  2 15:51:20 kmxbilr2 2 15:51:20.099 T:1423962336   ERROR: ffmpeg[54DFF0E0]: [hevc_rpi] get_buffer() failed
Feb  2 15:51:20 kmxbilr2 2 15:51:20.100 T:1325678816   ERROR: ffmpeg[4F0440E0]: [hevc_rpi] thread_get_buffer() failed
Feb  2 15:51:20 kmxbilr2 2 15:51:20.100 T:1325678816   DEBUG: ffmpeg[4F0440E0]: [hevc_rpi] Error parsing NAL unit #0.

Used firmware:

vcgencmd version
Jan  7 2021 18:34:00 
Copyright (c) 2012 Broadcom
version fb345a0c2d5544957f4ba1a2b9e968970e3312c4 (clean) (release) (start_x)

Tried gpu_mem=192/384/512 without any difference

Maybe I missed something?

popcornmix commented 3 years ago

I think this because /dev/vcsm no longer exists and so /dev/vcsm-cma is used instead. The former allocates from gpu_mem and the latter from cma (e.g add cma=256M to cmdline.txt). Try reducing gpu_mem and increasing cma. I think that will get it playing. But there is a performance hit to memory that is allocated from cma which isn't currently arm cacheable. We need to decide what the best solution for that is.

mkreisl commented 3 years ago

Yes, that was the solution I vaguely remember the discussion with cma=256M, but I didn't have it on my mind anymore

Thanks a lot @popcornmix

mkreisl commented 3 years ago

But there is a performance hit to memory that is allocated from cma which isn't currently arm cacheable. We need to decide what the best solution for that is.

Oh yes, now I see what you mean.HEVC is currently unusable with kernel 5.10 and RPi3, so please find an acceptable solution quickly.

Thanks

Maltz42 commented 3 years ago

I'm seeing this on a RPI4 running Kodi 18.7 as well. Since the kernel update, Kodi hangs shortly after HEVC videos begin to play.

amandamana commented 3 years ago

Found a report of kernel 5.10.11 breaking h264 1080p/50 playback (and degrading 720p) on a RPI2.

Maltz42 commented 3 years ago

Ok, so I've just set up a new RPi4 from scratch and ran into this same issue again. cma=256M did not solve the issue, but rolling back to a 5.4 kernel via rpi-update did.

Akrai commented 3 years ago

yeah I've been unable to play h265 videos anymore since i updated the kernel like two or three weeks ago, i hope a fix comes soon 🙏

harristom commented 3 years ago

Ok, so I've just set up a new RPi4 from scratch and ran into this same issue again. cma=256M did not solve the issue, but rolling back to a 5.4 kernel via rpi-update did.

I have the same problem - how did you do this?

timothyjward commented 3 years ago

Ok, so I've just set up a new RPi4 from scratch and ran into this same issue again. cma=256M did not solve the issue, but rolling back to a 5.4 kernel via rpi-update did.

I have the same problem - how did you do this?

I would assume that it was 

sudo BRANCH=stable rpi-update b7c8ef64ea24435519f05c38a2238658908c038e

As far as I can tell this is the last version of the stable firmware branch using a 5.4 kernel

Franck-Berry commented 3 years ago

FYI, I have the same problem on a Raspberry Pi 4: unable to play HEVC videos with Kodi 18.7 since the new Kernel 5.10 update. (After switching back to Kernel 5.4.83, everything works again.)

I described the problem on the raspberrypi.org forum and provided a Kodi log (which doesn't help much): https://www.raspberrypi.org/forums/viewtopic.php?f=66&t=251645&hilit=kodi&sid=73f21dfa564d8f79c05ff5d2d08fee70&start=475#p1828887 https://www.raspberrypi.org/forums/viewtopic.php?f=66&t=251645&hilit=kodi&sid=73f21dfa564d8f79c05ff5d2d08fee70&start=475#p1829049

popcornmix commented 3 years ago

Currently the only solution to get hevc working well on Pi2/3 is to downgrade kernel to 5.4 (e.g. https://github.com/raspberrypi/linux/issues/4112#issuecomment-789582299)

Maltz42 commented 3 years ago

Is there a solution other than a kernel downgrade for Pi4? I'm wondering why you specified Pi2/3, since this is also a problem for Pi4.

popcornmix commented 3 years ago

hevc should still work on Pi4. You may need to increase cma memory allocation. e.g. remove the gpu_mem setting from config.txt and add a cma setting to the fkms overlay. e.g.

dtoverlay=vc4-fkms-v3d,cma-256
Maltz42 commented 3 years ago

As I mentioned above, that has not worked for my Pi4 that I just set up from scratch when running 5.10

popcornmix commented 3 years ago

I worked for me with a fresh sdcard image and an updated older image on Pi4 (both 8-bit 720p and 10-bit 4k hevc files) What resolution files are you testing? For 4K it may be necessary to increase cma futher. e.g.

dtoverlay=vc4-fkms-v3d,cma-512
Franck-Berry commented 3 years ago

On my Pi4, adding the cma option (with 256 or 512) does not correct the problem. Maybe the problem only appears with certain files?

Maltz42 commented 3 years ago

So here's what I've done. I'm beginning to think that this may not be a CMA issue. I've also discovered that booting to the 64bit version of 5.4 exhibits the exact same symptoms.

1 - Started with a fresh Raspbian Lite image from Jan 11 which includes the 5.4 kernel. Hardware is a 4GB RPi4. 2 - added gpu_mem=256 to config.txt, which is required for Kodi 3 - sudo apt-mark hold rasperrypi-kernel to hold kernel at v5.4 4 - update/upgrade everything and installed kodi

-- At this point, all videos tested play correctly --

5 - unhold the kernel and update to 5.10 6 - NO further changes to config.txt or cmdline.txt are made - None of the cma commands are added, etc.

-- At this point, all videos tested STILL play fine, except for some HEVC files. I have 1080p HEVC that files play fine, as do 480p H.264 files. But my 480p HEVC will no longer play. I'm not sure if this has anything to do with resolution, or some other mastering difference, but it doesn't seem memory related, since high-resolution files play fine and low-res files won't, even with cma-512.

7 - many combinations of CMA values are tried in both config.txt and cmdline.txt, hardware acceleration tried enabled/disabled, with and without gpu_mem=256, etc. Nothing works to get 480p HEVC files playing. 8 - Roll back to 5.4 using the hash timothyjward listed above

-- Everything works fine again, including all 480p HEVC videos --

9 - Add arm_64bit=1 to config.txt

-- I'm now running the 64-bit 5.4.83 kernel, but Kodi is back to the exact same symptoms as with the 5.10 kernel. I don't know if this provides a clue, but I found it interesting. More precise symptom descriptions and log output forthcoming...

Maltz42 commented 3 years ago

Ok, my specific symptoms are that when a video fails, it starts to play, video runs for a second or so, then freezes. Audio plays on for a few more seconds, then that freezes, too. Kodi is still responsive for a little while until you either manually stop the video or it gives up and tries to stop it itself, and then it hangs, trying and failing to kill the playback thread, which itself is hung. kill -9 is required to kill that thread.

Well, after pouring through logs at maximum debug level, with all possibly relevant component logging enabled in Kodi, the only thing I could find that was different between the successful playbacks and the failures was:

NOTICE: CVideoPlayerAudio::Process - stream stalled

Not terribly useful, I know. Sorry. There's nothing in kodi's stderr or stdout, either. I was able to confirm that it doesn't seem to matter the source - the error occurs identically whether the source is an HTTPS stream or a local file.

I also didn't have any luck turning MMAL on or off, however, that setting made no difference in frame rate or CPU usage for HEVC videos. It played 1080p HEVC smoothly at moderate CPU usage with or without MMAL enabled, yet 1080p H.264 videos often stuttered at 100% CPU without it. So I wonder if it's really even disabled for HEVC.

So yeah, I'm pretty well stumped at this point... Maybe the fact that the issue is identical with the 64-bit 5.4 kernel will hint at something I'm missing.

Aarda commented 3 years ago

@popcornmix Is it cma-256 or cma=256?

Anyhow my HEVC streams from DVB-T2 are playing fine on an updated RPI4 with Kodi.

But I have attached a HEVC file ( 5s

https://user-images.githubusercontent.com/53397439/110206300-27e0fc80-7e7d-11eb-8c46-1e318b80f35a.mp4

phone recording) that freezes Kodi on RPI4, doesn't play well on VLC on a 4, but plays reasonable well on a 3B (LibreElec 9.2)

(Dunno why attaching is embedding, must be AI-logic)

user-images.githubusercontent.com/53397439/110206300-27e0fc80-7e7d-11eb-8c46-1e318b80f35a.mp4

pelwell commented 3 years ago

It's dtoverlay=vc4-fkms-v3d,cma-256 - cma-256 is the name of a parameter, and without a value you are effectively saying cma-256=on. It's done this way (as a set of options rather than just taking any value) because the property it is modifying expects a value in bytes, which is a large number for 256MB.

Maltz42 commented 3 years ago

I've seen in several discussions "cma=256M" as a parameter in cmdline.txt, too, so I think it's both. But from what I can tell, 256M is the default anyway when I'm running the 5.10 kernel on my 4GB RPi4. But I've tried 256 and 512 in both cmdline.txt and config.txt (using appropriate syntax for each) and it made no difference to what I could and could not play. That's why I don't think anymore that this is a CMA issue, at least on my device.

timothyjward commented 3 years ago

I am also unable to play HEVC on Kodi 18.7 (the version in the raspbian buster repo). When I try Kodi crashes and restarts (probably because I’m using the standalone script).

I’m using the latest kernel and firmware (5.10.17) on my Pi 4 4GB. I have

dtoverlay=vc4-fkms-v3d,cma-512

in config.txt and no GPU memory defined.

In Kodi 19 I get the same problem. I also can’t play H.264 in Kodi 19, either I get a black screen, or the player freezes about 5 seconds in.

If it helps I see this in the crash log:

2021-03-06 17:00:50.784 T:2431639760  NOTICE: running thread: CVideoPlayerAudio::Process()
2021-03-06 17:00:50.785 T:2431639760  NOTICE: Creating audio stream (codec id: 86060, channels: 8, sample rate: 48000, no pass-through)
2021-03-06 17:00:50.791 T:2576146640  NOTICE: CDecoder::Open - fmt:187
2021-03-06 17:00:50.900 T:2973437024   ERROR: eglSwapBuffers failed (EGL_BAD_DISPLAY)

############### END LOG FILE ################

############ END Kodi CRASH LOG #############
pelwell commented 3 years ago

512MB may be too much CMA - try 384MB.

Aarda commented 3 years ago

dtoverlay=vc4-fkms-v3d,cma-384

Unfortunately it doesn't help.

Can anyone play this HEVC file on RPI4 Kodi/Buster?

wget user-images.githubusercontent.com/53397439/110206300-27e0fc80-7e7d-11eb-8c46-1e318b80f35a.mp4

It's freezing here:

NOTICE: Creating video codec with codec id: 173`
ERROR: CMMALVideo::Open : Video codec unknown: ad
mkreisl commented 3 years ago

@Aarda, I can XBian/Buster, Kernel 5.10.20

NOTICE: Creating video codec with codec id: 173` ERROR: CMMALVideo::Open : Video codec unknown: ad

Thats weird, because MMAL should no longer exist in Kodi v19, or are you usind Kodi v18?

Aarda commented 3 years ago

@mkreisl thx. Yes, 18.7 is still shipped in Raspberry Pi OS. Maybe it should be bumped to 19, I'll see if I can build Kodi 19 tomorrow

timothyjward commented 3 years ago

512MB may be too much CMA - try 384MB.

I’ve tried cma-384 and I still get EGL_BAD_DISPLAY when trying to decode HEVC using 18.7. Do I also need to set GPU memory? I’ve seen things saying yes and other things saying no.

Kodi 19 still gives a black screen for H.264 and crashes with a bad alloc for HEVC.

popcornmix commented 3 years ago

A few clarifications. /dev/vcsm allocates memory through the gpu interface and requires sufficient gpu_mem to be configured. /dev/vcsm-cma allocates memory through kernel side cma and requires sufficient cma to be configured.

/dev/vcsm an old firmware centric api that we are moving away from, and /dev/vcsm-cma is a preferred interface. vcsm has never existed on 64-bit kernel (even at 5.4) and doesn't exist in 5.10 kernel (on 32-bit).

If you need hevc playback on Pi2/3 then you need to stick to 32-bit 5.4 kernel for the foreseeable future. We do want to get this working again in vcsm-cma world but it's a big job as the pi2/3 accelerated software decoder relies on fine-grained cache operations that just don't exist in through cma.

On Pi4, the hardware decoder is happy with either vcsm or vcsm-cma. gpu_mem can be left unspecified (76M) and cma can be left unspecified (256M) for most content. You may need to boost cma for harder 4K streams. Don't go beyond 512M - that is more than enough.

cma can be specified either on cmdline.txt as cma=512M or in config.txt as dtoverlay=vc4-fkms-v3d,cma-512. They are different files with different conventions so don't mix up =512 and -512 - it is important.

The config.txt option is preferred as that guarantees the memory lives in bottom 1GB which the hardware requires. The cmdline.txt will typically use the bottom 1GB and be fine, but it's not gauranteed.

Aarda commented 3 years ago

@popcornmix Thank you for the explanation.

What's not totally clear to me. Has RPI4 still support for HEVC with Kodi 18.7 and the 5.10 kernel. Some HEVC files seem to play, some not.

I build Kodi 19 today in R Pi OS on a Pi400:

https://github.com/popcornmix/xbmc/archive/gbm_matrix.zip cmake .. -DCMAKE_INSTALL_PREFIX=/usr/local -DCORE_PLATFORM_NAME=gbm -DAPP_RENDER_SYSTEM=gles -DENABLE_VAAPI=OFF -DENABLE_VDPAU=OFF -DENABLE_DEBUGFISSION=OFF -DENABLE_TESTING=OFF -DENABLE_INTERNAL_FLATBUFFERS=ON -DENABLE_INTERNAL_FFMPEG=ON -DENABLE_INTERNAL_SPDLOG=ON -DENABLE_INTERNAL_FMT=ON

I got h264 working with DRM PRIME HW acceleration, but HEVC files were crashing directly: (they played only with HW disabled) std::bad_alloc CDVDVideoCodecDRMPRIME::AddData - send packet failed: Cannot allocate memory (-12)

Is it a missing udev rule? Could not find a permission error in trace. Did I miss something else?

Akrai commented 3 years ago

@Aarda how much CMA are you allocating? Maybe with the default preset it is not enough for 1080p 60fps

mkreisl commented 3 years ago

I'm sorry guys, but this is now totally off topic and has nothing to do with the original problem.

Why don't you discuss this in the Kodi forum?

Akrai commented 3 years ago

@mkreisl what do you mean? We are still talking about the same issue from the beggining

Aarda commented 3 years ago

@mkreisl yes and no.

It's about a RPI4 on Kodi 18.x not playing HEVC videos anymore with kernel 5.10 (Raspberry Pi OS). You state in the issue that a RPI4 does play them, people report some do not.

You mentioned Kodi 19, so I decided to give that a go. Downgrading the kernel is a workaround and not a solution.

Where to discuss? Good question. I'm happy to open another issue.

Akrai, tried different settings: default, 256, 384, 512

timothyjward commented 3 years ago

It's about a RPI4 on Kodi 18.x not playing HEVC videos anymore with kernel 5.10 (Raspberry Pi OS). You state in the issue that a RPI4 does play them, people report some do not.

This is certainly my issue. The OP mentions the Pi 3 not playing HEVC with Kodi 18.x and kernel 5.10.y - I am seeing the same thing with my Pi 4.

harristom commented 3 years ago

Ok, so I've just set up a new RPi4 from scratch and ran into this same issue again. cma=256M did not solve the issue, but rolling back to a 5.4 kernel via rpi-update did.

I have the same problem - how did you do this?

I would assume that it was

sudo BRANCH=stable rpi-update b7c8ef64ea24435519f05c38a2238658908c038e

As far as I can tell this is the last version of the stable firmware branch using a 5.4 kernel

Thanks, this fixed it for me on my Pi 4

Akrai commented 3 years ago

So I just tried the cma options and I can confirm using it (for example cma-512) does work for some easy normal bitrate 1080p movies, but it doesn't for higher res, high bitrate ones, the system still crashes

harristom commented 3 years ago

Apt updated to the latest stable kernel and the problem still isn't fixed. Had to roll back to b7c8ef64ea24435519f05c38a2238658908c038e again.

In case it helps troubleshooting, I am on a Pi 4. Kodi crashes on some (but not all) HEVC files. Resolution doesn't seem to make a difference and anyway they are all 1080p or lower. Attaching my config.txt and cmdline.txt (basically at defaults, with 320M GPU memory added as required by Kodi). config.txt cmdline.txt

ZedeN1 commented 3 years ago

Apt updated to the latest stable kernel and the problem still isn't fixed. Had to roll back to b7c8ef64ea24435519f05c38a2238658908c038e again.

My RPi4 would not let me downgrade after apt upgrade (latest firmware already installed message). I had to do this first: sudo rm /boot/.firmware_revision sudo BRANCH=stable rpi-update b7c8ef64ea24435519f05c38a2238658908c038e

pelwell commented 3 years ago

[ Deleted off-topic comments ]

Octoton commented 3 years ago

On a raspberry pi 4 with kernel 5.10 and kodi 18.4 and after lot of crash I was able to read my hevc by following popcornmix answer in https://forum.kodi.tv/showthread.php?tid=354154

I found this after looking at the console output of kodi as the log had no info whatsoever. The console was outputing: can't open /dev/argon-hevcmem