vo_dmabuf_wayland: Crashes when uploading P010 to VAAPI (only AMD?)

[rmader]

Update description

This is on latest master and ffmpeg-6.0-11.fc38

P010 is basically the 10bit equivalent of NV12 and widely supported by modern GPUs. Right now I can't play it with vo_dmabuf_wayland, neither with software nor hardware decoding (see below). In both cases it does not like issues in vo_dmabuf_wayland, but opening the issue for it anyway so we can triage further.

STR:

• download a 10bit test file. I used "Camp by Sony" from https://kodi.wiki/view/Samples which is 4k, 60fps and 10bit and therefore a good benchmark
• play it with mpv --vo=dmabuf-wayland camp_by_sony.mp4 --fs
• software decoding crashes in ffmpeg/libavutil/imgutils.c with the stack below

mpv --vo=dmabuf-wayland camp_by_sony.mp4 --fs
 (+) Video --vid=1 (*) (hevc 3840x2160 59.940fps)
 (+) Audio --aid=1 --alang=eng (*) (aac 2ch 48000Hz)
Cannot load libcuda.so.1
AO: [pipewire] 48000Hz stereo 2ch floatp
Invalid HDR peak in stream: 0.002463
[autoconvert] HW-uploading to vaapi
[autoconvert] Converting yuv420p10 -> p010
[hwupload] upload p010 -> vaapi[p010]
Segmentation fault (core dumped)

bt full

``` #0 0x00007f4135028307 in __memmove_avx_unaligned_erms () at /lib64/libc.so.6 #1 0x00007f4135c57ff6 in memcpy (__len=7680, __src=, __dest=) at /usr/include/bits/string_fortified.h:29 #2 image_copy_plane (height=1080, bytewidth=7680, src_linesize=7680, src=, dst_linesize=7680, dst=) at libavutil/imgutils.c:353 #3 image_copy_plane (dst=, dst_linesize=7680, src=, src_linesize=7680, bytewidth=7680, height=) at libavutil/imgutils.c:344 #4 0x00007f4135c5de63 in image_copy (dst_data=0x7ca8340, dst_linesizes=dst_linesizes@entry=0x7ffcff4324b0, src_data=src_data@entry=0x7ffcff432520, src_linesizes=src_linesizes@entry=0x7ffcff432490, pix_fmt=AV_PIX_FMT_P010LE, width=3840, height=2160, copy_plane=0x7f4135c57f90 ) at libavutil/imgutils.c:415 h = bwidth = i = planes_nb = 2 desc = 0x7f4135ce4670 #5 0x00007f4135c5dfd1 in av_image_copy (dst_data=, dst_linesizes=, src_data=0x7ffcff432520, src_linesizes=, pix_fmt=, width=, height=2160) at libavutil/imgutils.c:434 dst_linesizes1 = {7680, 7680, 0, 0} src_linesizes1 = {7680, 7680, 0, 0} i = #6 0x00007f4135c49567 in frame_copy_video (src=0x7ca7bc0, dst=0x7ca8340) at libavutil/frame.c:712 src_data = {0x7f4042817040 "", 0x7f4043807040 "", 0x0, 0x0} i = #7 av_frame_copy (dst=0x7ca8340, src=0x7ca7bc0) at libavutil/frame.c:769 #8 0x00007f4135c51f68 in vaapi_transfer_data_to (hwfc=, dst=0x7beb200, src=0x7ca7bc0) at libavutil/hwcontext_vaapi.c:968 map = 0x7ca8340 err = 0 #9 0x00007f4135c490c0 in av_hwframe_transfer_data (dst=0x7beb200, src=0x7ca7bc0, flags=flags@entry=0) at libavutil/hwcontext.c:497 ctx = ret = #10 0x00000000004c4a55 in mp_image_hw_upload (src=0x78ef510, hw_img=0x15dd460) at ../video/mp_image_pool.c:350 ok = false dstav = 0x7beb200 srcav = 0x7ca7bc0 #11 mp_image_hw_upload (hw_img=0x15dd460, src=0x78ef510) at ../video/mp_image_pool.c:327 #12 0x00000000004c4c4a in mp_av_pool_image_hw_upload (hw_frames_ctx=, src=src@entry=0x78ef510) at ../video/mp_image_pool.c:438 av_frame = 0x0 dst = 0x15dd460 #13 0x0000000000458adc in process (f=0x6801770) at ../filters/f_hwtransfer.c:214 p = 0x78ed570 frame = {type = MP_FRAME_VIDEO, data = 0x78ef510} src = 0x78ef510 dst = map_images = #14 0x000000000046044d in mp_filter_graph_run (filter=) at ../filters/filter.c:262 next = 0x6801770 r = 0x160f9c0 __PRETTY_FUNCTION__ = "mp_filter_graph_run" end_time = 11075119 exit_req = false externals = #15 0x0000000000460618 in filter_recursive (p=p@entry=0x164fc40) at ../filters/filter.c:182 f = __PRETTY_FUNCTION__ = "filter_recursive" r = 0x160f9c0 #16 0x00000000004609d9 in mp_pin_out_request_data (p=) at ../filters/filter.c:327 #17 mp_pin_out_request_data (p=0x164fc40) at ../filters/filter.c:318 #18 mp_pin_out_read (p=0x164fc40) at ../filters/filter.c:340 #19 0x00000000004a66c4 in video_output_image (logical_eof=, mpctx=0x15990e0) at ../player/video.c:501 img = 0x0 frame = {type = MP_FRAME_NONE, data = 0x0} hrseek = false hrseek_pts = 0 tolerance = vo_c = 0x15cea60 r = 1 opts = 0x15a2bd0 track = 0x7f4120006930 vo_c = 0x15cea60 vo = 0x164b4e0 logical_eof = false r = p = rc = {x0 = 106768, y0 = 0, x1 = 469766016, y1 = 32577} time_frame = pts = __PRETTY_FUNCTION__ = "write_video" req = dummy = {pts = 0, duration = 0, vsync_interval = 0, vsync_offset = 1.130209551830802e-316, ideal_frame_duration = 4.4465908125712189e-323, num_vsyncs = 4606303, redraw = false, repeat = false, still = false, display_synced = false, can_drop = 176, current = 0x15990e0, num_frames = 23377120, frames = {0x15d0e30, 0x15a2bd0, 0x4ba13a0, 0x4ba13a8, 0x4b64fb0, 0x0, 0x4291e4 , 0x15990e0, 0x15990e0, 0x233d260}, frame_id = 4725788} frame = diff = #20 write_video (mpctx=mpctx@entry=0x15990e0) at ../player/video.c:1036 opts = 0x15a2bd0 track = 0x7f4120006930 vo_c = 0x15cea60 vo = 0x164b4e0 logical_eof = false r = p = rc = {x0 = 106768, y0 = 0, x1 = 469766016, y1 = 32577} time_frame = pts = __PRETTY_FUNCTION__ = "write_video" req = dummy = {pts = 0, duration = 0, vsync_interval = 0, vsync_offset = 1.130209551830802e-316, ideal_frame_duration = 4.4465908125712189e-323, num_vsyncs = 4606303, redraw = false, repeat = false, still = false, display_synced = false, can_drop = 176, current = 0x15990e0, num_frames = 23377120, frames = {0x15d0e30, 0x15a2bd0, 0x4ba13a0, 0x4ba13a8, 0x4b64fb0, 0x0, 0x4291e4 , 0x15990e0, 0x15990e0, 0x233d260}, frame_id = 4725788} frame = diff = #21 0x00000000004a1cc1 in run_playloop (mpctx=mpctx@entry=0x15990e0) at ../player/playloop.c:1215 #22 0x000000000049bd33 in play_current_file (mpctx=mpctx@entry=0x15990e0) at ../player/loadfile.c:1831 opts = playback_start = 10.444577000000001 __PRETTY_FUNCTION__ = "play_current_file" start_event = {playlist_entry_id = 1} end_event = {reason = MPV_END_FILE_REASON_EOF, error = 0, playlist_entry_id = 1, playlist_insert_id = 0, playlist_insert_num_entries = 0} watch_later = false play_start_pts = nothing_played = #23 0x000000000049c3b9 in mp_play_files (mpctx=mpctx@entry=0x15990e0) at ../player/loadfile.c:2017 new_entry = #24 0x000000000049d150 in mpv_main (argc=, argv=) at ../player/main.c:439 mpctx = 0x15990e0 options = r = 0 rc = reason = #25 0x00007f4134ef4b4a in __libc_start_call_main () at /lib64/libc.so.6 #26 0x00007f4134ef4c0b in __libc_start_main_impl () at /lib64/libc.so.6 #27 0x000000000041c695 in _start () ```

---

See also https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/2733

[llyyr]

This works on wlroots/sway git with both vaapi and no hwdec

[rmader]

This works on wlroots/sway git with both vaapi and no hwdec

What hardware is that? Intel by chance?

P.S.: and did you double-check that P010 is indeed used, not NV12?

[llyyr]

Works on both Intel i5-13600k iGPU, and RX 6600 XT.

and did you double-check that P010 is indeed used, not NV12?

yes

VO: [dmabuf-wayland] 3840x2160 vaapi[p010]

[rmader]

Hm - maybe a already fixed version of ffmpeg? Mine is ffmpeg-6.0-11.fc38, which do you have?

[llyyr]

I was indeed running off ffmpeg-git, but it works for me with ffmpeg-6 as well.

no vaapi case appears to work as it should too

[autoconvert] HW-uploading to vaapi
[autoconvert] Converting yuv420p10 -> p010
[hwupload] upload p010 -> vaapi[p010]

[Dudemanguy]

Oh this case definitely crashes for me too (rx 560; maybe not quite modern). I assume this is some driver weirdness or something, but I was never sure.

[rmader]

Hm, @llyyr, IIUC you use a Intel CPU, correct? Given that the crash is in memcpy()/__memmove_avx_unaligned_erms(), maybe that a bug with AMD CPUs?

The other thing that puzzles me why VA-API does not work here but for you, as it clearly works with Gstreamer. Guess will have to go debug :)

[llyyr]

Yep, I am on an Intel CPU. And if it matters, I also compile all of mesa/ffmpeg/mpv from git with no compiler optimizations. So it's possible that might be related as well? Still works for me with all of them compiled with -march=native and -O2

[rmader]

Hm interesting - I don't know what changed, but forcing VAAPI just started working for me (mpv --hwdec=vaapi --vo=dmabuf-wayland camp_by_sony.mp4 --fs).

So the two issues I still see:

1. VAAPI is not selected automatically for 10bit h265 files, while it is for e.g 8bit h264.
2. Converting yuv420p10 -> p010 crashes on certain setups.

[rmader]

VAAPI is not selected automatically

Oops, this actually never happens, my bad - --hwdec=vaapi is required.

So the only issue is Converting yuv420p10 -> p010 crashing on some setups.

[rmader]

Looking at the stack again, the crash happens in mp_av_pool_image_hw_upload() so AFAICS in the [hwupload] upload p010 -> vaapi[p010] step - which IIUC uploads from RAM to VRAM.

This crashing in memcpy()->__memmove_avx_unaligned_erms() appears to be the actual issue, which again wouldn't be as surprising. Maybe a kernel issue in fact.

[Traneptora]

Are you using the mesa driver? If so it might make sense to report it upstream to mesa.

[rmader]

Are you using the mesa driver? If so it might make sense to report it upstream to mesa.

Yep, I'm trying to reach out to some AMD folks. Given that Mesa doesn't show up in the backtrace I rather suspect the kernel driver though.

[Dudemanguy]

Given that the crash is in memcpy()/__memmove_avx_unaligned_erms(), maybe that a bug with AMD CPUs?

A bit late here, but I can indeed confirm the same thing. My intel laptop has zero problems with the p010 -> vaapi upload step. It works just fine. On my desktop where this crashes, I'm using a Ryzen 5 3600 so it really seems somehow related to AMD CPUs.

[rmader]

Likely related: https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/2733

[rmader]

See also https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/2935

[Dudemanguy]

This actually doesn't crash anymore for me. I went back and checked a commit from this issue to double check and the p010 -> vaapi upload works no problem. I guess it was fixed at some point somewhere by someone (the kernel?). I'm on 6.6.7.

[Dudemanguy]

I'm fairly certain this was fixed by someone somewhere. I've reliably had p010 uploads for the past few months on AMD.