search

Xpra-org / xpra

Persistent remote applications for X11; screen sharing for X11, MacOS and MSWindows.
https://xpra.org/
GNU General Public License v2.0
1.99k stars 169 forks source link

`libyuv` crash on MS Windows #4425

Open totaam opened 10 hours ago

totaam commented 10 hours ago

No idea what causes it or when it started happening.

2024-11-22 14:08:21,262 csc scores: {-200: [('RGBX', libyuv(YUV420P to RGBX)), ('RGB', libyuv(YUV420P to RGB))]}
2024-11-22 14:08:21,262 debug enabled for xpra.codecs.codec_constants / ('encoding',)
2024-11-22 14:08:21,262 make_instance() libyuv - instance count=0
2024-11-22 14:08:21,262 libyuv.ColorspaceConverter.init_context(1240, 900, 'YUV420P', 1240, 900, 'RGBX', {'speed': 1})
2024-11-22 14:08:21,262 YUV420P -> RGBX planes=1, yuv-scaling=0, rgb-scaling=0, output buffer-size=4464000
2024-11-22 14:08:21,276 do_video_paint new csc decoder: libyuv(YUV420P 1240x900 RGBX)
2024-11-22 14:08:21,276 convert_yuv420p_image(ImageWrapper(YUV420P:(0, 0, 1248, 928, 24):3)) to RGBX

Segfaults.

But only in v5.x and only on MS Windows.

totaam commented 6 hours ago

This is an overflow caused by nvenc padded frames.

totaam commented 5 hours ago

See 353e5ca2f1da63c5ee9369ee089360187bc92840

This changeset needs to be backported: 2ef81a3fc32abedba8361e76cb66181ef77e71b8

But a bigger problem will remain: we should not trust the dimensions provided by the bitstream. These may overflow the buffers we allocate based on the window paint dimensions - which are potentially quite a bit smaller!