search

CESNET / UltraGrid

UltraGrid low-latency audio and video network transmission system
http://www.ultragrid.cz
Other
489 stars 55 forks source link

yuv444p10le input getting converted to UYVY with vulkan_sdl2/sdl display #275

Closed alatteri closed 1 year ago

alatteri commented 1 year ago

Hello,

Given an input stream of 12bit RGB 444 3G SDI on encoder utilizing -t decklink:codec=R12L -c libavcodec:encoder=libsvt_hevc which uses pixfmt yuv444p10le.

[Decklink capture] Format change detected (display mode - RGB444, 12bit).
[Decklink capture] Using codec: R12L
[DeckLink capture] Enable video input: 2Kp24 DCI
[lavc] Using codec: H.265, encoder: libsvt_hevc
[lavc] Selected pixfmt: yuv444p10le

On the receiver side utilizing -d vulkan_sdl2:fs, the pix format for internal codec becomes v210 (10bit YUV 4:2:2) and the display becomes UYVY (8bit YUV 4:2:2) , resulting in signal degradation from the original 10bit YUV 444 (yuv444p10le). Significant banding is apparent. But when used with x265 encoder and the same -d vulkan_sdl2:fs, display is R10k and no banding. (see photo attached)

[2022-12-06 21:26:37] UltraGrid 1.7+ (tags/continuous rev f213c06 built Dec  6 2022 20:02:46)
[2022-12-06 21:26:37] 
[2022-12-06 21:26:37] Display device   : vulkan_sdl2
[2022-12-06 21:26:37] Capture device   : none
[2022-12-06 21:26:37] Audio capture    : none
[2022-12-06 21:26:37] Audio playback   : none
[2022-12-06 21:26:37] MTU              : 9000 B
[2022-12-06 21:26:37] Video compression: none
[2022-12-06 21:26:37] Audio codec      : PCM
[2022-12-06 21:26:37] Network protocol : UltraGrid RTP
[2022-12-06 21:26:37] Audio FEC        : none
[2022-12-06 21:26:37] Video FEC        : none
[2022-12-06 21:26:37] 
[2022-12-06 21:26:37] [SDL] Using driver: x11
[2022-12-06 21:26:37] SDL2 initialized successfully.
[2022-12-06 21:26:37] [VULKAN_SDL2] Path to shaders: "/usr/local/bin/../share/ultragrid/vulkan_shaders"
[2022-12-06 21:26:37] [VULKAN_SDL2] yCbCr feature supported.
[2022-12-06 21:26:37] [VULKAN_SDL2] Vulkan uses GPU called: NVIDIA GeForce GTX 1650
[2022-12-06 21:26:37] [VULKAN_SDL2] Used Vulkan API: 1.1
[2022-12-06 21:26:37] [VULKAN_SDL2] Recreating swapchain, size: 3840x2160, format: A2B10G10R10UnormPack32
[2022-12-06 21:26:37] [VULKAN_SDL2] Vulkan display initialised.
[2022-12-06 21:26:37] [VULKAN_SDL2] Recreating render_pipeline
[2022-12-06 21:26:37] Control socket listening on port 39863
[2022-12-06 21:26:37] [key control] Stdin is not a TTY - disabling keyboard control.
[2022-12-06 21:26:42] [VULKAN_SDL2] 0 frames in 5.01525 seconds = 0 FPS
[2022-12-06 21:26:47] [VULKAN_SDL2] 0 frames in 5.0677 seconds = 0 FPS
[2022-12-06 21:26:52] [VULKAN_SDL2] 0 frames in 5.06779 seconds = 0 FPS
[2022-12-06 21:26:57] [VULKAN_SDL2] 0 frames in 5.05919 seconds = 0 FPS
[2022-12-06 21:27:01] [VULKAN_SDL2] Supported codecs are: RGBA UYVY YUYV Y216 Y416 R10k 
[2022-12-06 21:27:01] [video dec.] New incoming video format detected: 2048x1080 @24.00p, codec H.265
[2022-12-06 21:27:01] [lavd] Using decoder: hevc
2022-12-06 21:27:02] [video dec.] Detected internal codec: v210
[2022-12-06 21:27:02] [video dec.] Successfully reconfigured display to 2048x1080 @24.00p, codec UYVY
[2022-12-06 21:27:04] [lavd] Using decoder: hevc
[2022-12-06 21:27:04] [VULKAN_SDL2] Recreating render_pipeline

Receiver log with an x265 encoder signal:

[2022-12-06 21:38:19] [SDL] Using driver: x11
[2022-12-06 21:38:19] SDL2 initialized successfully.
[2022-12-06 21:38:19] [VULKAN_SDL2] Path to shaders: "/usr/local/bin/../share/ultragrid/vulkan_shaders"
[2022-12-06 21:38:19] [VULKAN_SDL2] yCbCr feature supported.
[2022-12-06 21:38:19] [VULKAN_SDL2] Vulkan uses GPU called: NVIDIA GeForce GTX 1650
[2022-12-06 21:38:19] [VULKAN_SDL2] Used Vulkan API: 1.1
[2022-12-06 21:38:19] [VULKAN_SDL2] Recreating swapchain, size: 3840x2160, format: A2B10G10R10UnormPack32
[2022-12-06 21:38:19] [VULKAN_SDL2] Vulkan display initialised.
[2022-12-06 21:38:19] [VULKAN_SDL2] Recreating render_pipeline
[2022-12-06 21:38:19] Control socket listening on port 41643
[2022-12-06 21:38:19] [key control] Stdin is not a TTY - disabling keyboard control.
[2022-12-06 21:38:24] [VULKAN_SDL2] 0 frames in 5.06031 seconds = 0 FPS
[2022-12-06 21:38:29] [VULKAN_SDL2] 0 frames in 5.04978 seconds = 0 FPS
[2022-12-06 21:38:34] [VULKAN_SDL2] 0 frames in 5.0501 seconds = 0 FPS
[2022-12-06 21:38:39] [VULKAN_SDL2] 0 frames in 5.05029 seconds = 0 FPS
[2022-12-06 21:38:43] [VULKAN_SDL2] Supported codecs are: RGBA UYVY YUYV Y216 Y416 R10k 
[2022-12-06 21:38:43] [video dec.] New incoming video format detected: 2048x1080 @24.00p, codec H.265
[2022-12-06 21:38:43] [lavd] Using decoder: hevc
[2022-12-06 21:38:44] [video dec.] Detected internal codec: R12L
[2022-12-06 21:38:44] [video dec.] Successfully reconfigured display to 2048x1080 @24.00p, codec R10k

I don't believe the banding issue is an inherent difference between SVT and x265 as I am not seeing the same issue when using Decklink out.

IMG_0863

IMG_0864

MartinPulec commented 1 year ago

Confirmed. I've added appropriate ->Y416 and it seems that it works for it now. Vulkan display doesn't support v210, just Y416, which is better here, since it keeps 4:4:4 subsampling. As a side-effect, I suppose that Y416 would be used for GL instead of v210 as well.

Thanks for reporting, the codec negotiation stuff between decompress and display is currently not optimal, but seems to work more or less (if there are appropriate conversions).

alatteri commented 1 year ago

When running the lastest commit, performance is severely degraded and results in un-usable playback. Commit f213c0649f61481fb5af8d4a86c4a020e68fa065 performance is fine, but then the 8bit issue.

Given an input stream of 12bit RGB 444 3G SDI on encoder utilizing -t decklink:codec=R12L -c libavcodec:encoder=libsvt_hevc which uses pixfmt yuv444p10le.

Receiver: uv -d vulkan_sdl2 fs --param use-hw-accel

[2022-12-07 08:19:39] [SDL] Using driver: x11
[2022-12-07 08:19:39] SDL2 initialized successfully.
[2022-12-07 08:19:39] [VULKAN_SDL2] Path to shaders: "/usr/local/bin/../share/ultragrid/vulkan_shaders"
[2022-12-07 08:19:39] [VULKAN_SDL2] yCbCr feature supported.
[2022-12-07 08:19:39] [VULKAN_SDL2] Vulkan uses GPU called: NVIDIA GeForce GTX 1650
[2022-12-07 08:19:39] [VULKAN_SDL2] Used Vulkan API: 1.1
[2022-12-07 08:19:39] [VULKAN_SDL2] Recreating swapchain, size: 3840x2160, format: A2B10G10R10UnormPack32
[2022-12-07 08:19:39] [VULKAN_SDL2] Vulkan display initialised.
[2022-12-07 08:19:39] [VULKAN_SDL2] Recreating render_pipeline
[2022-12-07 08:19:39] Control socket listening on port 42011
[2022-12-07 08:19:39] [key control] Stdin is not a TTY - disabling keyboard control.
[2022-12-07 08:19:44] [VULKAN_SDL2] 0 frames in 5.06114 seconds = 0 FPS
[2022-12-07 08:19:49] [VULKAN_SDL2] 0 frames in 5.04845 seconds = 0 FPS
[2022-12-07 08:19:54] [VULKAN_SDL2] 0 frames in 5.04708 seconds = 0 FPS
[2022-12-07 08:19:59] [VULKAN_SDL2] 0 frames in 5.04714 seconds = 0 FPS
[2022-12-07 08:20:03] [VULKAN_SDL2] Supported codecs are: RGBA UYVY YUYV Y216 Y416 R10k 
[2022-12-07 08:20:03] [video dec.] New incoming video format detected: 3840x2160 @24.00p, codec H.265
[2022-12-07 08:20:03] [lavd] Using decoder: hevc
[2022-12-07 08:20:03] [lavc hevc @ 0x7f267c006cc0] PPS id out of range: 0

[2022-12-07 08:20:04] [lavc hevc @ 0x7f267c006cc0] A hardware frames or device context is required for hardware accelerated decoding.
[2022-12-07 08:20:04] [lavc hevc @ 0x7f267c006cc0] Failed setup for format vdpau: hwaccel initialisation returned error.
[2022-12-07 08:20:04] [video dec.] Detected internal codec: Y416
[2022-12-07 08:20:04] [video dec.] Detected internal codec: Y416
[2022-12-07 08:20:04] [video dec.] Detected internal codec: Y416
[2022-12-07 08:20:06] [video dec.] Successfully reconfigured display to 3840x2160 @24.00p, codec Y416
[2022-12-07 08:20:06] [lavd] Using decoder: hevc
[2022-12-07 08:20:06] [video dec.] Successfully reconfigured display to 3840x2160 @24.00p, codec Y416
[2022-12-07 08:20:06] [lavd] Using decoder: hevc
[2022-12-07 08:20:06] [video dec.] Successfully reconfigured display to 3840x2160 @24.00p, codec Y416
[2022-12-07 08:20:06] [lavd] Using decoder: hevc
[2022-12-07 08:20:06] [video dec.] Successfully reconfigured display to 3840x2160 @24.
[2022-12-07 08:20:06] [lavd] Hw. acceleration requested but incoming video has not 4:2:0 subsampling, which is usually not supported by hw. accelerators
[2022-12-07 08:20:06] [VULKAN_SDL2] 0 frames in 6.58108 seconds = 0 FPS
[2022-12-07 08:20:06] libva info: VA-API version 1.16.0
[2022-12-07 08:20:06] libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/iHD_drv_video.so
[2022-12-07 08:20:06] libva info: Found init function __vaDriverInit_1_15
[2022-12-07 08:20:06] libva info: va_openDriver() returns 0
[2022-12-07 08:20:06] [lavc AVHWFramesContext @ 0x7f2670ff50c0] Error allocating a VDPAU video surface
[2022-12-07 08:20:06] [hw accel] Unable to init hwframe_ctx: Cannot allocate memory
[2022-12-07 08:20:06] 
[2022-12-07 08:20:06] [lavd] Using swscale to convert from yuv444p16le to yuv444p10le.
[2022-12-07 08:20:06] [VULKAN_SDL2] Recreating render_pipeline
MartinPulec commented 1 year ago

Well, but you didn't say that you will be using --param use-hw-accel - it finally ends at a slightly different pixel format, it is a different story ;-) Could you try it out now? If I didn't make a mistake, it should work. (I cannot test easily now - libx265 is willing to produce at most 12-bit YCbCr and I use SW decoder.)

alatteri commented 1 year ago

Yup...fixed. Thanks!