search

obsproject / obs-studio

OBS Studio - Free and open source software for live streaming and screen recording
https://obsproject.com
GNU General Public License v2.0
58.82k stars 7.83k forks source link

[BUG] Crash on 10bpc screen #2654

Open dan-dex opened 4 years ago

dan-dex commented 4 years ago

Platform

Operating system and version: Gentoo Stable kernel-5.5.13

details

``` app-shells/bash: 4.4_p23-r1::gentoo dev-java/java-config: 2.2.0-r4::gentoo dev-lang/perl: 5.30.1::gentoo dev-lang/python: 2.7.17-r1::gentoo, 3.6.10::gentoo, 3.7.7::gentoo dev-util/cmake: 3.16.5::gentoo dev-util/pkgconfig: 0.29.2::gentoo sys-apps/baselayout: 2.6-r1::gentoo sys-apps/openrc: 0.42.1::gentoo sys-apps/sandbox: 2.13::gentoo sys-devel/autoconf: 2.13-r1::gentoo, 2.69-r4::gentoo sys-devel/automake: 1.16.1-r1::gentoo sys-devel/binutils: 2.33.1-r1::gentoo sys-devel/gcc: 9.2.0-r2::gentoo sys-devel/gcc-config: 2.2.1::gentoo sys-devel/libtool: 2.4.6-r6::gentoo sys-devel/make: 4.2.1-r4::gentoo sys-kernel/linux-headers: 5.5::gentoo (virtual/os-headers) sys-libs/glibc: 2.29-r7::gentoo ```

OBS Studio version: OBS Studio - 25.0.1

Expected Behavior

Not to crash.

Current Behavior

Application crashes.

log

``` Attempted path: share/obs/obs-studio/locale/en-US.ini Attempted path: /usr/share/obs/obs-studio/locale/en-US.ini Attempted path: share/obs/obs-studio/locale.ini Attempted path: /usr/share/obs/obs-studio/locale.ini Attempted path: share/obs/obs-studio/themes/Dark.qss Attempted path: /usr/share/obs/obs-studio/themes/Dark.qss info: CPU Name: AMD Ryzen 9 3950X 16-Core Processor info: CPU Speed: 4066.386MHz info: Physical Cores: 16, Logical Cores: 32 info: Physical Memory: 32090MB Total, 25197MB Free info: Kernel Version: Linux 5.5.13-gentoo info: Distribution: Gentoo Unknown info: Window System: X11.0, Vendor: The X.Org Foundation, Version: 1.20.7 info: Portable mode: false Attempted path: share/obs/obs-studio/themes/Dark/no_sources.svg Attempted path: /usr/share/obs/obs-studio/themes/Dark/no_sources.svg QMetaObject::connectSlotsByName: No matching signal for on_advAudioProps_clicked() QMetaObject::connectSlotsByName: No matching signal for on_advAudioProps_destroyed() QMetaObject::connectSlotsByName: No matching signal for on_actionGridMode_triggered() QMetaObject::connectSlotsByName: No matching signal for on_program_customContextMenuRequested(QPoint) info: OBS 25.0.1 (linux) info: --------------------------------- info: --------------------------------- info: audio settings reset: samples per sec: 44100 speakers: 2 info: --------------------------------- info: Initializing OpenGL... info: Loading up OpenGL on adapter X.Org Radeon Vega Frontier Edition (VEGA10, DRM 3.36.0, 5.5.13-gentoo, LLVM 9.0.1) info: OpenGL loaded successfully, version 4.6 (Core Profile) Mesa 20.0.3, shading language 4.60 info: --------------------------------- info: video settings reset: base resolution: 1920x1080 output resolution: 1280x720 downscale filter: Bicubic fps: 30/1 format: NV12 YUV mode: 601/Partial info: NV12 texture support not available info: --------------------------------- warning: Failed to load 'en-US' text for module: 'decklink-ouput-ui.so' libDeckLinkAPI.so: cannot open shared object file: No such file or directory warning: A DeckLink iterator could not be created. The DeckLink drivers may not be installed info: No blackmagic support info: FFMPEG VAAPI supported info: --------------------------------- info: Loaded Modules: info: text-freetype2.so info: rtmp-services.so info: obs-x264.so info: obs-transitions.so info: obs-outputs.so info: obs-libfdk.so info: obs-filters.so info: obs-ffmpeg.so info: linux-jack.so info: linux-decklink.so info: linux-capture.so info: linux-alsa.so info: image-source.so info: frontend-tools.so info: decklink-ouput-ui.so info: --------------------------------- info: ==== Startup complete =============================================== error: Service '' not found info: All scene data cleared info: ------------------------------------------------ info: xshm-input: Geometry 5120x2880 @ 0,0 info: Switched to scene 'Scene' info: ------------------------------------------------ info: Loaded scenes: info: - scene 'Scene': info: - source: 'Screen Capture (XSHM)' (xshm_input) info: ------------------------------------------------ libGL error: failed to create drawable libGL error: failed to create drawable error: X Error: GLXBadContext, Major opcode: 151, Minor opcode: 26, Serial: 34 error: Failed to make context current. error: Failed to fetch parent window geometry! Attempted path: share/obs/obs-studio/images/overflow.png Attempted path: /usr/share/obs/obs-studio/images/overflow.png error: Failed to fetch parent window geometry! error: Failed to fetch parent window geometry! error: Failed to fetch parent window geometry! error: X Error: GLXBadDrawable, Major opcode: 151, Minor opcode: BadAlloc (insufficient resources for operation), Serial: 52 libGL error: failed to create drawable libGL error: failed to create drawable error: X Error: GLXBadContext, Major opcode: 151, Minor opcode: 26, Serial: 52 error: Failed to make context current. libGL error: failed to create drawable libGL error: failed to create drawable error: X Error: GLXBadContext, Major opcode: 151, Minor opcode: 26, Serial: 52 error: Failed to make context current. Segmentation fault ```

Steps to Reproduce

  1. Have xorg with 30bit depth 
    Section "Screen"
Identifier "Screen1"
DefaultDepth 30
EndSection
  2. Launch obs

Additional information

Actually I tried to create some quick fix for this configuration. Firstly, change of some hardcode allowed me to start the app.

code

``` diff -ruN orig/libobs-opengl/gl-x11.c obs-studio-25.0.1/libobs-opengl/gl-x11.c --- orig/libobs-opengl/gl-x11.c 2020-03-19 18:51:06.000000000 +0200 +++ obs-studio-25.0.1/libobs-opengl/gl-x11.c 2020-04-05 17:15:28.743653276 +0300 @@ -64,7 +64,7 @@ GLX_BUFFER_SIZE, 32, GLX_ALPHA_SIZE, - 8, + 2, GLX_DOUBLEBUFFER, true, GLX_X_RENDERABLE, @@ -406,7 +406,7 @@ xcb_create_colormap(xcb_conn, XCB_COLORMAP_ALLOC_NONE, colormap, parent, visual); - xcb_create_window(xcb_conn, 24 /* Hardcoded? */, wid, parent, 0, 0, + xcb_create_window(xcb_conn, 30 /* Hardcoded? */, wid, parent, 0, 0, geometry->width, geometry->height, 0, 0, visual, mask, mask_values); ```

But as expected colors was corrupted. So I decided to create filter that just converts 2101010 to 8888 as the quick way to fix this.

full patch

``` diff -ruN orig/libobs-opengl/gl-x11.c obs-studio-25.0.1/libobs-opengl/gl-x11.c --- orig/libobs-opengl/gl-x11.c 2020-03-19 18:51:06.000000000 +0200 +++ obs-studio-25.0.1/libobs-opengl/gl-x11.c 2020-04-05 17:15:28.743653276 +0300 @@ -64,7 +64,7 @@ GLX_BUFFER_SIZE, 32, GLX_ALPHA_SIZE, - 8, + 2, GLX_DOUBLEBUFFER, true, GLX_X_RENDERABLE, @@ -406,7 +406,7 @@ xcb_create_colormap(xcb_conn, XCB_COLORMAP_ALLOC_NONE, colormap, parent, visual); - xcb_create_window(xcb_conn, 24 /* Hardcoded? */, wid, parent, 0, 0, + xcb_create_window(xcb_conn, 30 /* Hardcoded? */, wid, parent, 0, 0, geometry->width, geometry->height, 0, 0, visual, mask, mask_values); diff -ruN orig/plugins/obs-filters/CMakeLists.txt obs-studio-25.0.1/plugins/obs-filters/CMakeLists.txt --- orig/plugins/obs-filters/CMakeLists.txt 2020-03-19 18:51:06.000000000 +0200 +++ obs-studio-25.0.1/plugins/obs-filters/CMakeLists.txt 2020-04-05 00:21:42.242268677 +0300 @@ -35,6 +35,7 @@ set(obs-filters_SOURCES obs-filters.c + color-10bpc-workaround-filter.c color-correction-filter.c async-delay-filter.c gpu-delay.c diff -ruN orig/plugins/obs-filters/color-10bpc-workaround-filter.c obs-studio-25.0.1/plugins/obs-filters/color-10bpc-workaround-filter.c --- orig/plugins/obs-filters/color-10bpc-workaround-filter.c 1970-01-01 03:00:00.000000000 +0300 +++ obs-studio-25.0.1/plugins/obs-filters/color-10bpc-workaround-filter.c 2020-04-05 03:59:22.168879746 +0300 @@ -0,0 +1,100 @@ +#include +#include +#include +#include + +/* clang-format off */ + +/* clang-format on */ + +struct color_10bpc_workaround_filter_data { + obs_source_t *context; + + gs_effect_t *effect; +}; + +static const char *color_10bpc_workaround_name(void *unused) +{ + UNUSED_PARAMETER(unused); + return obs_module_text("Color10bpcWorkaroundFilter"); +} + +static void color_10bpc_workaround_update(void *data, obs_data_t *settings) +{ +} + +static void color_10bpc_workaround_destroy(void *data) +{ + struct color_10bpc_workaround_filter_data *filter = data; + + if (filter->effect) { + obs_enter_graphics(); + gs_effect_destroy(filter->effect); + obs_leave_graphics(); + } + + bfree(data); +} + +static void *color_10bpc_workaround_create(obs_data_t *settings, obs_source_t *context) +{ + struct color_10bpc_workaround_filter_data *filter = + bzalloc(sizeof(struct color_10bpc_workaround_filter_data)); + char *effect_path = obs_module_file("color_10bpc_workaround_filter.effect"); + + filter->context = context; + + obs_enter_graphics(); + + filter->effect = gs_effect_create_from_file(effect_path, NULL); + + obs_leave_graphics(); + + bfree(effect_path); + + if (!filter->effect) { + color_10bpc_workaround_destroy(filter); + return NULL; + } + + color_10bpc_workaround_update(filter, settings); + return filter; +} + +static void color_10bpc_workaround_render(void *data, gs_effect_t *effect) +{ + struct color_10bpc_workaround_filter_data *filter = data; + + if (!obs_source_process_filter_begin(filter->context, GS_RGBA, + OBS_ALLOW_DIRECT_RENDERING)) + return; + + obs_source_process_filter_end(filter->context, filter->effect, 0, 0); + + UNUSED_PARAMETER(effect); +} + +static obs_properties_t *color_10bpc_workaround_properties(void *data) +{ + obs_properties_t *props = obs_properties_create(); + + UNUSED_PARAMETER(data); + return props; +} + +static void color_10bpc_workaround_defaults(obs_data_t *settings) +{ +} + +struct obs_source_info color_10bpc_workaround_filter = { + .id = "color_10bpc_workaround_filter", + .type = OBS_SOURCE_TYPE_FILTER, + .output_flags = OBS_SOURCE_VIDEO, + .get_name = color_10bpc_workaround_name, + .create = color_10bpc_workaround_create, + .destroy = color_10bpc_workaround_destroy, + .video_render = color_10bpc_workaround_render, + .update = color_10bpc_workaround_update, + .get_properties = color_10bpc_workaround_properties, + .get_defaults = color_10bpc_workaround_defaults, +}; diff -ruN orig/plugins/obs-filters/data/color_10bpc_workaround_filter.effect obs-studio-25.0.1/plugins/obs-filters/data/color_10bpc_workaround_filter.effect --- orig/plugins/obs-filters/data/color_10bpc_workaround_filter.effect 1970-01-01 03:00:00.000000000 +0300 +++ obs-studio-25.0.1/plugins/obs-filters/data/color_10bpc_workaround_filter.effect 2020-04-05 05:13:28.852087805 +0300 @@ -0,0 +1,59 @@ +uniform float4x4 ViewProj; +uniform texture2d image; + +sampler_state textureSampler { + Filter = Linear; + AddressU = Clamp; + AddressV = Clamp; +}; + +struct VertData { + float4 pos : POSITION; + float2 uv : TEXCOORD0; +}; + +VertData VSDefault(VertData v_in) +{ + VertData vert_out; + vert_out.pos = mul(float4(v_in.pos.xyz, 1.0), ViewProj); + vert_out.uv = v_in.uv; + return vert_out; +} + +float4 ProcessColorKey(float4 rgba, VertData v_in) +{ + float b = rgba.b * 255.0f; + float g = rgba.g * 255.0f; + float r = rgba.r * 255.0f; + float a = rgba.a * 255.0f; + + float bPartInG = mod(g, 4); + b += bPartInG * 256.0f; + + g /= 4; + float gPartInR = mod(r, 16); + g += gPartInR * 64.0f; + + r /= 16; + float rPartInA = mod(a, 64); + r += rPartInA * 16.0f; + + a /= 64; + + return float4(r/1023.0f, g/1023.0f, b/1023.0f, a/3.0f); +} + +float4 PSColorKeyRGBA(VertData v_in) : TARGET +{ + float4 rgba = image.Sample(textureSampler, v_in.uv); + return ProcessColorKey(rgba, v_in); +} + +technique Draw +{ + pass + { + vertex_shader = VSDefault(v_in); + pixel_shader = PSColorKeyRGBA(v_in); + } +} diff -ruN orig/plugins/obs-filters/data/locale/en-US.ini obs-studio-25.0.1/plugins/obs-filters/data/locale/en-US.ini --- orig/plugins/obs-filters/data/locale/en-US.ini 2020-03-19 18:51:06.000000000 +0200 +++ obs-studio-25.0.1/plugins/obs-filters/data/locale/en-US.ini 2020-04-05 00:30:51.639294383 +0300 @@ -6,6 +6,7 @@ ScrollFilter="Scroll" ChromaKeyFilter="Chroma Key" ColorKeyFilter="Color Key" +Color10bpcWorkaroundFilter="Color 10bpc Workaround" SharpnessFilter="Sharpen" Sharpness="Sharpness" ScaleFilter="Scaling/Aspect Ratio" diff -ruN orig/plugins/obs-filters/obs-filters.c obs-studio-25.0.1/plugins/obs-filters/obs-filters.c --- orig/plugins/obs-filters/obs-filters.c 2020-03-19 18:51:06.000000000 +0200 +++ obs-studio-25.0.1/plugins/obs-filters/obs-filters.c 2020-04-05 00:35:39.535307853 +0300 @@ -16,6 +16,7 @@ extern struct obs_source_info scroll_filter; extern struct obs_source_info gpu_delay_filter; extern struct obs_source_info color_key_filter; +extern struct obs_source_info color_10bpc_workaround_filter; extern struct obs_source_info color_grade_filter; extern struct obs_source_info sharpness_filter; extern struct obs_source_info chroma_key_filter; @@ -40,6 +41,7 @@ obs_register_source(&scroll_filter); obs_register_source(&gpu_delay_filter); obs_register_source(&color_key_filter); + obs_register_source(&color_10bpc_workaround_filter); obs_register_source(&color_grade_filter); obs_register_source(&sharpness_filter); obs_register_source(&chroma_key_filter); ```

But problems came from unexpected side. Everything is ok with r,g,b channels in the input texture, but alpha-channel have weird data. And as we have 6 bits of r-channel in a-channel, resulting r-channel is broken as well.

notr1ch commented 4 years ago

This seems very similar to https://github.com/obsproject/obs-studio/issues/2645

kkartaltepe commented 4 years ago

I cannot replicate your crash when using a 30bit depth xorg as you outline (instead obs fails to start, maybe more expected).

Please provide the full output of glxinfo when running in a 30bit depth xorg.

dan-dex commented 4 years ago

glxinfo.txt

kkartaltepe commented 4 years ago

Thanks looks like your driver provides a compatibility visual for 8 bit RGBA in 30bit depth x11 so OBS is able to load on that platform.

dan-dex commented 4 years ago

Yes, just checked, OBS starts when I delete screen capture sources. But adding one back triggers that crash.

kkartaltepe commented 4 years ago

Currently OBS depends on praying that the context we create headlessly works with whatever context is chosen by qt for displaying windows. Until we stop praying and make sure this context is compatible or rebuild graphics objects in the new context after swapchain init I dont think is solvable.

--- edit But im not very wise about graphics so there may be an alternative solution im not aware of.

dan-dex commented 4 years ago

So, second try to find what's wrong:) Actually not so better that previous, but at least it provide correct colors) There is something wrong with opengl part in frame processing chain that alpha is gone, from xorg we receive correct colors. So after slow cpu-based conversion we have correct image. Here is the patch attached, maybe it will be helpful in case someone have the same issue. 10bpc_patch.patch.txt

RytoEX commented 1 year ago

Please confirm if this is still an issue in OBS Studio 28.0.2.

dan-dex commented 1 year ago

Hi! It's almost there:)

I have this in log: ``` error: Swapchain window creation failed: BadMatch (invalid parameter attributes), Major opcode: 1, Minor opcode: 0 error: gl_platform_init_swapchain failed error: obs_display_init: Failed to create swap chain ```

and preview is not working.

After these changes: ``` --- obs-studio-28.0.2/libobs-opengl/gl-x11-egl.c 2022-09-22 19:40:51.000000000 +0300 +++ obs-studio-28.0.2-fix/libobs-opengl/gl-x11-egl.c 2022-09-23 01:24:20.374991786 +0300 @@ -58,7 +58,7 @@ EGL_DEPTH_SIZE, 0, EGL_BUFFER_SIZE, - 24, + 32, EGL_ALPHA_SIZE, 0, EGL_RENDERABLE_TYPE, @@ -417,7 +417,7 @@ visual); xcb_void_cookie_t window_cookie = xcb_create_window_checked( - xcb_conn, 24 /* Hardcoded? */, wid, parent, 0, 0, + xcb_conn, 30 /* Hardcoded? */, wid, parent, 0, 0, geometry->width, geometry->height, 0, 0, visual, mask, mask_values); xcb_generic_error_t *err = xcb_request_check(xcb_conn, window_cookie); ```

Window Capture(Xcomposite):

I can see preview(main window) but in dim colors(maybe I forget about some filter?) Small previews in Properties/Filters windows have correct colors. In screenshot of the scene or in recording everything is ok with colors as well.

Screen Capture(XSHM):

Colors corrupted everywhere.

kkartaltepe commented 1 year ago

xshm would need to either convert the captured pixel data into 32bit format (i think pixmaps are 30bits?) and upload to a GS_R10G10B10A2 format texture instead of the current GS_GBRA. So you could probably try patching that and see if it works.

But yea this probably hasnt changed much since it was reported. Except that we now support 10bit textures.

dan-dex commented 1 year ago

10bit textures is already a good thing. At least in my case it became usable with 30bit config:) Yes, with little patch but usable though(in my previous attempts it was either with corrupted colors or critically slow)

So, thanks for your work!:)