obsproject / obs-studio

OBS Studio - Free and open source software for live streaming and screen recording
https://obsproject.com
GNU General Public License v2.0
60.21k stars 7.97k forks source link

BadWindow (invalid Window parameter) on Xcomposite capture for steam #5917

Closed nicokimmel closed 2 years ago

nicokimmel commented 2 years ago

Operating System Info

Ubuntu 20.04

Other OS

No response

OBS Studio Version

27.2.0 RC 1

OBS Studio Version (Other)

Tried with different versions.

OBS Studio Log URL

https://obsproject.com/logs/QiiL5jkZQ0ZntfK9

OBS Studio Crash Log URL

No response

Expected Behavior

Window should be captured.

Current Behavior

It only shows a black screen and the cursor.

Steps to Reproduce

  1. Open OBS
  2. Add window capture (xcomposite)
  3. Select application
  4. See the black box + cursor in preview

(https://www.youtube.com/watch?v=GUXJuLssKw0)

Anything else we should know?

I tried the following things:

Problem does happen for Steam (native, not via Wine), Wine/Proton apps, Nautilus and some other random(?) applications but not for OBS itself, Discord, Spotify and some more.

kkartaltepe commented 2 years ago

Please provide an additional log where you do the same thing but for a non-steam window that also triggers this issue please (and tell me what the program is that triggers the issue). Also please add your steam "interface" settings (is hardware acceleration enabled in steam?)

nicokimmel commented 2 years ago

https://obsproject.com/logs/evURvh2YBLJmYXLp

I disabled hardware acceleration in Steam, same problem. In the log I added Steam, Nautilus and then Spotify. Spotify is working even it shows the same error in the log file.

kkartaltepe commented 2 years ago

https://obsproject.com/logs/MwvNv0ARqII9E568 another similar looking log.

Specimen197 commented 2 years ago

https://obsproject.com/logs/MwvNv0ARqII9E568 another similar looking log.

This looks like the log I posted on the forums. Im not sure if this bit of information will help, but a user on the OBS discord noticed that as long as the window we're trying to record (In Gnome 40.4.0) has a title bar, OBS will allow capturing of the window. This goes for games and software applications.

https://imgur.com/Zz53oBc

spapadim commented 2 years ago

For anyone else searching: I believe client-side decorations (e.g., anything GTK-based, or Chromium/Electron-based) breaking window capture on OBS is a known issue for years (at least under GNOME). The workaround of disabling CSD (at least temporarily) may be a little cumbersome, but works. It used to be the case that resizing window made capture work at random (ie success/fail wasn't deterministic function of window size), but that no longer seems to be the case. I was quite hopeful about recent commits that fix some X11/XComposite issues, but sadly didn't help in my case.

I did try to dig around a bit further, but being mostly clueless about both OpenGL and X11 APIs, I didn't get very far... if I remember right, just noticed that (a) the GNOME compositor allocates a pixmap (texture?) that is "dilated" by 20px (if I remember right) wrt visible window size, for some reason I didn't understand, and (b) whenever bug occurs, the GLX texture allocated by OBS has size totally unrelated to the window (eg sth like just 10 pixels wide), whereas when it worked the sizes matched. Also, I think (totally subjective impression) that when I added a ton of logging statements, the frequency of random successes went up (i.e., it could be sth timing/contention-related). Since Chrome HTML5 capture works, took a peek at their code, but it seems they just don't use XComposite (forget details, but was sth far more CPU-intensive?). FWIW, I'm on Gnome 3.36, with NVidia drivers (I think I also tried Nouveau at some point, without luck, but not 100% sure.. I could try again, if anyone thinks it relevant, but I doubt..?). Decided not to "spam" with yet another log that's basically identical to those in multiple issue reports over the years.

Quite the Heisenbug, it seems. :) Just my 2c "coredump" -- good luck!

kkartaltepe commented 2 years ago

this has been root caused to nvidia drivers not supporting multiple glx pixmaps for a single window. And gnome being the only compositor that captures windows via glx pixmaps. Because of this windows that are open before obs typically wont be captured unless gnome decided not to capture via glx pixmaps for whatever reason.

Closing as wont fix because we cannot modify nvidia drivers. If anyone is interested in implementing a shared memory capture for windows thats probably the only solution but is not performant.