Vintage story fails to capture mouse on wayland

[fazo96]

Describe the bug

the game Vintage Story can't capture the mouse, causing it to be unplayable.

Steps To Reproduce

Steps to reproduce the behavior:

1. use a sway or hyprland session
2. run vintagestory from the vintagestory package
3. enter a game and move the camera around - this step requires having a copy of the game as it is paid proprietary software

Expected behavior

The camera moves around correctly like any other first person game. Instead, due to the bug, the mouse stays visible on the screen and the camera jerks around.

Additional context

I thought this was an upstream bug, but then I noticed the game is running as an xwayland client even though the developers says it supports wayland. It seems the game has SDL2 has a dependency on nixpkgs so it might be using SDL. I tried setting SDL_VIDEODRIVER=wayland but it still runs as an XWayland client.

It might also be a wlroots bug. The bug has been seen on hyprland and sway, so it is unlikely to be a bug in both WMs (or maybe I am wrong).

Maybe something is wrong in the packaging, and running it as a native wayland client would fix the issue.

The bug was reported upstream by @neirenoir here: https://github.com/anegostudios/VintageStory-Issues/issues/3152

Notify maintainers

@GiggleSquid @Artturin

Metadata

Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

 - system: `"x86_64-linux"`
 - host os: `Linux 6.6.0, NixOS, 23.11 (Tapir), 23.11pre544052.85f1ba3e5167`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.17.1`
 - channels(root): `"home-manager, nixos"`
 - channels(fazo): `""`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`

[GiggleSquid]

I can confirm this issue on Hyprland. I had completely overlooked this in testing.

I'll have a look into this as soon as I have the time, thanks for making us aware!

[GiggleSquid]

As a slight temporary remedy, I suggest enabling direct mouse mode in the settings to alleviate the jerky camera movement. This should(TM) make the experience at least playable.

As far as I can tell, from cursory testing, the cursor is constrained to the game window, but just isn't hidden or zeroed (at least in Hyprland) This should be working correctly (constrained and hidden) in most games since Hyprland v0.30.0.

I'm unsure if Vintagestory actually natively supports Wayland, hence why it runs in an xwayland window, but this may be why Anego Studios and/or user reports says it supports wayland. In any case I personally can't seem to get it to run natively in wayland. If anyone can report if they have vintagestory running natively in wayland to prove me wrong that would be handy in redirecting my debugging

[fazo96]

thanks @GiggleSquid unfortunately that didn't work for me, with direct mouse mode the camera just spins endlessly. Last night I experimented a bit and got it to work using gamescope, specifically the option to force grab the mouse cursor

Using that and non-direct mouse mode, the game is fully playable, the mouse behaves normally however the cursor stays visible during gameplay while moving the camera.

[fazo96]

I tried again and with Direct mouse mode enabled in the game settings (no gamescope), the game is playable and the only issue is the mouse cursor remains visible when it should not be

Not sure what changed but the game hasn't been updated since my last attempt so I guess something else got updated and works better with it.

[neirenoir]

Can confirm this is the exact behavior I am getting.

At least the game is playable, but it still would be nice to know why is this happening in order to fix the visible cursor.

[ImUrX]

it happens on KDE and Gnome wayland too (nixOS), the AUR package does not replicate this behavior in Sway

[QuoteNat]

This is happening for me on x11 gnome as well with the current unstable channel version (1.19.8). The flatpak however does seem to work for me in terms of capturing the cursor.

[fazo96]

I tried the flatpak too after reading @QuoteNat's comment and it does fix the issue. So it looks like it's not a bug in the game but some kind of packaging/library issue.

[Artturin]

diff --git a/pkgs/games/vintagestory/default.nix b/pkgs/games/vintagestory/default.nix
index b32417d7754b..36be4cb099cb 100644
--- a/pkgs/games/vintagestory/default.nix
+++ b/pkgs/games/vintagestory/default.nix
@@ -4,6 +4,7 @@
 , makeWrapper
 , makeDesktopItem
 , copyDesktopItems
+, autoPatchelfHook
 , xorg
 , gtk2
 , sqlite
@@ -15,6 +16,7 @@
 , libglvnd
 , pipewire
 , libpulseaudio
+, wayland
 , dotnet-runtime_7
 }:

@@ -28,9 +30,15 @@ stdenv.mkDerivation rec {
   };

-  nativeBuildInputs = [ makeWrapper copyDesktopItems ];
+  nativeBuildInputs = [ makeWrapper copyDesktopItems autoPatchelfHook ];

-  buildInputs = [ dotnet-runtime_7 ];
+  buildInputs = [
+    dotnet-runtime_7
+
+    wayland
+    xorg.libX11
+    stdenv.cc.cc.lib
+  ];

   runtimeLibs = lib.makeLibraryPath ([
     gtk2

Does not fix it

[Mohaim-1]

I've found the solution! I added the following to my config and now the cursor disappears when it should, the cursor changes to the ingame cursor instead of the system cursor, and mouselook works without enabling "Direct mouse mode".

nixpkgs.overlays = [ (final: prev: {
  vintagestory = prev.vintagestory.overrideAttrs (old: {
    preFixup = builtins.replaceStrings
      ["--prefix LD_LIBRARY_PATH"]
      ["--set LD_PRELOAD ${final.xorg.libXcursor}/lib/libXcursor.so.1 --prefix LD_LIBRARY_PATH"]
      old.preFixup;
  });
})];

[Artturin]

I've found the solution! I added the following to my config and now the cursor disappears when it should, the cursor changes to the ingame cursor instead of the system cursor, and mouselook works without enabling "Direct mouse mode".

nixpkgs.overlays = [ (final: prev: {
  vintagestory = prev.vintagestory.overrideAttrs (old: {
    preFixup = builtins.replaceStrings
      ["--prefix LD_LIBRARY_PATH"]
      ["--set LD_PRELOAD ${final.xorg.libXcursor}/lib/libXcursor.so.1 --prefix LD_LIBRARY_PATH"]
      old.preFixup;
  });
})];

https://github.com/NixOS/nixpkgs/pull/349063