Test on Windows with WSL2

ABeltramo commented 3 years ago

WSL2 is missing the uinput kernel module. There's a way to get it which involves recompile the kernel, try following the instructions at https://gist.github.com/cerebrate/d40c89d3fa89594e1b1538b2ce9d2720

TheSamDickey commented 3 years ago

Hello! I stumbled across this repo today and it looks very interesting. I am curious, would HW acceleration work with WSL?

ABeltramo commented 3 years ago

That's a very good question to which I don't have an answer. I'm not even sure that WSL will expose the underlying GPU like you do on Linux via /dev/dri.

We are currently focusing our efforts on Linux, to be fair, the more I think about running GOW on Windows, the more I'm convinced it's not worth the effort compared to just running the .exe

ABeltramo commented 3 years ago

It might be worth to investigate on https://github.com/microsoft/wslg

Poprox198 commented 2 years ago

There is a published nvidia guide for WSL2 and docker : https://docs.nvidia.com/cuda/wsl-user-guide/index.html

jfrgagnon commented 1 year ago

Did anyone retried to implement GOW with WSL2 since the addition of systemd and the 2022 new kernel features?

ABeltramo commented 1 year ago

Did anyone retried to implement GOW with WSL2 since the addition of systemd and the 2022 new kernel features?

AFAIK no, and to be fair, I'm not sure I see the point; as things are right now Sunshine works perfectly well natively on Windows, and why would you run games in a Docker container via proton or the likes when you can just run them natively on the host?

I'm open to suggestions and POW, but right now, I'm focusing all my efforts in Linux.

lostmsu commented 1 year ago

Just tried on my WSL 2 Ubuntu 20.04 install + WSLg + CUDA 11.4 which I use for ML.

Headless

Fails to start X11. The suggested hack with EDID did not work. I still get Fatal server error: no screens found.

Host

I installed and ensured GEdit works.

Launching Steam container actually shows Steam login window in Windows, but Sunshine fails to start:

Cannot load libcuda.so.1
terminate called after throwing an instance of
'std::filesystem::__cxx11::filesystem_error'
   what():  filesystem error: directory iterator
     cannot open directory: No such file or directory [/dev/dri]
Error: Couldn't load cuda: -1
Warning: Couldn't find /dev/dri, kmsgrab won't be enabled
gow-sunshine-1 exited with code 139

Windows

why would you run games in a Docker container via proton or the likes when you can just run them natively on the host?

Two goals really: containerization and running headless (+ as a bonus in arbitrary resolutions and multiple games at once from the same host?)

lostmsu commented 1 year ago

After updating wsl with wsl --update, /dev/dri is now available and so Sunshine starts, and I can access the web UI, but Moonlight still can not connect to it. sunshine-wsl2.log

The relevant bits appear to be:

Couldn't expose some/all drm planes for card: /dev/dri/card0
...
Error: Could not create Sunshine Mouse: No such file or directory
Error: Could not create Sunshine Touchscreen: No such file or directory
Error: Could not create Sunshine Keyboard: No such file or directory
Error: Unable to create any input devices! Are you a member of the 'input' group?
...
Info: Screencasting with X11
Error: Couldn't load cuda: -1
Info: Encoder [nvenc] failed
Cannot load libcuda.so.1
...
Info: Trying encoder [vaapi]
Info: Screencasting with X11
MESA-LOADER: failed to open vgem: /usr/lib/dri/vgem_dri.so: cannot open shared object file: No such file or directory (search paths /usr/lib/x86_64-linux-gnu/dri:\$${ORIGIN}/dri:/usr/lib/dri, suffix _dri)
failed to load driver: vgem
libva info: VA-API version 1.14.0
Info: SDR color coding [Rec. 601]
Info: Color range: [JPEG]
Info: vaGetDriverNameByIndex() failed with unknown libva error, driver_name = (null)
Error: Couldn't initialize va display: unknown libva error

Moonlight complains about access to UDP 47999, which I take it means Sunshine failed to start the control stream.

ABeltramo commented 1 year ago

Thanks for reporting this, by a quick look at the logs I think you are hitting multiple issues:

Issues

Error: Could not create Sunshine Mouse is uinput present in the WSL kernel? (You can check with: lsmod | grep uinput) When I originally opened this issue it wasn't present, I'm not sure if that's changed now. Without uinput you'll not be able to create virtual devices and that means that you'll not be able to control the stream remotely.
Cannot load libcuda.so.1, failed to load driver: vgem, Couldn't initialize va display: unknown libva error looks like you have issues with your video drivers, which GPU are you using? Can you try running vainfo and see what's reported?

Goals

Two goals really: containerization and running headless (+ as a bonus in arbitrary resolutions and multiple games at once from the same host?)

Sunshine supports multiple users sharing a single screen (ex: co-op games). We are building a completely different solution in order to support headless virtual HW accelerated displays for multiple users which is called Wolf. It's still under heavy development and not really ready to be used as is but, since we are building our own Wayland compositor, this could potentially work under WSL.

I can only say, keep an eye on this space because we are getting close to something that looks like a solution. 😅

lostmsu commented 1 year ago

Can you try running vainfo and see what's reported?
libva info: VA-API version 1.7.0
libva error: vaGetDriverNameByIndex() failed with invalid VADisplay, driver_name = (null)
vaInitialize failed with error code 3 (invalid VADisplay),exit
looks like you have issues with your video drivers Perhaps I need to upgrade my mesa as described here: https://devblogs.microsoft.com/commandline/d3d12-gpu-video-acceleration-in-the-windows-subsystem-for-linux-now-available/

KeenMaron commented 3 months ago

Any updates with WSL2? I tried the guide for Nvidia but got several issues to run it on ubunutu WSL2 from windows

ABeltramo commented 3 months ago

Last I've checked WSL2 was still missing both /dev/uinput and /dev/uhid this means that the current implementation of virtual inputs wouldn't work in it.

I'm going to implement https://github.com/games-on-whales/gst-wayland-display/issues/4 soon, with that, at least mouse and keyboard should work. Not sure what's the status of HW acceleration in WSL is, but hopefully that should work.

Stay tuned!

KeenMaron commented 3 months ago

Thanks, it would be awesome to make it run through the ubuntu subsystem of windows 10 :)

ABeltramo commented 3 months ago

Anyone willing to test out the latest on WSL? Checkout the latest quickstart guide

yousecjoe commented 3 months ago

Anyone willing to test out the latest on WSL? Checkout the latest quickstart guide

Me! I'll do it tonight. This is exciting and I have been looking forward to this, great job everyone!

KeenMaron commented 3 months ago

Anyone willing to test out the latest on WSL? Checkout the latest quickstart guide

Do you have updated the instruction for nvidia user in wsl? Because the subsystem recognize the gpu and it is useable, but there is no native driver installed in the subsystem. I had issues to make the container run.

namquang93 commented 2 months ago

looks like it still doesn't work since WSL doesn't have uinput

ABeltramo commented 2 months ago

I assume you followed the wrong tab of the quickstart guide, in WSL2 there's no mention of /dev/uinput 😉

Anyway, others have reported on Discord that HW acceleration is not properly working; seems that Windows exposes the GPU as a different virtual vendor and our code doesn't recognise that MS specific card.

Even with a temporary fix for that it seems that Sway doesn't run in WSL (or at least not without further fixes). If anyone wants to make this work I'm afraid they have to get their hands dirty and write some code!

namquang93 commented 2 months ago

I followed the Nvidia tab since my laptop has an Intel iGPU and a NVIDIA dGPU. I'm gonna try using the WSL2 guide.

namquang93 commented 2 months ago

looks like I got it to run with WSL but I can't seem to access the web interface either in https://localhost:47984 or http://localhost:47989. Anyone got an idea why?

Asofwar commented 2 months ago

В Windows Docker по умолчанию работает с использованием виртуальной машины (через WSL 2 или Hyper-V). Если вы используете --network=host, это может не работать так, как ожидалось, потому что host сеть работает иначе в Windows.

Рассмотрите возможность использования портового перенаправления, вместо --network=host:

docker run \ --name wolf \ -e XDG_RUNTIME_DIR=/tmp/sockets \ -v /tmp/sockets:/tmp/sockets:rw \ -e NVIDIA_DRIVER_VOLUME_NAME=nvidia-driver-vol \ -v nvidia-driver-vol:/usr/nvidia:rw \ -e HOST_APPS_STATE_FOLDER=/etc/wolf \ -v /etc/wolf:/etc/wolf:rw \ -v /var/run/docker.sock:/var/run/docker.sock:rw \ --device /dev/dri/ \ --gpus all \ -p 47984:47984/tcp \ -p 47989:47989/tcp \ -p 47999:47999/udp \ -p 48010:48010/tcp \ -p 48100-48110:48100-48110/udp \ -p 48200-48210:48200-48210/udp \ ghcr.io/games-on-whales/wolf:stable

StimzRx commented 2 months ago

looks like I got it to run with WSL but I can't seem to access the web interface either in https://localhost:47984 or http://localhost:47989. Anyone got an idea why?

From what I remember, you have to get the direct IP of WSL2 and connect to it via that. If you list all network interfaces in the windows command prompt it should show the WSL2's IP.

LanHikari22 commented 1 month ago

Hi. Coming from https://github.com/games-on-whales/wolf/issues/129,

I have already tried --gpus all prior but I have added according to @ABeltramo's suggestion NVIDIA_DRIVER_CAPABILITIES=all and NVIDIA_VISIBLE_DEVICES=all and removed the PUID and PGID environmental variables:

docker run \
    --name wolf \
    -e XDG_RUNTIME_DIR=/tmp/sockets \
    -e TZ=America/New_York \
    -e HOST_APPS_STATE_FOLDER=/etc/wolf \
    -v /tmp/sockets:/tmp/sockets:rw \
    -v /etc/wolf:/etc/wolf:rw \
    -v /var/run/docker.sock:/var/run/docker.sock:rw \
    --device /dev/dri/ \
    -p 47984:47984/tcp \
    -p 47989:47989/tcp \
    -p 47999:47999/udp \
    -p 48010:48010/tcp \
    -p 48100-48110:48100-48110/udp \
    -p 48200-48210:48200-48210/udp \
    -e NVIDIA_DRIVER_CAPABILITIES=all \
    -e NVIDIA_VISIBLE_DEVICES=all \
    --gpus all \
    ghcr.io/games-on-whales/wolf:stable

Some docker logs from wolf I have noted:

2024-09-28 14:31:41 2024-09-28T11:31:41.139473Z ERROR smithay::backend::egl::ffi: [EGL] 0x3001 (NOT_INITIALIZED) eglInitialize: DRI2: failed to load driver
2024-09-28 14:31:41 2024-09-28T11:31:41.141046Z ERROR smithay::backend::egl::ffi: [EGL] 0x3001 (NOT_INITIALIZED) eglInitialize: DRI2: failed to load driver

And the encoders being detected now:

2024-09-28 14:31:21 11:31:21.691233310 WARN  | Unable to recognise GPU vendor: vendor 3900
2024-09-28 14:31:21 11:31:21.693637485 INFO  | Using H264 encoder: x264
2024-09-28 14:31:21 11:31:21.693705388 WARN  | Software H264 encoder detected
2024-09-28 14:31:21 11:31:21.695386740 INFO  | Using HEVC encoder: x265
2024-09-28 14:31:21 11:31:21.695450142 WARN  | Software HEVC encoder detected
2024-09-28 14:31:21 11:31:21.696459074 INFO  | Using AV1 encoder: aom
2024-09-28 14:31:21 11:31:21.696531476 WARN  | Software AV1 encoder detected

Black screen issue persists when running Firefox. I'll include the docker logs for wolf, WolfPulseAudio, and WolfFirefox below.

Please note that there is a long string of trace logs in wolf when I am able to move the mouse in the firefox moonlight stream and finally this last log after going in and out of the firefox control which disables my mouse access within it:

... 
2024-09-28 14:34:43 0:03:23.155820051     1 0x7fbd84029370 WARN                  appsrc gstappsrc.c:2469:gst_app_src_push_internal:<wolf_wayland_source> Dropping old item buffer: 0x7fbe0ee7cc10, pts 0:01:16.033330292, dts 0:01:16.033330292, dur 0:00:00.016666666, size 8294400, offset none, offset_end none, flags 0x0
2024-09-28 14:34:43 0:03:23.287723632     1 0x7fbd84029370 WARN                  appsrc gstappsrc.c:2469:gst_app_src_push_internal:<wolf_wayland_source> Dropping old item buffer: 0x7fbe0edb9990, pts 0:01:16.083330290, dts 0:01:16.083330290, dur 0:00:00.016666666, size 8294400, offset none, offset_end none, flags 0x0
2024-09-28 14:34:43 0:03:23.417682748     1 0x7fbd84029370 WARN                  appsrc gstappsrc.c:2469:gst_app_src_push_internal:<wolf_wayland_source> Dropping old item buffer: 0x7fbe0c00d2f0, pts 0:01:16.133330288, dts 0:01:16.133330288, dur 0:00:00.016666666, size 8294400, offset none, offset_end none, flags 0x0
2024-09-28 14:34:43 
2024-09-28 14:34:43 (wolf:1): GLib-WARNING **: 07:34:43.547: Idle source dispatched without callback. You must call g_source_set_callback().

240928 wolf docker logs 1.txt 240928 WolfFirefox docker logs 1.txt 240928 WolfPulseAudio docker logs 1.txt

Thanks, Mohammed

PeterTucker commented 1 month ago

Current docker-compose setup on WSL2:

services:
  wolf:
    image: ghcr.io/games-on-whales/wolf:stable
    container_name: wolf
    environment:
      - XDG_RUNTIME_DIR=/tmp/sockets
      - PUID=1000
      - PGID=1000
      - NVIDIA_DRIVER_CAPABILITIES=all
      - NVIDIA_VISIBLE_DEVICES=all
      - TZ=America/New_York
      - HOST_APPS_STATE_FOLDER=/etc/wolf
#    volumes:
#      - /tmp/sockets:/tmp/sockets:rw
#      - /etc/wolf:/etc/wolf:rw
#      - /var/run/docker.sock:/var/run/docker.sock:rw
    volumes:
      - /etc/wolf/:/etc/wolf
      - /tmp/sockets:/tmp/sockets:rw
      - /var/run/docker.sock:/var/run/docker.sock:rw
      - /dev/:/dev/:rw
      - /run/udev:/run/udev:rw
    devices:
      - /dev/dri/
    ports:
      - "47984:47984/tcp"
      - "47989:47989/tcp"
      - "47999:47999/udp"
      - "48010:48010/tcp"
      - "48100-48110:48100-48110/udp"
      - "48200-48210:48200-48210/udp"
    runtime: nvidia
    deploy:
      resources:
        reservations:
          devices:
            - capabilities: [gpu]

When I run steam this is what I get:

wolf  | 2024-10-08T03:37:56.692703Z ERROR smithay::backend::egl::ffi: [EGL] 0x3001 (NOT_INITIALIZED) eglInitialize: DRI2: failed to load driver
wolf  | libEGL warning: MESA-LOADER: failed to open vgem: /usr/lib/dri/vgem_dri.so: cannot open shared object file: No such file or directory (search paths /usr/lib/x86_64-linux-gnu/dri:\$${ORIGIN}/dri:/usr/lib/dri, suffix _dri)
wolf  |
wolf  | 2024-10-08T03:37:56.693543Z ERROR smithay::backend::egl::ffi: [EGL] 0x3001 (NOT_INITIALIZED) eglInitialize: DRI2: failed to load driver
wolf  | libEGL warning: MESA-LOADER: failed to open vgem: /usr/lib/dri/vgem_dri.so: cannot open shared object file: No such file or directory (search paths /usr/lib/x86_64-linux-gnu/dri:\$${ORIGIN}/dri:/usr/lib/dri, suffix _dri)
wolf  |
wolf  | libEGL warning: NEEDS EXTENSION: falling back to kms_swrast
wolf  | 03:37:56.752099588 WARN  | Unable to recognise GPU vendor: vendor 6a40
wolf  | 03:37:56.762226966 WARN  | Unable to recognise GPU vendor: vendor 72a0
wolf  | 03:37:56.794643856 WARN  | [DOCKER] Container WolfSteam_5038615994044444484 already present, removing first
wolf  | x265 [info]: HEVC encoder version 3.5+1-f0c1022b6
wolf  | x265 [info]: build info [Linux][GCC 13.2.0][64 bit] 8bit+10bit+12bit
wolf  | x265 [info]: using cpu capabilities: MMX2 SSE2Fast LZCNT SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2
wolf  | x265 [info]: Main profile, Level-3.1 (Main tier)
wolf  | x265 [debug]: No pool thread available. Deciding frame-threads based on detected CPU threads
wolf  | x265 [warning]: No thread pool allocated, --wpp disabled
wolf  | x265 [warning]: No thread pool allocated, --lookahead-slices disabled
wolf  | x265 [info]: Slices                              : 1
wolf  | x265 [info]: frame threads / pool features       : 1 / none
wolf  | x265 [info]: Coding QT: max CU size, min CU size : 32 / 8
wolf  | x265 [info]: Residual QT: max TU size, max depth : 32 / 1 inter / 1 intra
wolf  | x265 [info]: ME / range / subpel / merge         : hex / 57 / 1 / 2
wolf  | x265 [info]: Keyframe min / max / scenecut       : disabled
wolf  | x265 [info]: Lookahead / bframes / badapt        : 0 / 0 / 0
wolf  | x265 [info]: b-pyramid / weightp / weightb       : 0 / 0 / 0
wolf  | x265 [info]: References / ref-limit  cu / depth  : 1 / off / off
wolf  | x265 [info]: Rate Control                        : CQP-28
wolf  | x265 [info]: tools: rd=2 psy-rd=2.00 early-skip rskip mode=1 signhide tmvp
wolf  | x265 [info]: tools: fast-intra strong-intra-smoothing deblock
wolf  | 0:00:20.718408800     1 0x7f60a4029140 WARN                  appsrc gstappsrc.c:2469:gst_app_src_push_internal:<wolf_wayland_source> Dropping old item buffer: 0x7f610eb9a200, pts 0:00:00.066666666, dts 0:00:00.066666666, dur 0:00:00.033333333, size 3686400, offset none, offset_end none, flags 0x0
wolf  | 0:00:20.872058705     1 0x7f609c000e20 WARN                basesink gstbasesink.c:1248:gst_base_sink_query_latency:<udpsink0> warning: Pipeline construction is invalid, please add queues.
wolf  | 0:00:20.872140211     1 0x7f609c000e20 WARN                basesink gstbasesink.c:1248:gst_base_sink_query_latency:<udpsink0> warning: Not enough buffering available for  the processing deadline of 0:00:00.020000000, add enough queues to buffer  0:00:00.020000000 additional data. Shortening processing latency to 0:00:00.000000000.
wolf  |
wolf  | (wolf:1): GLib-WARNING **: 23:37:57.216: Idle source dispatched without callback. You must call g_source_set_callback().
wolf  | 03:37:57.701503234 INFO  | [DOCKER] Starting container: /WolfSteam_5038615994044444484

Shows a black background with a cursor. Never seems to finish loading.

salty2011 commented 1 month ago

Correct me if I'm wrong, but WSL 2 doesn't exactly expose the actual graphics card. Instead you'll find it's a Microsoft virtual adapter that's good enough for rendering light weight UI's but doesn't expose any of the actual GPU capabilities. In the case of Wolf, it looks for Intel / Nvidia or AMD gpu's to determine what encoding pipeline to use.

if you do the following command in WSL

lspci | grep VGA

What adapters are returned?

PeterTucker commented 1 month ago

Unfortunately, I don't see much info.

However if I run this:

I do get specific information about my 3060:

salty2011 commented 1 month ago

So looking at the output, WSL isn't listing your Nvidia cards. instead WSL just has a basic 3d controller (first entry in screen shot).

I am not aware of any method to expose your actual graphics card into WSL so that docker can leverage in Wolf. It is this reason why WSL is considered unsupported by Wolf. As understood WSL has a technical limitation that prevent Wolf from being able to leverage the actual graphics card.

Given this limitation this its not currently a focus to get it working, but welcome comments / feedback from others who may know how get around this limitation and submit a PR to update our doc's.

In the meantime I will review the docs for WSL and call out explicitly the current known limitations as we understand them.

games-on-whales / gow