Support Wayland protocol to allow GUI apps to work.

[barry-scott]

Given the Linux world is moving to Wayland as the step on from X11 it would be good to see Wayland supported in WSL.

Off the top of my head think there is enough protocol/API to completely isolate a windows Wayland server as part of the normal windows desktop from the wayland clients running in WSL.

All the important GUI tool kits have Wayland support today and its will become mainstream on Linux distros on the next year.

At this point in time doing work for X11 may be wasted effort.

[iz0eyj]

Maybe I'm wrong because I'm not a Linux developer, but I think that the future for Canonical is Mir and not Wayland.

[pharaone]

@iz0eyj right!

[fpqc]

The main problem with Mir and Wayland is that both require DRM/GL. Right now, WSL does not expose graphics hardware, and rendering/compositing is done over remote X, which partially uses AIGLX and partially uses software rendering due to missing plugins in vcXsrv.

In order for DRM/GL to work, Microsoft would need to have an API for driver makers to expose their devices inside of WSL, which is not yet planned. Wayland and Mir do not support remote indirect opengl rendering, and they won't for the foreseeable future, although perhaps it might be possible in the future (I have a thread on this called something like "WSL should add a DRM api for Windows driver makers" or something like that).

In theory, I also don't see a reason why MS couldn't also provide a translation layer for DRM openGL calls to the corresponding Windows openGL driver, since I don't think the syscalls would even have to be translated.

This is what the Usermode side of Linux is looking for: https://en.wikipedia.org/wiki/Direct_Rendering_Manager#API

The LXSS and LXCore might support just forwarding this on to the Graphics drivers, since I know that the win32 graphics system was moved to the kernelmode in like 1995 (see below)

From Wikipedia's article on the native NT api:

user32.dll and gdi32.dll include several other calls that trap into kernel mode. These were not part of the original Windows NT design, as can be seen in Windows NT 3.5. However, due to performance issues of hardware of that age, it was decided to move the graphics subsystem into kernel mode. As such, system call in the range of 0x1000-0x1FFF are satisfied by win32k.sys (instead of ntoskrnl.exe as done for 0-0x0FFF), and are declared in user32.dll and gdi32.dll. These functions have the NtUser and NtGdi prefix (e.g. NtUserLockWorkStation and NtGdiEnableEudc).

However, I'm not positive how this works, since I am pretty sure that both OpenGL and DirectX on Windows bypass GDI entirely, and I'm not sure how the Win32 subsystem interfaces at the kernel level with Graphics drivers. I have some speculation and research that I've done below:

Usermode to kernelmode for hw-accelerated OpenGL in Windows

Apparently the situation with openGL on Windows is that the built-in library opengl32.dll is an ancient library used as a loader to get to the graphics card's openGL interface with basic functionality (sort of like how VGA used to work), which then loads a series of OpenGL extensions provided by GLEW in a special header in order to extend the functionality to a modern version of OpenGL.

This means that the "magic" happens inside of opengl32.dll, which opens the tunnel to the graphics hardware, which is then quickly widened in scope to the full modern OpenGL4.5 API by loading OpenGL extensions.

Maybe someone who has experience writing graphics drivers for Windows could explain this at a high level?

Usermode to kernelmode for hw-accelerated OpenGL in Linux

Presumably, however OpenGL32.dll works, the WSL team understands it, and the difficulty would be either mapping the usermode functionality of Linux's Direct Rendering Manager to a corresponding Windows native component, or emulate it in lxss.sys. Since NT understands its own drivers, this would probably actually avoid driver makers having to support LXSS directly. I'm not exactly sure how this would work, but that is probably what you would need to do.

[iz0eyj]

@fpqc would be possible to build a layer OVER WSL that intercepts the graphic calls and transmit it to the Win side?

[fpqc]

@iz0eyj Not with Wayland or Mir. Both of them, I believe, do not have server-side rendering functionality. All rendering is done at the client side via libdrm or perhaps with a fallback to software rendering.

The compatibility layer would have to be part of WSL, although conceptually it actually doesn't seem that difficult to imagine MS being able to emulate the direct rendering manager itself (it would be part of the WSL layer sitting between libdrm and libgl on the Linux usermode side and whatever kernel-mode interface Windows uses to provide hardware-accelerated rendering in the kernelmode).

The main problem with DRM is that emulating it would be much harder than emulating the Linux Standard Base API, since DRM is much newer and much less well documented, and potentially less stable. It also happens to be one of the most active areas of Linux Kernel development over the past few years, and could potentially be much more difficult to emulate, especially if there is no corresponding native NT component that provides the same functionality as DRM. It could entail essentially rewriting 5-10 years of recent (and therefore less-well-documented) work by Linux Kernel devs from scratch, which seems like a pretty daunting task.

[barry-scott]

@iz0eyj Oh yeah Mir... (I wonder how long Canonical will keep that project going, they do like going out on there own).

@fpqc Server side rendering is almost dead these days. Even with X11 modern tool kits do lots of rendering client side and just give X11 the pixmap to draw. Also note that X11 has security issues which Wayland has fixed.

@fpqc From what I have seen the Linux DRM has lots of documentation, which can always be improved if that is necessary.

I'm sure if Microsoft in interesting it adding DRM they can talk to the developers from Intel, AMD etc that wrote the specs and implementation.

I have not looked but I expect that Mir sits on top of DRM as well.

[barry-scott]

All the graphics cards these day have Windows and Linux driver. The graphic card vendors could easily expose there Linux API for WSL if Microsoft specs the interface.

[fpqc]

@barry-scott Not exactly how it would work. They would have to be Windows drivers that attached to some kind of DRM translation interface (or Windows could manage access). Using the Linux driver directly (at least the kernel-mode part) is totally out of the question.

[barry-scott]

@fpqc I must not have been clear.

Assuming that MS wish to do this they provide a Windows Kernal API that allows the DRM interface into the graphics driver.

The vendors have DRM code. Its not beyond the wit of man to put that code into the same Windows driver with the Windows Kernel interface code.

The Linux Kernal DRM API can then be exposed to a WSL user process.

That is how it could work.

[anidotnet]

Now as Mir is out of the picture, implementing Wayland support is the only hurdle in supporting GUI apps in wsl.

[oleid]

Are you sure DRM is required at all? The way I understood it DRM would only be a requirement for running weston or gnome-shell on the Linux side. But that is not what you're want to do.

Wayland doesn't specify how the compositor does the job. C.f. the weston branch using raspberry pi's dedicated hardware or Sailfish's compositor, which uses proprietary android drivers.

Theoretically, one could implement a Direct X12 based compositor.

If the client is using hardware accelerated drawing, that's of course a different story. OpenGL calls would need to be for to the host operating system. Maybe VMWare's gallium driver could be adapted. But for the time being software rendering would do.

[fpqc]

Theoretically one could implement a DX12 based compositor but it would be a lot of work and would not be packaged with ordinary Linux distros.

[oleid]

I'm not saying one should use DX12. I just wanted to point out, that the Linux guest doesn't have to know details about the compositor just to get the basics working. It would definitely make sense to have some means of doing hardware accelerated rendering on the client side and only implement a basic compositor on Windows side supporting client compositors like gnome-shell or kwin. AFAIR this is what login managers like gdm do. But for the rootless case, I assume a full featured compositor on the windows side would be required. I'm not sure if libweston is tied to drm. If not, that would make implementing a Windows native compositor a lot easier, I guess.

[nanonyme]

@oleid I don't think it's that simple. The libdrm part is just the kernel-facing part. Then there's also libGL which is the entrypoint for userspace OpenGL implementation. Both have to work as expected so you get accelerated graphics. Both GL drivers and libdrm are very vendor-specific so I reckon a lot of tuits getting things to work. If we had Vulkan compositors, things might be simpler since it should in theory be possible to translate Vulkan into DX12 for rendering. (the other way is under work already) Alas, Vulkan under Linux is still too new for Wayland compositors to have picked up support

[trissylegs]

Wayland doesn't require drm, weston works, on fbdev, drm, headless, and rdp (raspberry pi I think is being removed). (Weston can also use X11 or another wayland compositor as a backend).

Drm is used to implement OpenGL and Vulkan on Linux for most drivers (basically all except Nvdia's official drivers although they've added some drm support).

So you can get CPU rendering by just implement some Wayland compositor that passes on the frame buffers to Windows somehow. (I assume X11 has some way of doing this).

On Linux+Wayland the EGL library (interally) talks to the wayland compositor to determine what driver's in use and loads the appropriate OpenGL implementation, which then uses DRM to talk directly to the Graphics Driver. (and sends PRIME-FD's to share objects with the compositor).

A simpler solution to "Intercepting the DRM driver and direct it to the windows driver" (fun fact the Intel graphics drivers for Linux and Windows are complete separate from each other). Would be: to implement an EGL that loads GL libraries that make calls to Windows GL (and does some sync with wayland). (Something similar for Vulkan).

(Although then you have the fun fact that programs that might work with Intels' Linux OpenGL don't work with Windows)

[fpqc]

@hornetblack ifyou're ok with cpu rendering, might as well use X

[WSLUser]

May as well ask AMD to do some conversion between cpu rendering and graphics rendering on their APUs (comes with both) if that's the route hornetblack wants to take. I personally would rather utilize my 290x AMD GPUs because of performance boost. At work, computer specs aren't as great so cpu rendering could possibly be acceptable.

[fpqc]

@DarthSpock There is no kernelmode API right now for driver-writers to add GPU support to WSL. Ben Hillis mentioned a while ago that an approach they are considering is a generic compatibility layer lying over the Windows driver stack and a WPF application for display.

If you are familiar with the history of WSL/LXSS, it originated as something called ADSS (an android compatibility layer for Windows Phone), which did have graphical application support, so the team does likely have some of this work done in some deactivated parts of the codebase.

[nanonyme]

Eh, there already is native X server on Windows called xming but it requires boilerplate (eithet X forwarding over SSH or opening up TCP sockets which is a vuln basically) and you cannot have Wayland apps with it

[nanonyme]

I have to disagree. The only difference with the two is packaging. If Microsoft packaged xming so you could install it, would it make it more native? Anyhow, it's not the preferred solution. It'd be better that Linux' graphics servers could render using Windows' driver facilities. It's more about not breaking WSL isolation layers than nativeness

[fpqc]

The problem is not X, it's lack of hardware rendering support. When you render out to X on Linux, there is a usermode driver called "libgl-Vendor" that calls into the Linux kernelmode driver for rendering. Since there is no graphical hardware available in theWSL environment and no proper libgl to drive the hardware, everything is rendered in software.

[therealkenc]

Is libglencompassed by the UserVoice for OPENGL/CUDA? If libgl is so integral then should a UserVoice be created for that specifically?

There was here. It was locked because FAQ.

Can I run ALL Linux apps in WSL? No! WSL is a tool aimed at enabling users who need them to run Bash and core Linux command-line tools on Windows.

WSL does not aim to support GUI desktops or applications (e.g. Gnome, KDE, etc.)

[therealkenc]

The CUDA UserVoice is a CLI development use case scenario. The closed issue is not. No, CUDA will not get you DRM. Someone very clever could implement OpenGL with CUDA. But even if you did, you still need to paint the pixels with something like #230 or a network transparent protocol to a Windows server.

So again, a new one for libgl specifically?

If it will make this issue# go quiet again, yes. Be sure to cite your use case.

[barry-scott]

The problem is that without kernel support there is no window to draw on. This is not a software vs. OpenGL problem. But once you have the window to draw on you wil l have the API to use OpenGL on that Window.

[xinyazhang]

Actually a DRM driver is not mandatory for this goal. NVIDIA is trying to avoid this with EGLStreams and EGLStreams is at least supported at least by Gnome

We can have a guest EGL driver that manages resources, as well as a guest OpenGL driver that translates OpenGL calls to some intermediate representation (e.g. Vulkan commands). Both drivers are talking with the counter parts on the Windows subsystem.

The whole architecture would be similar to hardware accelerated OpenGL in VMWare Workstation, with the kernel driver part in the guest os merged into the hypervisior.

[fpqc]

There was a huge religious war between nvidia and the Wayland team over EGLstreams, and they have agreed to develop a replacement interface in the future. I doubt that MS is going to get involved in developing a driver for something that Wayland has said they will not support.

[WSLUser]

Oh and also...

Someone very clever could implement OpenGL with CUDA

http://www.nvidia.com/content/gtc/documents/1055_gtc09.pdf

[barry-scott]

You get to X11 by using a running the wayland X11 server. MS would be crazy to attempt to support X11 at this point in time.

It does not matter if its drm or egl streams without a kernel interface nothing can be shown. CUDA needs a kernel interface asd well.

[xinyazhang]

@barry-scott I don't think a kernel interface is essential. An alternative is to pipe commands to a corresponding server under WIndows subsystem.

This can be done totally in user space, except for the UNIX socket interop.

[trissylegs]

With Windows supporting AF_UNIX: https://blogs.msdn.microsoft.com/commandline/2017/12/19/af_unix-comes-to-windows/ If they add SCM_RIGHTS and WSL <-> Windows support, you can implement a wayland server in windows that handles WSL clients (and technically windows clients as well, but that's a bit weird, I guess mingw and cygwin).

Getting GPU support is the rest of it. (Which is already a pain point on Linux, but at least it works for Intel and open source AMD drivers).

[barry-scott]

@xinyazhang and @HornetBlack

You could write all the code to forward from linux work over a AF_UNIX socket to a win32 process. But that is man years of work for someone. Oh and the whole point of Wayland/DRM is that sending commands over a socket, as X11 does, is far to slow for modern rich and responsive GUIs.

On the other hand the DRM interface is a stable ABI and a far smaller body of code to implement for the WSL folks. Once done all the rest of the code already exists and is debugged. Mesa, GTK, KDE libs etc.

[xinyazhang]

@barry-scott Wayland over WSL actually shares a lot with other projects, e.g. running wayland GUI remotely. For performance perspective, check virtualGL videos for a reference, or try vmware player by yourself, which basically are using the same arch and the first one is even working over the network.

Other the other hand, I think you really underestimate the workload of developing a DRM driver stack.

The most important thing is you can hardly reuse any existing driver code other than the state tracker in mesa3d. Reusing any other part of this stack (say, radeonsi in mesa and/or AMDGPU in drm) would make the WSL application talk with the GPU hardware directly. Now we have two drivers fighting for the same GPU on the machine, congrats if it didn't crash.

Secondly, the DRM interface is not small; the only sharing part is those APIs that resemble VGA functions, any acceleration (3D and video enc/dec) is implemented with GPU specific ioctls. This means, the pseudo DRM driver in WSL should behave like i915, radeon or nouveau depending on the GPU installed on your system, otherwise your off-the-shelf mesa driver won't work.

The only way feasible is the VMWare's approach: having a virtual DRM in the guest OS, and do the rendering at the host OS, and this is also not an easy task: vmwgfx by itself consists of 32,053 LOC as of 4.14.16 ^{note 1}. More importantly, fundamentally this solution is no difference from my proposal except for an additional DRM interface. (Well, MS might purchase a licence from VMWare to use their existing infrastructure (mostly the renderer) and then dramatically reduce the development cost)

note1: For comparison, VirtualGL consists of 111,716 LOC including both server and client

[therealkenc]

@xinyazhang

The only way feasible is the VMWare's approach

There are a couple of feasible approaches. Talking about them for 38 messages advances neither, of course.

The first step would be to do a framebuffer driver per #230 and go from there. Which has been doable since November 2016 with lxdrv but no one has stepped up to do that (let alone navel gazing about DRM). Of course you'll also need a raw keyboard and mouse character device too.

Someone probably would have stepped up, except for the utility proposition. No one has actually stated a use case for any of this. And no, running VS Code on WSL is not a use case. It runs fine. Running VS Code without VcXsrv isn't a use case either. Because no one is going to spend thousands of man-hours replicating GLX by another name so you don't have to install VcXsrv -- at least while there are 826 -label:console issues open in Github. Something like Blender might be a use case, were it not for the fact that running Blender on WSL would make no kind of sense; because Blender for Windows.

@barry-scott

MS would be crazy to attempt to support X11 at this point in time.

There have been probably a dozen posts from MSFT over the last two years agreeing with that statement.

[trissylegs]

@therealkenc, could an application set up shared memory via LxDrv? (In as much as a Windows and WSL process can share a memory-mapped temp file)

[therealkenc]

could an application set up shared memory via LxDrv?

Allegedly yes. I think you might run into challenges for lack of synchronisation primitives, but those could be overcome. Either way #230 is proof by existence it is possible. It would be possible even if you didn't have mmap and just streamed the frames. You can watch youtube videos in Firefox/Chrome on WSL just fine. The lxdrv interface is not documented, but then if anyone had the intention of implementing /dev/fb0 they would have posted code as far as they could get and asked.

I should in fairness also mention LoWe, since the guy made the effort. Let's call it a novel approach. Not as novel as XSixel, mind you.

[tara-raj]

Thanks for the ask. We currently don't support GUI apps in WSL. If you would like us to support this, however, please upvote a feedback request on our UserVoice page. Voting on that page helps us prioritize asks.

[tianon]

@tara-raj the UserVoice page you've linked appears to have been closed back in 2016, which prevents folks from voting on it -- was that the page you meant to link to?

Edit: perhaps https://wpdev.uservoice.com/forums/266908-command-prompt-console-windows-subsystem-for-l/suggestions/16524991-wsl-wayland-display-manger-support-may-be-easie or https://wpdev.uservoice.com/forums/266908-command-prompt-console-windows-subsystem-for-l/suggestions/17730418-wayland-compositor-implementation-and-library-inte were intended instead?

[tara-raj]

Yes, the first link is the proper one. Thanks for catching that. Updated my original comment

[L-Bouzid]

why not ... just "open source" the project. then others can do the job ???

[WSLUser]

There's technically no need anymore as WSL2 resolves this issue. Under WSL2, it's no longer a syscall issue, just an X server issue.

[nanonyme]

@WSLUser this issue has nothing to do with X Server.

[WSLUser]

You mean other than the fact an X server is required to run GUI apps right? I already stated this issue is resolved with WSL2. Plus this issue is closed already.

[nanonyme]

You can run increasing amounts of GUI's on native Wayland without X server. That is what this ticket was about as it says in the first comment.

[L-Bouzid]

Interesting project (i think), which is trying to develop a framebuffer for wsl : https://github.com/kpocza/LoWe

Microsoft can (if he wants), port or adapt hyper-v accelerated-framebuffer driver. https://www.phoronix.com/scan.php?page=news_item&px=MTMwMzg

which is the step just before ... a Wayland implementation.

[Zubnix]

Hi all,

Some background first: I'm the author of Greenfield and thus can claim to have a least a little bit of Wayland knowledge. I do unfortunately have little to no experience with WSL (yet).

Let's correct some things first:

The only hard requirement currently in the Wayland protocol is file descriptor passing between processes. The extra hard requirement if you decide to use the libwayland C library, which is an implementation of the core protocol, is the use of unix sockets for both server & client (they have to communicate somehow right?)

I'm not familiar with WSL, but if WSL applications can:

• use unix sockets
• draw to the screen somehow or
• can send file descriptors to native windows applications

you actually have all the required pieces already in place to make 95% of all Wayland apps work using simple shared memory buffers (which is supported by nearly all Wayland clients).

However since this thread is still open, I assume the big hurdle is in getting a WSL to render to screen right?

The alternative is a Wayland compositor written as a native windows app and have WSL unix socket interop talk to the WSL GUI applications. If that is possible, then there is literally nothing stopping you from having Linux GUI apps running in WSL right now, except...

...that you need to write a whole Wayland compositor from scratch, or port Weston (the reference Wayland compositor) to Windows and implement a custom Windows backend (display output+user input). Not a trivial task (believe me...).

Another alternative, one I'm investigating myself (and also the reason I found this thread), is to have Greenfield be the compositor and have all server/client communication happen over a localhost tcp socket. The advantage being that the code is already there, I just have to find out if there exists a working setup.

[therealkenc]

The only hard requirement currently in the Wayland protocol is file descriptor passing between processes. I'm not familiar with WSL, but if WSL applications can:

• use unix sockets

We got unix sockets, but not fd passing or credential passing between Windows and WSL.

draw to the screen somehow

Yep that.

However since this thread is still open, I assume the big hurdle is in getting a WSL to render to screen right?

That's one. Inputs (libinput) is the other.

Another alternative, one I'm investigating myself (and also the reason I found this thread), is to have Greenfield be the compositor and have all server/client communication happen over a localhost tcp socket.

No reason Greenfield won't run. WSL is a headless Linux server.

Greenfield is listening on the $WAYLAND_DISPLAY. Unfortunately something something window_create_internal: Assertion ``custom || display->xdg_shell || display->ivi_application' failed.\n","v":1} out of weston-terminal. Longer complaints out of gedit etc. Had hoped for a prettier screencap but that's my time-sink allotment for the day.

Whatever's happening there, it is going to be environmental difficulties, not Greenfield or WSL. Or just errors between the chair and keyboard on my part. If you pursue it further, start with Insiders and WSL2. There isn't any particular reason it won't run on WSL1, but on the chance one of the native modules node is using hits some unimplemented surface or edge-case differences in WSL1's reimagination of the Linux kernel, you'll have a happier experience starting with WSL2 (and working back to WSL1 later). Above is on WSL2 with Ubuntu Disco and Greenfield master.

[Zubnix]

@therealkenc the issue seems to be gstreamer failing to initialize. I think the path of least effort to get something working is to use docker on wsl2 along with this local docker-compose configuration. This local demo environment currently only allows you to launch a single app due the way it's set up. However with some minor modifications it should be able to launch any app you want.

[therealkenc]

the issue seems to be gstreamer failing to initialize.

That's possible if gstreamer is trying to grab any physical device; we have none. Also dbus --system and systemd-udevd aren't alive in a typical WSL userspace. With an strace(1) log leading up to the gstreamer fail I could probably tell you for sure. I don't really "do" node except on television or I'd probably dig further myself. AFAIK though, those aren't hard-blockers to get gstreamer proper running.

I think the path of least effort to get something working is to use docker on wsl2 along with this local docker-compose configuration

Docker is a path of great resistance until localhost networking is sorted. I got a bad gateway error when I tried the docker path right out of the gate. Was the first thing I did yesterday before resorting to doing a build, as that was the Quick Demo instructions at the top of your github, naturally. There are open issues tracking localhost transparency. Localhost isn't really (Windows) localhost on WSL2.

Long-short, if you can get it up on a headless (no graphic devices) VM on a different IP segment (its own localhost) it'll probably run on WSL

[Zubnix]

@therealkenc localhost is not a hard requirement, that is, the docker containers running in WSL should simply be able to communicate with each other. The only hard requirement is being able to make a websocket connection from your browser to the application-endpoint process running inside WSL.

I have a feeling getting this to work should be really trivial as you say, as I actually do run Greenfield on headless cloud VMs, both GPU HW accel instances as well as pure SW instances.

I'll see if I can get WSL up on a virtualbox VM or an old machine and create a working docker-compose.

[therealkenc]

I have a feeling getting this to work should be really trivial as you say, as I actually do run Greenfield on headless cloud VMs

It probably is. Weston with its RDP backend works on WSL2 without modification FWIW. I've just been quiet about it. Your "Let's correct some things first" is all correct.

If you already have Greenfield running on a no-GPU cloud scenario, then there is a much better than even chance it is going to be something of the very obvious with hindsight variety.

Your project is awesome BTW 💯 (I had taken a look at it some time back in a non-WSL context). I mean if the Stadia people can stream 4k games from a server far away, surely it's a reasonable enough approach.... ;). Doing a win32 Wayland server is "doable", but at the end of the day you'll always be fighting against the fact Wayland isn't a wire protocol and WSL is fundamentally remote. That can probably be overcome with waypipe or something like it, but doing a win32 port of all the Linux/*NIX assumptions in the underlying Wayland/freedesktop ecosystem (which are many and varied) would not be pretty. Doable, just not pretty.