Remote desktop support

[fredldotme]

I'd be willing to implement support for remote desktop functionality into Mir. Are there any current plans, expectations and/or code snippets available? If not, what would be in the expected feature set (application-only remoting, protocol support, etc)?

[AlanGriffiths]

Hello @beidl,

thanks for taking an interest. This is certainly something we'd welcome.

There are no plans currently. We had some internal discussions a few years ago, but the context has changed a lot since then.

As your post implies, the requirements are the first thing that needs capturing.

The most recent relevant discussion was with the UBports group: https://forums.ubports.com/topic/1389/reverse-convergence-view-control-your-phone-from-computer-like-vnc-rdp/

Clearly, that's not the only usecase to consider as we'd want to support desktop sharing and connecting to other types of remote devices/computer.

The approach I outlined in the above UBports discussion (putting the support into the "android" platform) is not the the most flexible, it would better to integrate support into libmirserver.

That raises a host of concerns about security and choice of protocols that we would need to consider. I've not researched this (yet) do you have some ideas?

Also, any sort of remote desktop implementation would need to concern itself with efficiency. That would probably entail getting into the damage tracking in the compositor logic. But, being a long term proponent of "making working code fast" over "making fast code work", I think that's probably best left until something is working.

As for prior art:

You can find the code for "screencast" support starting from: src/include/server/mir/frontend/screencast.h

We did have some support for "display casting": https://wiki.ubuntu.com/Touch/DisplayCasting (I'd have to check the extent to which that was tied into the android/libhybris code - @albaguirre were you involved?).

Both the "graphics platform" and "input platform" are dynamically loaded, this might also provide a route into the code.

Mentioning @gerboland @RAOF and @wmww as they may have further thoughts.

[AlanGriffiths]

A couple of prior art links (courtesy of "GizmoChicken" on the community forums):

• https://www.ctrl.blog/entry/wayland-gnome-remote-desktop
• https://fedoraproject.org/wiki/Changes/WaylandRemoteDesktop

[fredldotme]

AFAICT the aethercast support follows a "push" approach (well, Miracast) where as remote desktop protocols involve a pull approach, but there could be some useful pieces in there.

Implementing support into libmirserver would imply making the display server itself aware of remote connections. Is this intended and what are the pros and cons?

Also, the Mir screencast utility seems to use Mir's internal protocol. Are input events also exposed through the Mir socket? Then remote screencasting could be implemented as a separate application/service proxying events to the compositor, allowing a single solution for all graphics platforms. PipeWire could also be implemented on this proxy service side, keeping the display server/compositor process clean from networking code.

On the other hand:

• How's the long-term plan with keeping support for Mir's own protocol (is the internal protocol even affected by the move to the Wayland client protocol?)
• What kinds of input events are expected to be passed through? (IMO preferably all, including touch)

[AlanGriffiths]

Yes, the references were intended as a way into the code rather than something that's directly useful.

As I said before, we've no current plans to the extent of not having gathered requirements. My instinct would be to have a local process to connect to the mir server and act as an intermediary but I've not thought through the implications. Can you provide any guidance?

Mir's internal protocol and the libmirclient API are "legacy". We're not using or developing them for any new features. There are projects using it (and our test suite) but they are on notice that support will be withdrawn at some (distant) point.

It would likely be useful to find a forum with more bandwidth to discuss this. Can you use google hangouts? What timezone are you in? When is a good time?

[fredldotme]

Yes, the intermediary process would be what I described as proxy service.

I'm Central Eastern, though I'm on a capped connection right now so Hangouts wouldn't work out quite well, I don't know when this particular situation is going to change.

[RAOF]

The aethercast work is probably not what you want to start with - that was screencasting only, and didn't have a way to get input.

I would approach this as a combined input/graphics platform, like the X11 platform. That lets you do all the things you need, and as a bonus implementing the post_if_optimisable hook would let you do some bandwidth optimisations.

The downside here is that it precludes desktop sharing - we (currently) only load a single platform module, so the server would be either remote or local, not both.

One of the tasks for 100%-complete-really-done hybrid support is to change this so that multiple platforms are loaded, so the sharing issue would eventually be resolvable.

[fredldotme]

@RAOF having multiple graphics/input platforms handle this kind of use case implies treating every platform the same, where as a Screenbuffer-Sharing platform (to me at least) seems to have different intrinsic behaviour (take something someone else rendered and present it VS render and present by itself). This also brings networking into play where a full networking stack (specifically, having remote socket connections to open) would have to be loaded into the compositor process, which is something I'm not really fond of for a core component like a system's compositor.

[AlanGriffiths]

I too like the combined input/graphics platform approach. One might side-step the multiple platform issue by implementing as an "adaptor" that can load another platform. Alternatively, it might be better to solve that part of the problem in libmirserver.

[fredldotme]

Implementing parts ("most things") in platform code makes sense indeed, though I would really like to have networking separated into a different process instead of dragging protocol-specific issues into the main compositor process, which means some kind of fd passing, protocol, what have you, between processes. This also raises the question of making the server itself aware of remoting.

Is the "adaptor" idea already at some stage or should we discuss this first before proceeding?

[wmww]

How do we feel about implementing https://github.com/swaywm/wlr-protocols/blob/master/unstable/wlr-export-dmabuf-unstable-v1.xml?

[GizmoChicken]

Reportedly, Fedora plans to rely on PipeWire and GNOME Remote Desktop (in conjunction with ScreenCast and RemoteDesktop APIs provided by xdg-desktop-portal) to bring Wayland remote desktop support to Fedora 29. More information is available here, here, and here.

Would it be feasible to implement APIs that would allow using PipeWire and GNOME Remote Desktop with Mir?

P.S. I don't mean to assert that the GNOME/Fedora approach is better (or worse) than another other approach. (I'm not qualified to make such an assertion.) But the GNOME/Fedora approach will be widely adopted, for what that's worth.

P.P.S. This discussion is way over my head, so unless prompted, this will likely be my only comment in this thread.

[fredldotme]

I've had experience with a similar approach, pushing input events through DBus in a compositor-type scenario. Eavesdropping on the bus doesn't seem to be of any concern as they seemingly aim to solve sandboxing issues through DBus APIs.

This would mean:

• implementing an input injection DBus listener object on the input platform side (as some kind of "adaptor" as Alan called it)
• passing a PipeWire stream from Mir through another DBus listener object when requested, replacing the internal Mir protocol in use by mirscreencast in the long run
• have an application outside of Mir's scope handle transport and encryption shenanigans

My current concerns here revolve around DBus eavesdropping and dbus-daemon CPU usage when spamming the host with input events (I'm running dbus-broker on Arch, so my mileage regarding DBus performance varies), as well as replacing all this fine C++ code. :)

EDIT: additionally, this answers the remote-session awareness question for the display server, there'd be need for some accounting regarding RemoteDesktop and ScreenCast session lifecycles.

[wmww]

It seems to me it would be more appropriate to write or pull in a Wayland protocol for input injection instead of using D-Bus. Something along the lines of this: https://github.com/KDE/kwayland/blob/master/src/client/protocols/fake-input.xml (but with auth moved to an external protocol, and some other changes needed)

[fredldotme]

So, what we are currently discussing seems to be which input protocol to use, which could be changed over time (if and/or when needed) through the input platform facilities in Mir.

I'd agree that adding DBus in the mix would cause more complexity in terms of security concerns, having a private IPC channel between display server and remote desktop server ala KWayland makes sense here, keeping display-server <-> wl-client and display-server <-> remote-desktop-server communications distinctly separate.

Does adding a platform-type abstraction layer for display buffer sharing make sense? I'm not sure if PipeWire is exactly there yet.

[AlanGriffiths]

I concur with "have an application outside of Mir's scope handle transport and encryption shenanigans" I'll call it "proxy-mir" for the sake of having a name.

A platform-type abstraction makes sense for both display and input remoting. I'm open to argument whether this belongs outside or inside libmirserver. Without a clear argument either way I'd suggest trying "outside" first (as I think it would likely be easier to migrate out to in than in to out when we gain more knowledge).

There's a bunch of stuff to address that might be beyond the scope of a "platform-type abstraction": is the connection an additional display device (cf the UBports discussion)? or a view onto existing display devices? Controlling that certainly feels more like a separate "proxy-mir" app with a UI than part of the server.

Protocols: Between Mir and "proxy-mir" I don't know of good candidates (that's ignorance, not an assertion they don't exist).

Between "proxy-mir" and a remote desktop I guess VNC has wide support.

[fredldotme]

Remote desktop functionality to my understanding inherently means showing content of an existing & connected display, but thinking of Windows Server RDP sessions I can see an argument for headless systems as well. Ideally we would solve more than just Remote Desktop-related problems. What this basically comes down to is transfering prerendered display frames to the RDP/VNC-enabled proxy (though I wouldn't put this functionality in scope of the Mir project yet, except for a reference implementation), which could ease work for implementing convergence over similar technologies (having the PC/device render display content offscreen onto a virtual display), basically passing the "hot potato" around. Where the hot potato comes from shouldn't matter, we'd just pass it to the proxy (over PipeWire, shared memory, memfd or whatever).

Additionally, passing damage information about which parts of the screen were re-rendered to the proxy would be useful (where was the potato pierced?). Can this be accounted for along the way to mirscreencast with existing code? I'd suggest adding the abstraction near where ever this information is stored.

[RAOF]

This is (one of) the advantage(s) of having the Mir end of this use the existing graphics platform API - there's still reasonably rich scene information available at the DisplayBuffer level. A good Renderer implementation will pass through windows individually, making all sorts of optimisations possible or easier. If the remoting protocol allows it, you could even do the final compositing step on the client, which would make things like moving windows around basically free. We don't actually track damage information currently, but once we do we would provide it to the DisplayBuffer (things like DisplayLink USB hardware can optimise based on framebuffer damage, too).

This would also fit in nicely with the long-term hybrid-graphics plans. Our hybrid support is currently the simplest thing that could work - a single mesa-kms instance drives all GPUs. This has drawbacks, the most obvious of which is that it only works if all the GPUs are supported by Mesa. The long-term plan is to make each GPU be driven by a separate platform instance, with render-GPU communicating with display-GPU via an explicit dma-buf interface.

Implementing this as a graphics platform makes no demands on how you get the “hot potato” to the remote desktop process - the graphics platform can register whatever Wayland extensions it desires (indeed, it has to register at least one in order for clients to submit GPU buffers), the graphics platform can expose a DBus interface, or whatever other IPC is necessary between Mir and the remote desktop server process. (A similar hook for input platforms to register Wayland extensions would be trivial).

[RAOF]

For requirements gathering, I think we have three use-cases:

1. Desktop sharing - remote access to an existing desktop session, where both the existing and new user want to interact with the desktop.
2. Remote login - remote access to a new session on a server; each connection is basically a new seat, with a separate set of input/output devices
3. a. (ish) Desktop extension - the UBports use case: add an extra display to an existing desktop which just happens to be driven by a GPU not physically attached to the system. b. (ish) Weird Input - related UBports use case: use an app on your phone to turn your touchscreen into a touchpad for the desktop.

I am undecided whether all 3ish of these should use the same infrastructure. 2. is easily and obviously implementable as a combination input/graphics platform exactly like the X11 platform, right now. 3a looks like a graphics platform, and 3b looks like an input platform, but implementing them as such requires core changes to allow multiple platform implementations to be loaded.

1 looks the least like a graphics platform - rather than adding a new output, this wants to clone (some subset of?) the existing outputs. This can expressed by a graphics platform - simply set each relevant output to a clone - but this is not necessarily what is meant to be expressed by a DisplayConfiguration? Maybe we'd need an extra is_virtual bit to help things which inspect display configuration?

[RAOF]

We discussed this in the regular Mir team meeting last night, and the consensus was that making a “remoting-proxy” input/graphics platform would be the sensible way to begin this. That would let you satisfy use-case (2) with the least amount of work (particularly, with no core Mir code changes), and can be incrementally extended to cover the other use-cases.

Suggested implementation plan

Because of the current state of the graphics platform API you'll need to implement both a RenderingPlatform and a DisplayPlatform. The simplest thing to do here is to copy src/platforms/mesa/server/kms and the associated helpers into a new platform.

You should be able to leave most of the code as-is; the Display implementation is the relevant point of difference (for configuration and cursor management), and particularly the DisplayBuffer implementation which is what handles the actual rendered output. (This needs to implement renderer::gl::RenderTarget). GBMOutputSurface, and particularly ::lock_front(), give you a gbm_bo* you can slurp the dma-buf from.

For input I'd suggest checking out src/platforms/mesa/server/x11/input; that should be a good example of creating input devices to proxy from another source.

That will give you all the display output and allow you to inject input events, so all the necessary Mir-side interfaces.

We don't particularly mind which project you choose to do the remote-desktop-protocol bit, nor do we mind how you choose to communicate between the “remoting-proxy” platform and that process. Using whatever GNOME is planning to use is a reasonable choice, but there may well be other reasonable choices. As a loadable platform module it's quite easy to have multiple implementations, and select the one you want to use at runtime.

This should give a viable implementation for remoting into a headless server. Next steps would be to add dynamic platform loading and better separate display from rendering; then the platform can be used for the other use-cases.

[Saviq]

Would the same approach allow for zero-copy in-GPU encoding? Most GPUs have built-in efficient h264(5?) encoders which can cater for screencasting, as we implemented for Ubuntu Touch and its Miracast support.

[RAOF]

Yup! You get a dma-buf out of the renderer, which is what you need to feed to the encoding APIs.

Now, for some encoder hardware (particularly embedded) you'll need buffers allocated with all sorts of interesting constraints; this might require a negotiation interface between RenderPlaftorm and the DisplayPlatform(s), but could also be done (at the cost of a second render pass) in the platform itself. IIRC the Miracast support used the render-twice approach.

[fredldotme]

Getting headless remote desktop support into Mir could require awareness about sessions, creating a new session/seat for a new user. This would additionally require configurability through the shell in the long run, allowing shell implementations to select behaviour through libmirserver. Should an incoming remote connection create a new session/seat when connecting or are those bound to a VT at the moment? May I join the next Mir meeting to discuss details?

[RAOF]

Mir won't require awareness about sessions and such; that's a job for some other subsystem - for example, you can get LightDM to handle remote connections by starting up a remote display server. I believe GDM has some similar sort of support.

We suggest the implementation plan above because we think it's the path of least work required to get something working that can then be incrementally expanded to cover all the identified use-cases.

Particularly, it doesn't require any core Mir infrastructure before it can work, but can take advantage of (and drive) core Mir changes once they happen.

This is only a suggestion; I certainly wouldn't reject a merge proposal that covered desktop sharing but not headless remoting out of hand, but I think it would likely be a large PR that would need to be broken up into multiple stages, each one of which wouldn't make complete sense without the whole series.

A remote-proxy platform would be self-contained, an easy PR to review and accept. Further PRs to implement the core Mir changes would then have an existing user, and be easier to do incrementally while validating that they actually do what we want.

[AlanGriffiths]

@beidl I imagine the next Mir meeting is at a bad time for you (Tuesday, 10:30 UTC).

I hope you already have enough answers but if not, and you're online at the same time as me (Western European Time) or @RAOF (Australian Eastern Time) then either of us can help.

[fredldotme]

Yup, should be enough. Agreed, core Mir infrastructure should be changed when needed. I can't give you an estimate when I have an initial implementation + PR available, maybe within a month.

[Flohack74]

Thanks @AlanGriffiths for pointing me on this one.

As I am working in both "worlds" of Windows and Linux, I was always missing the quick & decent way of getting a remote graphical desktop for a Linux machine. First, on servers this was never installed. Okay, you might argue that a server does not need a GUI - but, in todays graphic world, this is not so true anymore. Imagine downloading a needed software pack with Lynx. Getting a spreadsheet report of data transfer per day. Quickly dumping a few files in a diff viewer - all that is troublesome on a text terminal.

X11 is too complicated and has no easy usable software for Windows. VNC is not the same, first of all you must be logged in to the machine, and then the protocol has issues with screen scaling and color tearing etc.

Plus with VNC only one person can work on a machine. In some scenarios, I can imagine having Windows clients or Thin clients deployed to access Linux machines.

I strongly would vote for implementing RDP right away. It has performance, authentication and encryption built-in, can convey Video, Mouse, Keyboard, Sound, Printers, Local drives, etc. I am just not sure all of it is open sourced :)

[Flohack74]

Regarding sessions, RDP has the notion of the "Admin" console vs. new logins. Connecting to the Admin console will simply connect the user with the session associated with the local hardware. Interestingly in such a case the local console will log off and transfer the session to RDP. Other people cannot spy on you what you are doing.

Also, if the logged on user is a different identity he will be informed that someone wants to connect to his session. Around that principle Microsoft put the whole blingbling of Remote Support technology where in the end it IS possible to share the screen and the mouse inputs ;)

[GizmoChicken]

From an article published on Phoronix:

The KDE Plasma/KWin developers have been pursuing Wayland remote desktop support along a similar route to the GNOME Shell camp by making use of PipeWire and the XDG-Desktop-Portal. Bits are already in place for KDE Plasma 5.13 and the upcoming 5.14 release, but for the 5.15 release is now where it sounds like the support may be in good shape for end-users.

[Fuseteam]

thanks for pointing me to this issue alan, i'll watch this issue and probably re-read to properly understand what is neccesary to forward a mir session aka use case 2 and 3

[AlanGriffiths]

A project relevant to this enhancement is Waypipe: https://mstoeckl.com/notes/gsoc/blog.html

[edit]

I realize it isn't the same thing (it is more like X forwarding than remote desktop). But I still think it is something that might be part of a solution.

[Flohack74]

Uh I think this one fell asleep pretty much, myself has no skills and time to contribute but I am here as a product manager anytime needed ^^

[fredldotme]

Yeah I found relatively little time for it due to involvement in UBports and Halium. I hope we could gather enough information for someone else to step in.

[Fuseteam]

Waypipe definitely looks like a part of a solution if not an alternative solution

[abmyii]

Apologies for reviving such an old thread, but I have made some good progress with getting VNC working but only for Mir 1.2.0 - not Mir 2. This is because UBPorts is still using 1.2.0 and the aim of my project is to be able to control the devices over VNC. This may still be useful in getting VNC working for Mir 2 if anyone is interested in giving it a go. My project (which is a port of https://github.com/ycheng/mir-vnc-server) is on GitLab: https://gitlab.com/abmyii/ubports-mir-vnc-server

Note it uses libevdev to simulate input events, not mir::events::make_event directly which I might need to do.

[Fuseteam]

no need to apologize, this thread was looking for someone to work on it if memory serves

[AlanGriffiths]

It probably isn't what everyone is looking for in this issue (there are several scenaro) but...

It is possible to run a Mir 2.x based shell on a remote X server:

ssh <login@remote> snap install --edge egmde
ssh -XC <login@remote> egmde.remote

[AlanGriffiths]

ssh <login@remote> snap install --edge egmde
ssh -XC <login@remote> egmde.remote

Small update for more recent egmde versions:

ssh <login@remote> snap install --edge egmde
ssh -XC <login@remote> sh -lc 'egmde --shell-enable-autostart`

[AlanGriffiths]

Poking an old thread (again) to keep anyone watching informed: Egmde over VNC

This doesn't cover all the usecases people are talking about, just what RAOF called "Remote login" above.

It also doesn't require any work in Mir, instead it piles up ssh, x11vnc, Xorg and Mir's "X11" platform to provide a VNC server.

Consider this a prototype as there's a lot that could be optimized. For example, a Mir "vnc" platform could be implemented that combines what x11vnc, Xvfb, and Mir's "X11" platform are providing in this stack.

What this prototype does prove though, is that Mir's "platform" abstraction is adequate to implement a VNC platform.

If anyone wants to see how this is done or play around it is just this shell script: /snap/egmde/current/bin/egmde.remote

[Fuseteam]

cool, i'd have to check how this works and if it can cover my usecase :smiley:

[Fuseteam]

so uh i was reading up on freerdp and apparently WSLg utilizes freerdp to achieve application remoting i forget if this falls in the scope of this issue but it feels relevant enough. perhaps mir could implement a similar backend based on weston's implementation?

........then again there's waypipe too 👀

[Fuseteam]

welp as i post that, this pops in my telegram feed today: https://www.omgubuntu.co.uk/2022/06/use-ipad-as-second-monitor-ubuntu-22-04

[wmww]

We're currently looking into supporting wayvnc. gnome-remote-desktop is difficult to make work because it depends on a bunch of gnome-specific d-bus APIs.

[AlanGriffiths]

Just poking around freerdp I found that wlfreerdp is a thing. Not looked but it probably uses similar Wayland extensions to wayvnc.

[wmww]

wlfreerdp is a client not a server

[Fuseteam]

Just poking around freerdp I found that wlfreerdp is a thing. Not looked but it probably uses similar Wayland extensions to wayvnc.

fwiw it appears weston has a rdp backend, which uses freerdp to enable wslg on windows

[Saviq]

2383 and #2504 have implemented the Mir side of this. wayvnc is one server implementation available, and our distribution of that is ubuntu-frame-vnc. A RDP implementation is possible on top of the same protocol extensions.

[Flohack74]

Very cool!

[AlanGriffiths]

2383 and #2504 have implemented the Mir side of this. wayvnc is one server implementation available, and our distribution of that is ubuntu-frame-vnc. A RDP implementation is possible on top of the same protocol extensions.

I'm not arguing against closing this rambling issue, but do we want to reopen as new issues any of the cases mentioned in https://github.com/MirServer/mir/issues/467#issuecomment-407254752? Vis:

1. Desktop sharing - remote access to an existing desktop session, where both the existing and new user want to interact with the desktop.

2. Remote login - remote access to a new session on a server; each connection is basically a new seat, with a separate set of input/output devices

3. a. (ish) Desktop extension - the UBports use case: add an extra display to an existing desktop which just happens to be driven by a GPU not physically attached to the system. b. (ish) Weird Input - related UBports use case: use an app on your phone to turn your touchscreen into a touchpad for the desktop.

I'm clear that we've addressed 1, and part of 2, but not 3

[Saviq]

I'm clear that we've addressed 1, and part of 2, but not 3

IMO we should open separate, dedicated issues, yes.