m1k1o / neko

A self hosted virtual browser that runs in docker and uses WebRTC.
https://neko.m1k1o.net/
Apache License 2.0
7.46k stars 546 forks source link

Wayland target, like sway #444

Open SimonBiggs opened 5 hours ago

SimonBiggs commented 5 hours ago

Hi @m1k1o,

Do you have an idea how I might get this working with a wayland target like sway. Also, for clarification, I don't mean something like this:

cat <<EOF | sudo tee /etc/neko/supervisord/sway.conf >/dev/null
[program:sway]
environment=HOME="/home/%(ENV_USER)s",
    USER="%(ENV_USER)s",
    DISPLAY="%(ENV_DISPLAY)s",
    XDG_RUNTIME_DIR="/run/user/%(ENV_USER_UID)s",
    WLR_BACKENDS=x11,
    WLR_LIBINPUT_NO_DEVICES=1,
    WLR_X11_DISPLAY=%(ENV_DISPLAY)s
command=/usr/bin/sway --verbose
stopsignal=INT
autorestart=true
priority=500
user=%(ENV_USER)s
stdout_logfile=/var/log/neko/sway.log
stdout_logfile_maxbytes=100MB
stdout_logfile_backups=10
redirect_stderr=true

As that is using sway with an x11 backend. The above config works fine enough. But I explicitly don't want to be using x11 so that I can have many nested sway instances without needing to have them all in RAM

m1k1o commented 4 hours ago

To get just the streaming done, it needs to be adjused in gstreamer. I would expect that there is similar element to ximagexrc like waylandsrc but there is not.

To get the controls ported to wayland, there would need to be implemented a new driver (possibly using libei) that satisfies this interace: https://github.com/m1k1o/neko/blob/5d62e56d601ebf66a81a702f68aa1f58072106b1/server/internal/types/desktop.go#L43-L77

SimonBiggs commented 4 hours ago

Also... I imagine multi cursor / seat would be quite helpful?

https://github.com/swaywm/wlroots/pull/352 https://youtu.be/WO2L_ihO_rI

SimonBiggs commented 4 hours ago

SEAT CONFIGURATION¶ Configure options for multiseat mode.

A seat is a collection of input devices that act independently of each other. Seats are identified by name and the default seat is seat0 if no seats are configured. While sway is running, - (hyphen) can be used as an alias for the current seat. Each seat has an independent keyboard focus and a separate cursor that is controlled by the pointer devices of the seat. This is useful for multiple people using the desktop at the same time with their own devices (each sitting in their own "seat"). The wildcard character, *, can also be used in place of to change settings for all seats.

https://manpages.debian.org/experimental/sway/sway-input.5.en.html

m1k1o commented 4 hours ago

Definitely, that would be great to have.

SimonBiggs commented 2 hours ago

I think what I'll need to do is just set up the "view only" no inputs version first with webRTC, and then potentially in the next couple of weeks I'll come back to this and see if I can sort out the inputs.

Do you have any preferences for how to do PRs or anything?

SimonBiggs commented 2 hours ago

Do you have a recommendation on how best to use / edit / refactor your tool if all I want to do with it initially is pass through Wayland screen captures through webRTC?

SimonBiggs commented 56 minutes ago

This seems like it would be something appropriate to use to pipe the video and audio through to Neko? https://github.com/ammen99/wf-recorder

SimonBiggs commented 52 minutes ago

Actually, maybe as a first step entirely I'll just have wf-recorder go to the video loopback and then just stream it with this for now (until I get Neko inputs sorted):

https://github.com/mpromonet/webrtc-streamer