Simula lacks ability to share pancake window from Google Hangouts

[georgewsinger]

A description of the problem from Discord:

[CaydenPierce]

It seems that full screen sharing is still in development for Wayland.

From what I understand, the program sharing your screen (the browser) attaches to a D-bus service which provides a pipewire video stream of your screen.

Wayland uses xdg-desktop-portal to request the screenshare stream. xdg-desktop-portal is just a middle-man that talks to the system specific backend impementation for the specific compositor. There is a wlroots version of xdg-desktop-portal backend called xdg-desktopxdg-desktop-portal-wlr. Not sure if this will work out of the box as it relies on XDG_CURRENT_DESKTOP:

$ echo $XDG_CURRENT_DESKTOP                                      
ubuntu:GNOME

$ echo $WAYLAND_DISPLAY                                           ✔
simula-0V

I think XDG_CURRENT_DESKTOP needs to be wlroots for xdg-desktop-portal to work with xdg-desktop-portal-wlr.

@georgewsinger @KaneTW

[CaydenPierce]

Pipewire is not properly integrated into Ubuntu < 21 : https://bugs.launchpad.net/ubuntu/+source/pipewire/+bug/1802533 https://www.reddit.com/r/wayland/comments/lcctwo/screen_sharing_on_wayland/gm01lz2/?utm_source=share&utm_medium=web2x&context=3

So, as a first step, we might want to make sure we can full screen share in Ubuntu on GNOME using Wayland, and then make it work in Simula.

If that's the case, then this issue depends on the GLIBC versioning issue being solved, so we can build Simula for Ubuntu 21

@georgewsinger @KaneTW

[georgewsinger]

Can we provide everything needed (pipewire, xdg-desktop-portal-wlr) via nix?

[CaydenPierce]

Can we provide everything needed (pipewire, xdg-desktop-portal-wlr) via nix?

Potentially... but I think we'll need to change either xdg-desktop-portal to work with Simula or change Simula to be recognized as wlroots by xdg-desktop-portal

Nix packages:

Pipewire - nixpkgs.pipewire, nixpkgs.pipewire_0_2 xdg-desktop-portal - nixpkgs.xdg-desktop-portal (Ubuntu, Debian, Arch (and likely other distros) have this installed (on the system, not Nix ofc) by default) xdg-desktop-portal-wlr - nixpkgs.xdg-desktop-portal-wlr

[CaydenPierce]

A bit of progress here. I changed 'simula-0' as the WAYLAND_DISPLAY in godot-haskell to 'sway' so that xdg-desktop-portal-wlr will recognise Simula as a supported client.

Pipewire runs in Simula successfully so far.

xdg-desktop-portal runs in Simula successfully so far.

xdg-desktop-portal-wlr fails to run with no error on apt install version 0.3.

Downloaded Github release of xdg-desktop-portal-wlr version 0.4 and built from source on Ubuntu 20.10, first run pipewire and xdg-desktop-portal (succesfully), then get this output from xdg-desktop-portal-wlr:

    2021/06/23 10:54:53 [TRACE] - config: trying config file /home/cayden/.config/xdg-desktop-portal-wlr/ubuntu
    2021/06/23 10:54:53 [TRACE] - config: trying config file /home/cayden/.config/xdg-desktop-portal-wlr/GNOME
    2021/06/23 10:54:53 [TRACE] - config: trying config file /home/cayden/.config/xdg-desktop-portal-wlr/config
    2021/06/23 10:54:53 [TRACE] - config: trying config file /usr/local/etc/xdg/xdg-desktop-portal-wlr/ubuntu
    2021/06/23 10:54:53 [TRACE] - config: trying config file /usr/local/etc/xdg/xdg-desktop-portal-wlr/GNOME
    2021/06/23 10:54:53 [TRACE] - config: trying config file /usr/local/etc/xdg/xdg-desktop-portal-wlr/config
    2021/06/23 10:54:53 [ERROR] - config: no config file found
    2021/06/23 10:54:53 [DEBUG] - config: outputname:  (null)
    2021/06/23 10:54:53 [DEBUG] - config: max_fps:  0.000000
    2021/06/23 10:54:53 [DEBUG] - config: exec_before:  (null)
    2021/06/23 10:54:53 [DEBUG] - config: exec_after:  (null)
    2021/06/23 10:54:53 [DEBUG] - config: chooser_cmd: (null)
    2021/06/23 10:54:53 [DEBUG] - config: chooser_type: default
    2021/06/23 10:54:53 [DEBUG] - dbus: connected
    2021/06/23 10:54:53 [DEBUG] - wlroots: wl_display connected
    2021/06/23 10:54:53 [DEBUG] - pipewire: pw_loop created
    2021/06/23 10:54:53 [DEBUG] - pipewire: establishing connection to core
    2021/06/23 10:54:53 [DEBUG] - wlroots: interface to register wl_data_device_manager  (Version: 3)
    2021/06/23 10:54:53 [DEBUG] - wlroots: interface to register wl_shm  (Version: 1)
    2021/06/23 10:54:53 [DEBUG] - wlroots: |-- registered to interface wl_shm (Version 1)
    2021/06/23 10:54:53 [DEBUG] - wlroots: interface to register xdg_wm_base  (Version: 2)
    2021/06/23 10:54:53 [DEBUG] - wlroots: interface to register wl_seat  (Version: 6)
    2021/06/23 10:54:53 [DEBUG] - wlroots: interface to register org_kde_kwin_idle  (Version: 1)
    2021/06/23 10:54:53 [DEBUG] - wlroots: interface to register zwp_idle_inhibit_manager_v1  (Version: 1)
    2021/06/23 10:54:53 [DEBUG] - wlroots: interface to register wl_output  (Version: 3)
    2021/06/23 10:54:53 [DEBUG] - wlroots: |-- registered to interface wl_output (Version 1)
    2021/06/23 10:54:53 [DEBUG] - wlroots: interface to register wl_compositor  (Version: 4)
    2021/06/23 10:54:53 [DEBUG] - wlroots: interface to register wl_subcompositor  (Version: 1)
    2021/06/23 10:54:53 [DEBUG] - wayland: registry listeners run
    2021/06/23 10:54:53 [ERROR] - Compositor doesn't support zxdg_output_manager_v1!
    [1]    30669 segmentation fault (core dumped)  /usr/local/libexec/xdg-desktop-portal-wlr -l TRACE
    ERROR: ld.so: object './submodules/wlroots/build/libwlroots.so.0' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.

Next steps

-Try making a dummy config file -look into the error before seg fault: Compositor doesn't support zxdg_output_manager_v1

@georgewsinger

[CaydenPierce]

Making a config doesn't help, xdg-desktop-portal-wlr gives same error.

I built and installed wayland-info as the xdg-desktop-portal-wlr says it uses wayland-info to get the output: https://gitlab.freedesktop.org/ofourdan/wayland-info

This runs in Simula and gives this output:

interface: 'wl_data_device_manager',                     version:  3, name:  1
interface: 'wl_shm',                                     version:  1, name:  2
    formats: 'XB24'(0x34324258) 'AB24'(0x34324241) XRGB8888 ARGB8888
interface: 'xdg_wm_base',                                version:  2, name:  3
interface: 'wl_seat',                                    version:  6, name:  4
    name: seat0
    capabilities: pointer keyboard
    keyboard repeat rate: 25
    keyboard repeat delay: 600
interface: 'org_kde_kwin_idle',                          version:  1, name:  5
interface: 'zwp_idle_inhibit_manager_v1',                version:  1, name:  6
interface: 'wl_output',                                  version:  3, name:  7
    x: 0, y: 0, scale: 1,
    physical_width: 0 mm, physical_height: 0 mm,
    make: 'Godot', model: 'Godot',
    subpixel_orientation: unknown, output_transform: normal,
    mode:
        width: 2560 px, height: 1440 px, refresh: 90.000 Hz,
        flags: current
interface: 'wl_compositor',                              version:  4, name:  8
interface: 'wl_subcompositor',                           version:  1, name:  9

However still up against the same bug: Compositor doesn't support zxdg_output_manager_v1!

xdg_output_manager is a thing in wlroots. It must be turned off somehow in Simula. I will poke around.

[georgewsinger]

@CaydenPierce

*Plan of attack for getting `xdg-desktop-portal` integrated with SimulaVR.** I just read your bug report, and thought about how I would approach this problem from my perspective. This is actually a challenging project for a first bug, but I believe you can do it (also, we can help):

1. Add ./result/bin/xdg-desktop-portal to Simula.nix

2. Update the xdg-desktop-portal-wlr nix expression to support version 0.4.0, and add to ./result/bin/xdg-desktop-portal-wlr

   • We might need to patch the code here to point to "simula-0" instead of "sway"

3. Add a ./config/xdg-desktop-portal-wlr/config file, and figure out how to xdg-desktop-portal-wlr to point to it on launch (probably a command line flag, or environment variable?).

4. Add calls to readProcess .. "./result/bin/xdg-desktop-portal -r v" and readProcess .. "./result/bin/xdg-desktop-portal-wlr .." at the bottom of ready :: GodotSimulaServer -> IO () in SimulaServer.hs. We probably want verbosity enabled for now (-v) so we can see what's going on. (Also, add the config from step (3)).

5. Either add ./zxdg_output_manager_v1.cpp to SimulaVR/gdwlroots, or add it to ./wlr_xdg_shell.cpp. Populate a basic/minimum working example (could literally be a few lines of code; not sure). Fire it up somewhere/somehow (possibly from our Haskell code, or in our initialization code in ./wlr_xdg_shell.cpp).

6. Add PipeWire 0.3 to Simula.nix, and ensure that ${pipewire}/lib is added to our launch LD_LIBRARY_PATH in ./result/bin/simula_local, etc.

7. Make sure ./result/bin/simula_local is launched with XDG_CURRENT_DESKTOP=simula-0. Not sure if this is sufficient though, but what I'd try first. Some other potentially needed flags: https://github.com/calbrecht/nixpkgs-overlays

@kanetw might have some thoughts as well.

Other things:

• xdg-desktop-portal-wlr won't support Zoom out of the box, so we might have to add support for that via gnome-dbus-emulation-wlr:

  

• I'm slightly worried this project will force us to update our pinned SimulaVR/wlroots. Really hope that isn't the case, but we'll see.

[CaydenPierce]

@CaydenPierce

*Plan of attack for getting `xdg-desktop-portal` integrated with SimulaVR.** I just read your bug report, and thought about how I would approach this problem from my perspective. This is actually a challenging project for a first bug, but I believe you can do it (also, we can help):

1. Add ./result/bin/xdg-desktop-portal to Simula.nix

2. Update the xdg-desktop-portal-wlr nix expression to support version 0.4.0, and add to ./result/bin/xdg-desktop-portal-wlr

   • We might need to patch the code here to point to "simula-0" instead of "sway"
3. Add a ./config/xdg-desktop-portal-wlr/config file, and figure out how to xdg-desktop-portal-wlr to point to it on launch (probably a command line flag, or environment variable?).
4. Add calls to readProcess .. "./result/bin/xdg-desktop-portal -r v" and readProcess .. "./result/bin/xdg-desktop-portal-wlr .." at the bottom of ready :: GodotSimulaServer -> IO () in SimulaServer.hs. We probably want verbosity enabled for now (-v) so we can see what's going on. (Also, add the config from step (3)).
5. Either add ./zxdg_output_manager_v1.cpp to SimulaVR/gdwlroots, or add it to ./wlr_xdg_shell.cpp. Populate a basic/minimum working example (could literally be a few lines of code; not sure). Fire it up somewhere/somehow (possibly from our Haskell code, or in our initialization code in ./wlr_xdg_shell.cpp).
6. Add PipeWire 0.3 to Simula.nix, and ensure that ${pipewire}/lib is added to our launch LD_LIBRARY_PATH in ./result/bin/simula_local, etc.
7. Make sure ./result/bin/simula_local is launched with XDG_CURRENT_DESKTOP=simula-0. Not sure if this is sufficient though, but what I'd try first. Some other potentially needed flags: https://github.com/calbrecht/nixpkgs-overlays

@KaneTW might have some thoughts as well.

Other things:

• xdg-desktop-portal-wlr won't support Zoom out of the box, so we might have to add support for that via gnome-dbus-emulation-wlr:

  

• I'm slightly worried this project will force us to update our pinned SimulaVR/wlroots. Really hope that isn't the case, but we'll see.

Yes, thanks. Steps 1 - 4, 5 make sense to me. Thus far I have just been building from source or installing with apt in Ubuntu. But of course we will want to move it into Nix.

5., I'm unsure how to go about this. I asked in the #sway-devel Libera Chat and emersion said:

"emersion15:03:09
hackalackalot20: it's possible that your use-case doesn't fit wlr_output_layout"

"[15:05:10] <emersion> hackalackalot20: i think wlroots' xdg-output requires a wlr_output_layout right now. i'd be open to allowing people to use it without a layout

[15:05:21] <emersion> basically wlr_xdg_output_manager_v1_create with a NULL layout

[15:05:36] <emersion> patches welcome"

Which from my limited understanding... make me believe that xdg-output, which is essentially an add-on to a wl_output, can only "attach" itself to wlr_output_layout.

Perhaps I just need to go deeper into the wlroots code and actually read where they implement wlr_xdg_output_manager_v1_create

[CaydenPierce]

4. Add calls to readProcess .. "./result/bin/xdg-desktop-portal -r v" and readProcess .. "./result/bin/xdg-desktop-portal-wlr .." at the bottom of ready :: GodotSimulaServer -> IO () in SimulaServer.hs. We probably want verbosity enabled for now (-v) so we can see what's going on. (Also, add the config from step (3)).

@georgewsinger won't this bring up the same issue as our previous glibc issue - i.e. readProcess starts an external process outside of our Nix env?

Thus far I have been using locally built/installed xdg* in Ubuntu and starting manually from terminal within Simula. Obviously that's not a solution either... it's just been working for development.

[KaneTW]

As I mentioned earlier, the glibc issue only arises due to LD_PRELOAD, which we're moving away from.

[CaydenPierce]

Thanks. So calling readProcess will start that process within our nix env?

[KaneTW]

When you launch a nixifed binary it'll automatically refer to its nix env regardless of where or how it's launched.

[CaydenPierce]

Got it.

[CaydenPierce]

1. Add ./result/bin/xdg-desktop-portal to Simula.nix

Done

2. Update the xdg-desktop-portal-wlr nix expression to support version 0.4.0, and add to ./result/bin/xdg-desktop-portal-wlr

Done

This was recently updated to 0.4.0 (not by me). Result added to ./result/bin/xdg-desktop-portal-wl. HOWEVER this gives the same result as running the 0.3 version from Ubuntu reps... fails at early stage... reverting back to using locally built version until this is working then move it in to Nix.

• We might need to patch the code here to point to "simula-0" instead of "sway"

Yes. As a quick fix I have my dev version naming the WAYLAND_DISPLAY=sway, but we will want to add "simula-0" to the xdg-desktop-portal-wlr config

3. Add a ./config/xdg-desktop-portal-wlr/config file, and figure out how to xdg-desktop-portal-wlr to point to it on launch (probably a command line flag, or environment variable?).

Half done.

$XDG_CONFIG_HOME is the environment variable that xdg-desktop-portal-wlr uses to find its config (see man 5 xdg-desktop-portal-wlr). Changing this to ./config breaks the openvr library when loading SteamVR... because SteamVR also uses this environment variable. Fix for now is to put the config file in ~/.config/xdg-desktop-portal-wlr/config, which is where xdg-desktop-portal-wlr will look

4. Add calls to readProcess .. "./result/bin/xdg-desktop-portal -r v" and readProcess .. "./result/bin/xdg-desktop-portal-wlr .." at the bottom of ready :: GodotSimulaServer -> IO () in SimulaServer.hs. We probably want verbosity enabled for now (-v) so we can see what's going on. (Also, add the config from step (3)).

Not yet.

5. Either add ./zxdg_output_manager_v1.cpp to SimulaVR/gdwlroots, or add it to ./wlr_xdg_shell.cpp. Populate a basic/minimum working example (could literally be a few lines of code; not sure). Fire it up somewhere/somehow (possibly from our Haskell code, or in our initialization code in ./wlr_xdg_shell.cpp).

Not yet.

6. Add PipeWire 0.3 to Simula.nix, and ensure that ${pipewire}/lib is added to our launch LD_LIBRARY_PATH in ./result/bin/simula_local, etc.

EDIT:

Not done. Creating a ./result/bin/pipewire and running that in simula will run, but there isn't a long list of events as pipewire detects hardware and such (like there is a long list of events when running pipewire locally installed with apt-get in Ubuntu). Not sure how to make pipewire Nix built executable work the same as the installed pipewire.

7. Make sure ./result/bin/simula_local is launched with XDG_CURRENT_DESKTOP=simula-0. Not sure if this is sufficient though, but what I'd try first. Some other potentially needed flags: https://github.com/calbrecht/nixpkgs-overlays

Not sure where these flags would go... Simula.nix?

[georgewsinger]

Emersion's comments make sense: since we don't have an output_layout construct anywhere, there's no reason to think this will work out of the box. Hmm.

An output_layout in a traditional compositor encodes where all of the rectangular windows are laid out in virtual space (and e.g. computer monitors then occupy portions of that layout space so that users can actually see the windows). We don't have anything like that right now, so wlroots won't know what to send to a screenshare.

Now what we do have is our pancake window display. If we could figure out a way to just directly send that buffer to these screensharing services, then our problems would be solved.