search

sony / flutter-embedded-linux

Embedded Linux embedding for Flutter
BSD 3-Clause "New" or "Revised" License
1.16k stars 122 forks source link

Support multiple wl_seats for remote desktop sharing #409

Closed mmerah closed 1 month ago

mmerah commented 4 months ago

Hello,

I was trying to open a VNC session to interact remotely with a flutter application. However, I've noticed that once a VNC session is opened, touch input on the target does not work anymore and even if the VNC session is closed the touch inputs are still not functioning.

Environment:

I use Torizon OS on an iMX8M Plus, so I have to work through containers. However, it should be reproducible with similar steps on any platform. I have an image with flutter-elinux built into it:

FROM --platform=linux/arm64/v8 torizon/weston-vivante:3 AS runtime
ARG DEBIAN_FRONTEND=noninteractive

# Build environment dependencies for flutter-elinux
RUN apt-get -y update && apt-get install -y \
    build-essential clang cmake \
    curl git pkg-config \
    ninja-build unzip wget \
    libgtk2.0-dev libglib2.0-dev libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev \
    libgl-dev libegl-dev libegl1 libegl1-mesa-dev libgtk-3-dev \
    libxkbcommon-dev libgles2-mesa-dev \
    libwayland-dev wayland-protocols \
    libavfilter8 libopencl1 \
    gstreamer1.0-plugins-base gstreamer1.0-plugins-good gstreamer1.0-plugins-bad gstreamer1.0-plugins-ugly \
    gstreamer1.0-libav gstreamer1.0-tools gstreamer1.0-x gstreamer1.0-alsa \
    gstreamer1.0-gl gstreamer1.0-gtk3 gstreamer1.0-pulseaudio v4l-utils \
    locales

# Set the locale
RUN sed -i '/en_US.UTF-8/s/^# //g' /etc/locale.gen && \
    locale-gen
ENV LANG en_US.UTF-8
ENV LANGUAGE en_US:en
ENV LC_ALL en_US.UTF-8

# Clone Flutter 3.16.0
RUN git clone --branch 3.16.0 https://github.com/sony/flutter-elinux.git

# Add flutter-elinux to the path
ENV PATH="${PATH}:/home/torizon/flutter-elinux/bin"

I run it with a weston image with ENABLE_VNC=1 on the target. Torizon OS uses that environment variable to call weston with --backend=vnc-backend.so:

docker run -e ACCEPT_FSL_EULA=1 -e ENABLE_VNC=1 -d --rm --name=weston --net=host --cap-add CAP_SYS_TTY_CONFIG -v /dev:/dev -v /tmp:/tmp -v /run/udev/:/run/udev/ --device-cgroup-rule='c 4:* rmw' --device-cgroup-rule='c 13:* rmw' --device-cgroup-rule='c 199:* rmw' --device-cgroup-rule='c 226:* rmw' torizon/weston-vivante:$CT_TAG_WESTON_VIVANTE --developer --tty=/dev/tty7

I called the flutter-elinux image flutter-client:

docker run -it --rm --name=flutter -v /tmp:/tmp -v /dev/dri:/dev/dri -v /dev/galcore:/dev/galcore --device-cgroup-rule='c 199:* rmw' --device-cgroup-rule='c 226:* rmw' nddrnd/flutter-client:arm64

Then, I create a sample app, run it to try, build flutter-client with the debug engine .so from the sample and open a VNC session:

flutter-elinux create sample
cd sample
flutter-elinux run -d elinux-wayland
cd ..
git clone https://github.com/sony/flutter-embedded-linux.git
mkdir build
cd build
cp ../../sample/build/elinux/arm64/debug/bundle/lib/* .
cmake -DUSER_PROJECT_PATH=examples/flutter-wayland-client -DCMAKE_BUILD_TYPE=Debug ..
cmake --build .
FLUTTER_LOG_LEVELS=TRACE ./flutter-client -f -b ../../sample/build/elinux/arm64/debug/bundle

While I use Torizon OS with containers, this should be reproducible elsewhere.

Steps to reproduce

  1. I open a VNC session (can be vinagre on Linux, tightVNC on Windows, ...) : 
    [TRACE][elinux_window_wayland.cc(53)] wl_registry_listener.global
[TRACE][elinux_window_wayland.cc(53)] wl_registry_listener.global
[TRACE][elinux_window_wayland.cc(287)] wl_seat_listener.capabilities
[TRACE][elinux_window_wayland.cc(316)] wl_seat_listener.name
[TRACE][elinux_window_wayland.cc(70)] xdg_wm_base_listener.ping
[TRACE][elinux_window_wayland.cc(327)] wl_pointer_listener.enter
[TRACE][elinux_window_wayland.cc(208)] wp_presentation_feedback_listener.sync_output
[TRACE][elinux_window_wayland.cc(221)] wp_presentation_feedback_listener.presented
[TRACE][elinux_window_wayland.cc(373)] wl_pointer_listener.motion
[TRACE][elinux_window_wayland.cc(373)] wl_pointer_listener.motion
[TRACE][elinux_window_wayland.cc(373)] wl_pointer_listener.motion
[TRACE][elinux_window_wayland.cc(373)] wl_pointer_listener.motion
[TRACE][elinux_window_wayland.cc(373)] wl_pointer_listener.motion
[TRACE][elinux_window_wayland.cc(373)] wl_pointer_listener.motion
[TRACE][elinux_window_wayland.cc(373)] wl_pointer_listener.motion
[TRACE][elinux_window_wayland.cc(373)] wl_pointer_listener.motion
[TRACE][elinux_window_wayland.cc(373)] wl_pointer_listener.motion
  2. I move the mouse, click on increment on the flutter app through VNC and then close the session: 
    [TRACE][elinux_window_wayland.cc(373)] wl_pointer_listener.motion
[TRACE][elinux_window_wayland.cc(373)] wl_pointer_listener.motion
[TRACE][elinux_window_wayland.cc(390)] wl_pointer_listener.button
[TRACE][elinux_window_wayland.cc(390)] wl_pointer_listener.button
[TRACE][elinux_window_wayland.cc(70)] xdg_wm_base_listener.ping
[TRACE][elinux_window_wayland.cc(208)] wp_presentation_feedback_listener.sync_output
[TRACE][elinux_window_wayland.cc(221)] wp_presentation_feedback_listener.presented
[TRACE][elinux_window_wayland.cc(208)] wp_presentation_feedback_listener.sync_output
[TRACE][elinux_window_wayland.cc(221)] wp_presentation_feedback_listener.presented
[TRACE][elinux_window_wayland.cc(208)] wp_presentation_feedback_listener.sync_output
[TRACE][elinux_window_wayland.cc(221)] wp_presentation_feedback_listener.presented
[TRACE][elinux_window_wayland.cc(208)] wp_presentation_feedback_listener.sync_output
[TRACE][elinux_window_wayland.cc(221)] wp_presentation_feedback_listener.presented
[TRACE][elinux_window_wayland.cc(208)] wp_presentation_feedback_listener.sync_output
[TRACE][elinux_window_wayland.cc(221)] wp_presentation_feedback_listener.presented
[TRACE][elinux_window_wayland.cc(208)] wp_presentation_feedback_listener.sync_output
[TRACE][elinux_window_wayland.cc(221)] wp_presentation_feedback_listener.presented
[TRACE][elinux_window_wayland.cc(208)] wp_presentation_feedback_listener.sync_output
[TRACE][elinux_window_wayland.cc(221)] wp_presentation_feedback_listener.presented
[TRACE][elinux_window_wayland.cc(208)] wp_presentation_feedback_listener.sync_output
[TRACE][elinux_window_wayland.cc(221)] wp_presentation_feedback_listener.presented
[TRACE][elinux_window_wayland.cc(373)] wl_pointer_listener.motion
[TRACE][elinux_window_wayland.cc(373)] wl_pointer_listener.motion
[TRACE][elinux_window_wayland.cc(60)] wl_registry_listener.global_remove
[TRACE][elinux_window_wayland.cc(60)] wl_registry_listener.global_remove
  3. I try the touchscreen on my device, nothing happens anymore (no logs):
  4. I reopen a VNC session 
    [TRACE][elinux_window_wayland.cc(53)] wl_registry_listener.global
[TRACE][elinux_window_wayland.cc(53)] wl_registry_listener.global
[TRACE][elinux_window_wayland.cc(287)] wl_seat_listener.capabilities
[TRACE][elinux_window_wayland.cc(316)] wl_seat_listener.name
  5. I click with the mouse anywhere in the VNC window multiple times and then close the session: 
    [TRACE][elinux_window_wayland.cc(70)] xdg_wm_base_listener.ping
[TRACE][elinux_window_wayland.cc(70)] xdg_wm_base_listener.ping
[TRACE][elinux_window_wayland.cc(70)] xdg_wm_base_listener.ping
[TRACE][elinux_window_wayland.cc(70)] xdg_wm_base_listener.ping
[TRACE][elinux_window_wayland.cc(70)] xdg_wm_base_listener.ping
[TRACE][elinux_window_wayland.cc(70)] xdg_wm_base_listener.ping
[TRACE][elinux_window_wayland.cc(60)] wl_registry_listener.global_remove
[TRACE][elinux_window_wayland.cc(60)] wl_registry_listener.global_remove

    As you can see, I don't find the wl_pointer_listener.motion and wl_pointer_listener.button events anymore, there is just xdg_wm_base_listener.ping whenever I click somewhere in the VNC window.

For me, this seems similar to a discussion I saw on the Torizon forums (https://community.toradex.com/t/weston-rdp-vnc-no-mouse-input/19841/11) and a patch was mentioned which consists in supporting multiple wl_seats (https://github.com/Igalia/cog/commit/8e8b8218a7133d410ab657ee69b27fb029d85be7). The related issue (https://github.com/Igalia/cog/issues/214) mentions that "Wayland platform module only picks input events from the first event sources (keyboard, touch, pointer, etc.) advertised by the compositor".

I also could reproduce the problem with RDP.

As a side note, both RDP/VNC work fine for me on the same weston backend (in other wayland apps). I don't lose touch input on my device. There is some wayland apps though where the behavior is different (e.g. certain versions of Chromium in a VNC session will not receive any mouse/keyboard input but input on the target keeps functioning).

Best regards,

HidenoriMatsubayashi commented 4 months ago

Thanks for reporting the issue.

HidenoriMatsubayashi commented 4 months ago

It looks the issue is the same as #341.

mmerah commented 4 months ago

It looks the issue is the same as #341.

I agree. It is probably the same. Feel free to close if that's better for you.