search

Xpra-org / xpra

Persistent remote applications for X11; screen sharing for X11, MacOS and MSWindows.
https://xpra.org/
GNU General Public License v2.0
1.98k stars 169 forks source link

shadow/seamless debugging latency #2617

Closed totaam closed 3 years ago

totaam commented 4 years ago

Issue migrated from trac ticket # 2617

component: client | priority: major

2020-03-03 20:31:54: stdedos created the issue

My main issue is that there is unexpected latency in the shadow server:

[[Image(xpra-shadow-high-latency.png)]]

It seems though that it's a "generic" issue: [[Image(xpra-gnome-terminal-latency.png)]] this is an empty gnome terminal, doing nothing. Typing seems like it takes about 3 seconds to manifest.

I have a "50 mbps line" (i.e. 34mbps). Via the VPN (that I have to establish), the connection looks like this [[Image(xpra-shadow-high-latency-vivaldi_2020-03-03_21-36-40.png)]]

Server config:

$ xpra showconfig | grep -P '\((used|default)\)'
xpra for python 2.7 is not installed
 retrying with python3
fake-xinerama         (used)   = 'no'                              <class 'str'>
fake-xinerama        (default) =                                   <class 'NoneType'>
log-file              (used)   = 'display-$DISPLAY-$TIMESTAMP.log'  <class 'str'>
log-file             (default) = '$DISPLAY.log'                    <class 'str'>
microphone            (used)   = 'disabled'                        <class 'str'>
microphone           (default) = 'off'                             <class 'str'>
min-quality           (used)   = 20                                <class 'int'>
min-quality          (default) = 30                                <class 'int'>
min-speed             (used)   = 50                                <class 'int'>
min-speed            (default) = 30                                <class 'int'>
pings                 (used)   = 3                                 <class 'int'>
pings                (default) = 5                                 <class 'int'>
speaker               (used)   = 'disabled'                        <class 'str'>
speaker              (default) = 'on'                              <class 'str'>
start-on-last-client-exit  (used)   = 'bash -c 'echo "now I will touch" ; touch /run/user/1000/xpra-test-`date +%s`*, 'bash -c 'echo "\$DISPLAY=$DISPLAY"*, 'bash -c 'echo "now I will fail!" ; `date`*, 'bash -c 'echo "now I will lock" ; . /run/user/1000/dbus-session; gnome-screensaver-command -l*, 'bash -c 'echo "now I will load/lock" ; env DBUS_SESSION_BUS_ADDRESS="$(cut -f 2- -d= /run/user/1000/dbus-session)" gnome-screensaver-command -l*, 'bash -c 'echo "now I will source/lock" ; . /run/user/1000/dbus-session; gnome-screensaver-command -l*, 'bash -c 'echo "now I will xdg-lock" ; xdg-screensaver lock''  <class 'list'>
start-on-last-client-exit (default) =                                   <class 'list'>
webcam                (used)   = 'no'                              <class 'str'>
webcam               (default) = 'auto'                            <class 'str'>

(idk why fake-xinerama=no, all the others are configured by me server-wide).

Client looks like this:

"Xpra-Python3-x86_64_4.0-[r25345](../commit/9c5bff0dc84d4d4537741f3f282c101ed84a81a7)\xpra_cmd" shadow ssh://user@ip/0 --ssh="plink -ssh -agent" --opengl=no --desktop-scaling=0.75 --min-speed=70 --webcam=no --speaker=off --microphone=off --pulseaudio=no --exit-with-client=no

2020-03-03 21:07:51,309 Xpra GTK3 client version 4.0-[r25345](../commit/9c5bff0dc84d4d4537741f3f282c101ed84a81a7) 64-bit
2020-03-03 21:07:51,311  running on Microsoft Windows 10
2020-03-03 21:07:52,491 GStreamer version 1.16.2 for Python 3.8.1 64-bit
2020-03-03 21:07:52,819 keyboard layout code 0x409
2020-03-03 21:07:52,820 identified as 'United States - English' : us
2020-03-03 21:07:53,413  keyboard settings: layout=us
2020-03-03 21:07:53,418  desktop size is 1600x900 with 1 screen:
2020-03-03 21:07:53,420   Default (423x238 mm - DPI: 96x96) workarea: 1600x860
2020-03-03 21:07:53,422     (Standard monitor types) Generic PnP Monitor (309x174 mm - DPI: 131x131)
2020-03-03 21:07:53,423  downscaled to 75%, virtual screen size: 2133x1200
2020-03-03 21:07:53,425   Default (423x238 mm - DPI: 128x128) workarea: 2133x1147
2020-03-03 21:07:53,430     (Standard monitor types) Generic PnP Monitor (309x174 mm - DPI: 175x175)
2020-03-03 21:08:11,951 enabled remote logging
2020-03-03 21:08:11,957 Xpra GTK3 shadow server version 3.0.6-[r25174](../commit/86fdb78c7bf461e68008d62bac8a4bf54dd5f12c) 64-bit
2020-03-03 21:08:11,960  running on Linux Ubuntu 16.04 xenial
2020-03-03 21:08:11,963  remote desktop size is 6400x1440

(xpra_cmd:11928): Pango-WARNING **: 21:08:13.287: couldn't load font "Bitstream Vera Sans Not-Rotated 14.662109375", falling back to "Sans Not-Rotated 14.662109375", expect ugly output.
2020-03-03 21:08:26,014 server is not responding, drawing spinners over the windows
2020-03-03 21:08:27,035 server is OK again

(xpra_cmd:11928): Gtk-WARNING **: 21:09:57.302: Could not find the icon 'pan-end-symbolic'. The 'hicolor' theme
was not found either, perhaps you need to install it.
You can get a copy from:
        http://icon-theme.freedesktop.org/releases
2020-03-03 21:10:01,305 UI thread is now blocked
2020-03-03 21:10:01,763 UI thread is running again, resuming
2020-03-03 21:10:05,796 server is not responding, drawing spinners over the windows
2020-03-03 21:10:20,335 server is not responding, drawing spinners over the windows
2020-03-03 21:10:20,848 server is OK again
2020-03-03 21:10:21,000 server is OK again
2020-03-03 21:13:48,414 Warning: long timer waiting time,
2020-03-03 21:13:48,416  UI thread polling waited 27.0 seconds longer than intended (27.5 vs 0.5)

(xpra_cmd:11928): Gtk-WARNING **: 21:13:48.433: Error loading theme icon 'document-save' for stock: Icon 'document-save' not present in theme Adwaita

(xpra_cmd:11928): Gtk-WARNING **: 21:13:48.597: Error loading theme icon 'document-save' for stock: Icon 'document-save' not present in theme Adwaita

(xpra_cmd:11928): Gtk-WARNING **: 21:13:48.915: Error loading theme icon 'document-save' for stock: Icon 'document-save' not present in theme Adwaita

(xpra_cmd:11928): Gtk-WARNING **: 21:13:58.705: Error loading theme icon 'document-save' for stock: Icon 'document-save' not present in theme Adwaita

(xpra_cmd:11928): Gtk-WARNING **: 21:13:58.739: Error loading theme icon 'document-save' for stock: Icon 'document-save' not present in theme Adwaita

(xpra_cmd:11928): Gtk-WARNING **: 21:13:58.796: Error loading theme icon 'document-save' for stock: Icon 'document-save' not present in theme Adwaita

(xpra_cmd:11928): Gtk-WARNING **: 21:13:58.844: Error loading theme icon 'document-save' for stock: Icon 'document-save' not present in theme Adwaita

(xpra_cmd:11928): Gtk-WARNING **: 21:13:58.907: Error loading theme icon 'document-save' for stock: Icon 'document-save' not present in theme Adwaita

(xpra_cmd:11928): Gtk-WARNING **: 21:14:15.331: Error loading theme icon 'document-save' for stock: Icon 'document-save' not present in theme Adwaita

(xpra_cmd:11928): Gtk-WARNING **: 21:14:15.646: Error loading theme icon 'document-save' for stock: Icon 'document-save' not present in theme Adwaita

(xpra_cmd:11928): Gtk-WARNING **: 21:14:15.709: Error loading theme icon 'document-save' for stock: Icon 'document-save' not present in theme Adwaita

(xpra_cmd:11928): Gtk-WARNING **: 21:14:15.742: Error loading theme icon 'document-save' for stock: Icon 'document-save' not present in theme Adwaita

(xpra_cmd:11928): Gtk-WARNING **: 21:14:15.759: Error loading theme icon 'document-save' for stock: Icon 'document-save' not present in theme Adwaita

(xpra_cmd:11928): Gtk-WARNING **: 21:14:15.793: Error loading theme icon 'document-save' for stock: Icon 'document-save' not present in theme Adwaita

(xpra_cmd:11928): Gtk-WARNING **: 21:14:15.813: Error loading theme icon 'document-save' for stock: Icon 'document-save' not present in theme Adwaita

(xpra_cmd:11928): Gtk-WARNING **: 21:14:15.832: Error loading theme icon 'document-save' for stock: Icon 'document-save' not present in theme Adwaita

Side issues you might want to look into:

  • The bajillion "missing icons". Both the logs and the actual missing icons. r25468 does not log anything about missing icons, but they are still missing [[Image(Xpra-save-graph_cmd_2020-03-03_22-24-26.png)]]
  • That message [[Image(Xpra_cmd_2020-03-03_21-19-19.png)]]
  • Saving the graph in r25345 froze everything for ~ 30 seconds (^^^^)
  • Add the filename extension when saving in Windows. Basically, unless the file is saved as *.png (in this case), append .png (even if someone saves with e.g. .jpg)
totaam commented 4 years ago

2020-03-03 20:32:17: stdedos uploaded file Xpra_cmd_2020-03-03_21-19-19.png (31.4 KiB)

Xpra_cmd_2020-03-03_21-19-19.png

totaam commented 4 years ago

2020-03-03 20:32:26: stdedos uploaded file xpra-gnome-terminal-latency.png (16.8 KiB)

xpra-gnome-terminal-latency.png

totaam commented 4 years ago

2020-03-03 20:32:35: stdedos uploaded file Xpra-save-graph_cmd_2020-03-03_22-24-26.png (31.9 KiB)

Xpra-save-graph_cmd_2020-03-03_22-24-26.png

totaam commented 4 years ago

2020-03-03 20:32:44: stdedos uploaded file xpra-shadow-high-latency.png (23.8 KiB)

xpra-shadow-high-latency.png

totaam commented 4 years ago

2020-03-03 20:32:54: stdedos uploaded file xpra-shadow-high-latency-vivaldi_2020-03-03_21-36-40.png (35.3 KiB)

xpra-shadow-high-latency-vivaldi_2020-03-03_21-36-40.png

totaam commented 4 years ago

2020-03-03 20:34:57: stdedos commented

Please replace the second It seems though that it's a "generic" issue: [[Image with this one xpra-gnome-terminal-latency.png

(Another) Side issue:

  • Add the filename extension when saving in Windows. Basically, unless the file is saved as *.png (in this case), append .png (even if someone saves with e.g. .jpg)
totaam commented 4 years ago

(idk why fake-xinerama=no, all the others are configured by me server-wide).

Fixed in r25485.

this is an empty gnome terminal, doing nothing. Typing seems like it takes about 3 seconds to manifest.

Please attach xpra info for the misbehaving session.

UI thread polling waited 27.0 seconds longer than intended (27.5 vs 0.5)

That's not normal at all.

The bajillion "missing icons". Both the logs and the actual missing icons. r25468 does not log anything about missing icons, but they are still missing

That has been fixed in r25463 and tested repeatedly on a blank win10 VM, again today with r25483.

Please replace the second...

I don't understand. You should now have permission to edit the ticket yourself.

totaam commented 4 years ago

2020-03-04 05:43:48: stdedos commented

Now I noticed it that the functionality only saves one graph (not both of them)

totaam commented 4 years ago

Saving the graph in r25345 froze everything for ~ 30 seconds (^^^^)

r25486 moves the IO to a thread, but the write_to_png function will still be running in the main thread - and that may be the culprit (PNG is a CPU pig - though 30s is certainly excessive)

Add the filename extension when saving in Windows

Done for all platforms in r25487

totaam commented 4 years ago

2020-03-05 21:24:37: stdedos commented

Since the server crashed that night (and subsequent debugging tonight did not give more than ~500ms latency), I'll call this my fault.

Apologies.

totaam commented 4 years ago

2020-03-31 11:38:23: stdedos commented

I'll re-use this for investigating a seamless session: And xpra-info attaching soon

totaam commented 4 years ago

2020-03-31 11:38:45: stdedos uploaded file communication-4.png (16.9 KiB)

communication-4.png

totaam commented 4 years ago

2020-03-31 11:38:56: stdedos uploaded file latency-4.png (7.2 KiB)

latency-4.png

totaam commented 4 years ago

2020-03-31 11:57:52: stdedos uploaded file redact-xpra-2617-20.log (442.7 KiB)

totaam commented 4 years ago

2020-04-16 08:24:40: stdedos commented

@Antoine, apologies for "mentioning". I know you are busy; I was wondering if you could pitch in (since I don't know what to look for).

It's annoying that e.g. Teamviewer would work "seamlessly", under any connection. I know I cannot update to v4 (vp9 encoding and stuff), but I guess something could be done 😕

totaam commented 4 years ago

2020-04-17 14:58:33: stdedos commented

Graph is saying that: [[Image(latency-broken.png)]]

But that's not the image I am getting. Client goes in-and-out of:

2020-04-17 16:53:31,301 server is not responding, drawing spinners over the windows
2020-04-17 16:53:31,556 server is OK again

And this is how server feels like:

>ping 172.16.57.121

Pinging 172.16.57.121 with 32 bytes of data:
Reply from 172.16.57.121: bytes=32 time=19ms TTL=61
Reply from 172.16.57.121: bytes=32 time=31ms TTL=61
Reply from 172.16.57.121: bytes=32 time=18ms TTL=61
Reply from 172.16.57.121: bytes=32 time=17ms TTL=61

Ping statistics for 172.16.57.121:
    Packets: Sent # 4, Received4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum # 17ms, Maximum31ms, Average = 21ms

[[Image(netdata_cpu-utilization_vivaldi_2020-04-17_16-54-59.png)]]

(and this was followed by the :20 monitor dying, while :2, :200 are still alive, well and stable)

totaam commented 4 years ago

2020-04-17 14:58:48: stdedos uploaded file netdata_cpu-utilization_vivaldi_2020-04-17_16-54-59.png (50.0 KiB)

netdata_cpu-utilization_vivaldi_2020-04-17_16-54-59.png

totaam commented 4 years ago

2020-04-17 14:59:01: stdedos uploaded file latency-broken.png (7.4 KiB)

latency-broken.png

totaam commented 4 years ago

2020-04-22 14:57:10: antoine commented

Have you tried setting a bandwidth-limit lower than what your ISP gives you? Maybe 10Mbps?

totaam commented 4 years ago

2020-04-23 08:38:05: stdedos commented

From a recently spawned shadow server:

"Xpra-Python3-x86_64_4.0-26124\xpra_cmd" shadow ssh://user@ip/0 --ssh="plink -ssh -agent" --opengl=no --bandwidth-limit=10Mbps --desktop-scaling=0.75 --min-speed=70 --webcam=no --speaker=off --microphone=off --pulseaudio=no

2020-04-23 10:22:50,479 Xpra GTK3 client version 4.0-26124 64-bit
2020-04-23 10:22:50,482  running on Microsoft Windows 10
2020-04-23 10:22:51,726 GStreamer version 1.16.2 for Python 3.8.2 64-bit
2020-04-23 10:22:52,129 keyboard layout code 0x409
2020-04-23 10:22:52,129 identified as 'United States - English' : us
2020-04-23 10:22:52,629  keyboard settings: layout=us
2020-04-23 10:22:52,635  desktop size is 4160x1440 with 1 screen:
2020-04-23 10:22:52,636   Default (1100x381 mm - DPI: 96x96) workarea: 4160x1400
2020-04-23 10:22:52,636     Generic PnP Monitor 1600x900 at 0x534 (309x174 mm - DPI: 131x131) workarea: 1600x860
2020-04-23 10:22:52,636     C32JG5x 2560x1440 at 1600x0 (697x392 mm - DPI: 93x93) workarea: 2560x1400
2020-04-23 10:22:52,637  downscaled to 75%, virtual screen size: 5547x1920
2020-04-23 10:22:52,637   Default (1100x381 mm - DPI: 128x128) workarea: 5547x1867
2020-04-23 10:22:52,637     Generic PnP Monitor 2133x1200 at 0x712 (309x174 mm - DPI: 175x175) workarea: 2133x1147
2020-04-23 10:22:52,637     C32JG5x 3413x1920 at 2133x0 (697x392 mm - DPI: 124x124) workarea: 3413x1867
2020-04-23 10:23:14,977 enabled remote logging
2020-04-23 10:23:14,979 Xpra GTK3 shadow server version 3.0.9-26132 64-bit
2020-04-23 10:23:14,981  running on Linux Ubuntu 16.04 xenial
2020-04-23 10:23:14,983  remote desktop size is 6400x1440

(xpra_cmd:15532): Pango-WARNING **: 10:23:15.571: couldn't load font "Bitstream Vera Sans Not-Rotated 14.662109375", falling back to "Sans Not-Rotated 14.662109375", expect ugly output.
2020-04-23 10:23:21,598 server is not responding, drawing spinners over the windows
2020-04-23 10:23:23,404 server is OK again
2020-04-23 10:24:53,455 server is not responding, drawing spinners over the windows
2020-04-23 10:24:57,222 server is OK again
2020-04-23 10:24:58,473 server is not responding, drawing spinners over the windows
2020-04-23 10:25:03,498 server is OK again
2020-04-23 10:25:29,915 server is not responding, drawing spinners over the windows
2020-04-23 10:25:37,716 server is OK again
2020-04-23 10:25:43,727 server is not responding, drawing spinners over the windows
2020-04-23 10:25:47,506 server is OK again
2020-04-23 10:25:57,928 server is not responding, drawing spinners over the windows
2020-04-23 10:26:00,705 server is OK again
2020-04-23 10:26:06,139 server is not responding, drawing spinners over the windows
2020-04-23 10:26:17,529 server is OK again
2020-04-23 10:26:23,448 server is not responding, drawing spinners over the windows
2020-04-23 10:26:42,065 server is OK again
2020-04-23 10:26:48,506 server is not responding, drawing spinners over the windows
2020-04-23 10:27:06,663 server is OK again
2020-04-23 10:27:06,833 server is not responding, drawing spinners over the windows
2020-04-23 10:27:07,592 server is OK again
2020-04-23 10:27:13,000 server is not responding, drawing spinners over the windows
2020-04-23 10:27:33,177 server is OK again
2020-04-23 10:27:39,086 server is not responding, drawing spinners over the windows
2020-04-23 10:28:01,005 server is OK again
2020-04-23 10:28:06,173 server is not responding, drawing spinners over the windows
2020-04-23 10:28:18,255 server is OK again
2020-04-23 10:28:24,203 server is not responding, drawing spinners over the windows
2020-04-23 10:28:32,294 server is OK again
2020-04-23 10:28:38,224 server is not responding, drawing spinners over the windows
2020-04-23 10:28:46,342 server is OK again
2020-04-23 10:28:51,507 server is not responding, drawing spinners over the windows
2020-04-23 10:29:22,225 server is OK again
2020-04-23 10:29:27,896 server is not responding, drawing spinners over the windows
2020-04-23 10:29:38,029 server is OK again
2020-04-23 10:30:03,002 server is not responding, drawing spinners over the windows
2020-04-23 10:30:46,846 server is not responding, drawing spinners over the windows
2020-04-23 10:30:47,762 server is OK again
2020-04-23 10:30:47,925 server is OK again
2020-04-23 10:30:53,089 server is not responding, drawing spinners over the windows
2020-04-23 10:31:06,193 server is OK again
2020-04-23 10:31:11,471 server is not responding, drawing spinners over the windows
2020-04-23 10:31:32,258 server is OK again
2020-04-23 10:31:37,200 server is not responding, drawing spinners over the windows
2020-04-23 10:31:47,786 server is OK again
2020-04-23 10:33:44,771 unknown string message: 0xc245 / 0x0 / 0x0
2020-04-23 10:35:36,627 server is not responding, drawing spinners over the windows
2020-04-23 10:35:37,153 server is OK again
2020-04-23 10:35:38,642 server is not responding, drawing spinners over the windows
2020-04-23 10:35:40,153 server is OK again
2020-04-23 10:36:06,150 unknown string message: 0xc245 / 0x0 / 0x0

[[Image(Xpra-2617_cmd_2020-04-23_10-30-18.png)]] [[Image(Xpra-2617_cmd_2020-04-23_10-31-01.png)]]

totaam commented 4 years ago

2020-04-23 08:38:18: stdedos uploaded file Xpra-2617_cmd_2020-04-23_10-30-18.png (32.1 KiB)

Xpra-2617_cmd_2020-04-23_10-30-18.png

totaam commented 4 years ago

2020-04-23 08:38:27: stdedos uploaded file Xpra-2617_cmd_2020-04-23_10-31-01.png (26.5 KiB)

Xpra-2617_cmd_2020-04-23_10-31-01.png

totaam commented 4 years ago

2020-04-23 08:38:38: stdedos uploaded file redact-xpra-2617_2020_04_23_10_32_35.log (341.3 KiB)

totaam commented 4 years ago

From the graphs on comment:11, your server latency spiked to 48 seconds! So something is very very wrong here. It would be interesting to see how much bandwidth xpra was using when things collapsed. From your graph, it was below 16Mbps... This could also be related to the VPN: maybe it's pushing back and not prioritising the xpra traffic enough. My guess is that you should lower the bandwidth limit even more to try to make things work.

I would need to be able to reproduce the problem to have a chance to fix this.

totaam commented 4 years ago

2020-04-23 11:34:38: stdedos commented

Replying to [comment:12 Antoine Martin]:

From the graphs on comment:11, your server latency spiked to 48 seconds! So something is very very wrong here. It would be interesting to see how much bandwidth xpra was using when things collapsed. From your graph, it was below 16Mbps...

Can you do dual-axis data amount graph? It's not straightforward to translate KB/s to Mbits (and both may be needed)

This could also be related to the VPN: maybe it's pushing back and not prioritising the xpra traffic enough. My guess is that you should lower the bandwidth limit even more to try to make things work.

A crude iperf speedtest to my workstation clocks it at ~8 Mbit. I guess I can try to set it to 6 Mbit for the shadow server, but all of that feels wrong 😕

I know that I have a v3 server, and extremely big screen to forward. I don't understand why xpra does not / can not lower its quality characteristics to attempt to keep up (server downscaling, color depth etc)

It's true I've set min-quality=30, min-speed=50 - I don't know what that means (I've talked about it on a separate ticket I think), but it made some kind of sense visually/responsive-ly at the time. I think that xpra would be able to keep up with quality=30, speed=50.

I would need to be able to reproduce the problem to have a chance to fix this.

totaam commented 4 years ago

Can you do dual-axis data amount graph? It's not straightforward to translate KB/s to Mbits (and both may be needed)

Wouldn't it be better to just show the bandwidth in bps? (Mbps / Kbps) And leave the pixels/s alone. Is that what you mean by dual-axis?

I don't understand why xpra does not / can not lower its quality characteristics to attempt to keep up (server downscaling, color depth etc)

It should. I don't understand why it doesn't.

It's true I've set min-quality=30, min-speed=50 - I don't know what that means..

Try min-quality=0, min-speed=0. This will compress harder, at a lower quality, which will use less bandwidth.

Two more things worth trying:

Important: do this locally, env vars are not forwarded with ssh remote starts.

totaam commented 4 years ago

2020-04-28 18:05:28: stdedos commented

Replying to [comment:14 Antoine Martin]:

Can you do dual-axis data amount graph? It's not straightforward to translate KB/s to Mbits (and both may be needed) Wouldn't it be better to just show the bandwidth in bps? (Mbps / Kbps) And leave the pixels/s alone. Is that what you mean by dual-axis?

I mean that, on one side you have b/s and on the other bps. I see your point about having only bps - the only annoying thing is when you are trying to translate the units on-the-fly.

Sometimes you are counting size (i.e. bytes), and some times bandwidth (in which "unit-franca" is bps)

I am not saying I have a solid case with b/s, but the fact that conversion is complicated, makes me want both.

I don't understand why xpra does not / can not lower its quality characteristics to attempt to keep up (server downscaling, color depth etc) It should. I don't understand why it doesn't.

It's true I've set min-quality=30, min-speed=50 - I don't know what that means.. Try min-quality=0, min-speed=0. This will compress harder, at a lower quality, which will use less bandwidth.

I can try that.

Two more things worth trying:

  • XPRA_SHADOW_REFRESH_DELAY=200 xpra shadow ... (limits the refresh rate to 5fps - you may even try to increase it even higher)
  • XPRA_MAX_SOFT_EXPIRED=0 xpra shadow ... (limits how many frames we send without receiving the echo)

Important: do this locally, env vars are not forwarded with ssh remote starts.

Doesn't --env= work with those? It is more straightforward than two-step connection.

totaam commented 4 years ago

2020-04-29 08:06:17: stdedos commented

from xpra info

 124   │ 2020-04-29 10:03:30,053 New unix-domain connection received
 125   │ 2020-04-29 10:03:30,053  on '/run/xpra/user-precision-t3620-0'
 126   │ Traceback (most recent call last):
 127   │   File "/usr/lib/python3/dist-packages/xpra/x11/gtk_x11/clipboard.py", line 159, in do_xpra_selection_request
 128   │     proxy.do_selection_request_event(event)
 129   │   File "/usr/lib/python3/dist-packages/xpra/x11/gtk_x11/clipboard.py", line 359, in do_selection_request_event
 130   │     assert requestor
 131   │ AssertionError
 132   │ 2020-04-29 10:03:30,101 New unix-domain connection received
 133   │ 2020-04-29 10:03:30,101  on '/run/user/1000/xpra/user-precision-t3620-0'
 134   │ Traceback (most recent call last):
 135   │   File "/usr/lib/python3/dist-packages/xpra/x11/gtk_x11/clipboard.py", line 159, in do_xpra_selection_request
 136   │     proxy.do_selection_request_event(event)
 137   │   File "/usr/lib/python3/dist-packages/xpra/x11/gtk_x11/clipboard.py", line 359, in do_selection_request_event
 138   │     assert requestor
 139   │ AssertionError
 140   │ Traceback (most recent call last):
 141   │   File "/usr/lib/python3/dist-packages/xpra/x11/gtk_x11/clipboard.py", line 159, in do_xpra_selection_request
 142   │     proxy.do_selection_request_event(event)
 143   │   File "/usr/lib/python3/dist-packages/xpra/x11/gtk_x11/clipboard.py", line 359, in do_selection_request_event
 144   │     assert requestor
 145   │ AssertionError
 146   │ Traceback (most recent call last):
 147   │   File "/usr/lib/python3/dist-packages/xpra/x11/gtk_x11/clipboard.py", line 159, in do_xpra_selection_request
 148   │     proxy.do_selection_request_event(event)
 149   │   File "/usr/lib/python3/dist-packages/xpra/x11/gtk_x11/clipboard.py", line 359, in do_selection_request_event
 150   │     assert requestor
 151   │ AssertionError
totaam commented 4 years ago

Doesn't --env= work with those?

It should. You can verify it did get applied with xpra info.

assert requestor

That's weird, r26192 converts this to a warning.

totaam commented 4 years ago

2020-04-29 08:41:51: stdedos commented

Server just dies over and over, it's so frustrating to keep it up

"Xpra-Python3-x86_64_4.0-26124\xpra_cmd" shadow ssh://user@ip/0 --ssh="plink -ssh -agent" --env=XPRA_MAX_SOFT_EXPIRED=0 --opengl=no --bandwidth-limit=6Mbps --desktop-scaling=0.75 --min-speed=70 --webcam=no --speaker=off --microphone=off --pulseaudio=no

and server goes

  24   │ 2020-04-29 10:34:58,152 GStreamer version 1.8.3 for Python 3.5.2 64-bit
  25   │ 2020-04-29 10:34:58,164 Warning: failed to query pulseaudio using 'pactl info'
  26   │ 2020-04-29 10:34:58,164  Connection failure: Connection refused
  27   │ 2020-04-29 10:34:58,165  pa_context_connect() failed: Connection refused
  28   │ 2020-04-29 10:35:01,184 Warning: cannot load dbus helper:
  29   │ 2020-04-29 10:35:01,184  org.freedesktop.DBus.Error.NoServer
  30   │ 2020-04-29 10:35:01,184  Failed to connect to socket /tmp/dbus-253Ig9KTwQ
  31   │ 2020-04-29 10:35:01,184  Connection refused
  32   │ 2020-04-29 10:35:01,192 printer forwarding enabled using pdf and postscript
  33   │ 2020-04-29 10:35:01,193 Warning: cannot watch for application menu changes without pyinotify:
  34   │ 2020-04-29 10:35:01,193  No module named 'pyinotify'
  35   │ 2020-04-29 10:35:01,210 xpra is ready.
  36   │ 2020-04-29 10:35:01,210 xpra GTK3 shadow version 3.0.9-26132 64-bit
  37   │ 2020-04-29 10:35:01,454  uid=1000 (usr), gid=1000 (usr)
  38   │ 2020-04-29 10:35:01,454  running with pid 14800 on Linux Ubuntu 16.04 xenial
  39   │ 2020-04-29 10:35:01,455  on display ':0' of size 6400x1440
  40   │ 2020-04-29 10:35:01,455  with 3 monitors:
  41   │ 2020-04-29 10:35:01,455   DP-0             2560x1440 at    0,0   
  42   │ 2020-04-29 10:35:01,455   DP-2             1920x1080 at 2560,180 
  43   │ 2020-04-29 10:35:01,455   DP-4             1920x1080 at 4480,180 
  44   │ 2020-04-29 10:35:01,540 New unix-domain connection received
  45   │ 2020-04-29 10:35:01,541  on '/run/xpra/usr-precision-t3620-0'
  46   │ 2020-04-29 10:35:01,668 New unix-domain connection received
  47   │ 2020-04-29 10:35:01,679  on '/run/xpra/usr-precision-t3620-0'
  48   │ 2020-04-29 10:35:01,739 Handshake complete; enabling connection
  49   │ 2020-04-29 10:35:02,231 New unix-domain connection received
  50   │ 2020-04-29 10:35:02,232  on '/run/xpra/usr-precision-t3620-0'
  51   │ 2020-04-29 10:35:02,237 New unix-domain connection received
  52   │ 2020-04-29 10:35:02,237  on '/run/user/1000/xpra/usr-precision-t3620-0'
  53   │ 2020-04-29 10:35:02,261 Warning: icon is quite large (273 KB):
  54   │ 2020-04-29 10:35:02,261  '/usr/share/pixmaps/com.visualstudio.code.insiders.png'
  55   │ 2020-04-29 10:35:03,010 62.7GB of system memory
  56   │ 2020-04-29 10:35:04,987 Error setting up client dbus instance:
  57   │ 2020-04-29 10:35:04,988   org.freedesktop.DBus.Error.NoServer
  58   │ 2020-04-29 10:35:04,988   Failed to connect to socket /tmp/dbus-DTfoSfnVTg
  59   │ 2020-04-29 10:35:04,988   Connection refused
  60   │ 2020-04-29 10:35:04,989  automatic picture encoding enabled, also available:
  61   │ 2020-04-29 10:35:04,989   rgb24, rgb32
  62   │ 2020-04-29 10:35:04,990 Python/GTK3 Microsoft Windows 10 aero client version 4.0-26124 64-bit
  63   │ 2020-04-29 10:35:04,990  connected
  64   │ 2020-04-29 10:35:04,991 Warning: remote clipboard request timed out
  65   │ 2020-04-29 10:35:04,991  request id 0, selection=CLIPBOARD, target=TARGETS
  66   │ 2020-04-29 10:35:04,996 shadow server: setting default keymap translation
  67   │ 2020-04-29 10:35:05,003  client root window size is 5547x1920
  68   │ 2020-04-29 10:35:05,053 client   1 @23.198 Xpra GTK3 shadow server version 3.0.9-26132 64-bit
  69   │ 2020-04-29 10:35:05,062 client   1 @23.199  running on Linux Ubuntu 16.04 xenial
  70   │ 2020-04-29 10:35:05,062 client   1 @23.199  remote desktop size is 6400x1440
  71   │ 2020-04-29 10:35:06,610 Warning: not adding duplicate printer 'OneNote'
  72   │ 2020-04-29 10:35:06,610 Warning: not adding duplicate printer 'Send To OneNote 16'
  73   │ 2020-04-29 10:35:06,611 Warning: not adding duplicate printer 'Microsoft Print to PDF'
  74   │ 2020-04-29 10:35:45,112 Warning: remote clipboard request timed out
  75   │ 2020-04-29 10:35:45,113  request id 2, selection=CLIPBOARD, target=UTF8_STRING
  76   │ 2020-04-29 10:36:55,764 client   1 @30.081 server is not responding, drawing spinners over the windows
  77   │ 2020-04-29 10:36:55,769 client   1 @03.359 server is OK again
  78   │ 2020-04-29 10:36:55,769 client   1 @05.104 server is not responding, drawing spinners over the windows
  79   │ 2020-04-29 10:36:55,770 client   1 @05.358 server is OK again
  80   │ 2020-04-29 10:36:55,770 client   1 @13.241 server is not responding, drawing spinners over the windows
  81   │ 2020-04-29 10:36:55,819 Warning: timeout on screen updates for window 2,
  82   │ 2020-04-29 10:36:55,819  already delayed for more than 108 seconds
  83   │ 2020-04-29 10:36:57,312 Warning: request id 2 not found
  84   │ 2020-04-29 10:36:57,312  timed out already?
  85   │ 2020-04-29 10:36:57,318 xpra client 1 disconnected.
  86   │ 2020-04-29 10:37:36,709 ignoring token for disabled clipboard 'CLIPBOARD'
  87   │ 2020-04-29 10:37:36,709 ignoring token for disabled clipboard 'CLIPBOARD'
  88   │ 2020-04-29 10:37:36,720 Last client has disconnected, terminating
  89   │ 2020-04-29 10:37:36,879 removing socket '/run/user/1000/xpra/usr-precision-t3620-0'
  90   │ 2020-04-29 10:37:36,879 removing socket '/run/xpra/usr-precision-t3620-0'
  91   │ xpra initialization error:
  92   │  cannot find live server for display :0
  93   │ xpra initialization error:
  94   │  cannot find live server for display :0
totaam commented 4 years ago

2020-04-29 09:02:43: stdedos commented

It seems that only XPRA_SHADOW_REFRESH_DELAY does the thing for me - with 2 out of 3 monitors/windows minimized at all times. Which is working really superb-ly

However, I have so many objections to what's happening.

This cannot be "very low quality": [[Image(xpra-shadow_Xpra_cmd_2020-04-29_10-43-22.png)]] [[Image(xpra-2617-all-shadow.png)]]

and definitely not 5fps (It would be too much to shoot a video, I guess you can take my word for it). From gaming experience, that would be a table-flip, not a seamless experience (disregarding the 1/3 monitors available issue).

Even though it seems applied:

$ xpra info :0 | grep XPRA_SHADOW_REFRESH_DELAY
env.XPRA_SHADOW_REFRESH_DELAY=200

However, I cannot find it by "conventional" means:

$ ps faux | grep '/usr/bin/python3 /usr/bin/xpra' | grep :0
sntentos  3027  0.0  0.0 276660 23188 ?        Sl   09:50   0:01  |           \_ /usr/bin/python3 /usr/bin/xpra _proxy :2
sntentos 22382  0.3  0.0 276664 23492 ?        Sl   10:41   0:01              \_ /usr/bin/python3 /usr/bin/xpra _proxy_shadow_start :0 --opengl=no --bandwidth-limit=6Mbps --ssh=plink -ssh -agent --desktop-scaling=0.75 --speaker=off --webcam=no --min-speed=70 --env=XPRA_SHADOW_REFRESH_DELAY=200
sntentos  3487  0.0  0.0  15456   984 pts/25   S+   10:51   0:00  |   \_ grep --color=auto /usr/bin/python3 /usr/bin/xpra
sntentos  4859  0.0  0.0 276664 16900 ?        Sl   Απρ14   0:04  \_ /usr/bin/python3 /usr/bin/xpra _proxy :2
sntentos  7754  0.0  0.0 276664 16964 ?        Sl   Απρ16   0:04  \_ /usr/bin/python3 /usr/bin/xpra _proxy :2
sntentos 25943  0.0  0.0 276664 18480 ?        Sl   Απρ20   0:00  \_ /usr/bin/python3 /usr/bin/xpra _proxy :2
sntentos 26512  0.0  0.0 276664 18568 ?        Sl   Απρ22   0:02  \_ /usr/bin/python3 /usr/bin/xpra _proxy :200
sntentos 14542  0.0  0.0 276676 18356 ?        Sl   Απρ22   0:07  \_ /usr/bin/python3 /usr/bin/xpra _proxy :2
root     22101  0.0  0.0 325340 27644 ?        Ssl  Απρ23   0:01 /usr/bin/python3 /usr/bin/xpra proxy :14500 --daemon=no --bind-tcp=0.0.0.0:14500 --tcp-auth=sys --ssl-cert=/etc/xpra/ssl-cert.pem --ssl=on --bind=/run/xpra/system --auth=peercred --socket-dirs=/run/xpra --socket-permissions=666 --log-dir=/var/log --pidfile=/run/xpra.pid --debug=
sntentos 31999  0.0  0.0 276672 23492 ?        Sl   Απρ28   0:02  \_ /usr/bin/python3 /usr/bin/xpra _proxy :20
sntentos 22414 68.6  0.7 2068812 462560 ?      Sl   10:41   6:11 /usr/bin/python3 /usr/bin/xpra shadow :0 --speaker=off --desktop-scaling=0.75 --bandwidth-limit=6Mbps --exit-with-client=yes --min-speed=70 --opengl=no --ssh=plink -ssh -agent --env=XPRA_SHADOW_REFRESH_DELAY=200 --env=XPRA_PROXY_START_UUID=21a59d1f24eb42a382263c574ea7dc60 --daemon=yes --systemd-run=no
$ PID=22414
$ cat "/proc/${PID}/environ" | tr '\0' '\n' | grep -i XPRA_SHADOW_REFRESH_DELAY
$
totaam commented 4 years ago

2020-04-29 09:02:56: stdedos uploaded file xpra-2617-all-shadow.png (330.6 KiB)

xpra-2617-all-shadow.png

totaam commented 4 years ago

2020-04-29 09:03:07: stdedos uploaded file xpra-shadow_Xpra_cmd_2020-04-29_10-43-22.png (1866.8 KiB)

xpra-shadow_Xpra_cmd_2020-04-29_10-43-22.png

totaam commented 4 years ago

Which is working really superb-ly (..) This cannot be "very low quality":

You are setting the minimum quality, auto-refresh will still sharpen the image when it is given time to do so. (ie: when the picture is not changing much - like the login screen)

and definitely not 5fps (It would be too much to shoot a video, I guess you can take my word for it). From gaming experience, that would be a table-flip, not a seamless experience

I thought it was working superbly? What fps are you getting? Shadow is not targeting a gaming-like experience for sure, especially not with 6Mbps and large screens.

However, I cannot find it by "conventional" means:

It is in both shadow processes (shadow and the remote _proxy_shadow_start).

totaam commented 4 years ago

2020-04-29 09:27:58: stdedos commented

Replying to [comment:20 Antoine Martin]:

Which is working really superb-ly (..) This cannot be "very low quality": You are setting the minimum quality, auto-refresh will still sharpen the image when it is given time to do so. (ie: when the picture is not changing much - like the login screen)

The only qualities I saw during testing was the fine-crisp login screen / terminal images above, and the spinner. (the other app was Evolution, and the WQHD monitor was showing the empty desktop)

and definitely not 5fps (It would be too much to shoot a video, I guess you can take my word for it). From gaming experience, that would be a table-flip, not a seamless experience I thought it was working superbly? What fps are you getting? Shadow is not targeting a gaming-like experience for sure, especially not with 6Mbps and large screens.

Oh yeah - that one was misleading, apologies. It is working superbly. While there was no fps counter, I feel fairly confident to say it felt like a solid 20fps.

However, I cannot find it by "conventional" means: It is in both shadow processes (shadow and the remote _proxy_shadow_start).

totaam commented 4 years ago

2020-04-30 08:40:58: stdedos commented

I also don't understand - why if the bandwidth limitation is a problem (i.e. I am maxing out the line and "server is not responding" because of no bandwidth left), why isn't the bandwidth maxed out as a flat line to a point, and instead it's a 0-flat line?

[[Image(spped-1.png)]]

... Actually, that is not the I image I clicked to save. I'll take only screenshots instead.

totaam commented 4 years ago

2020-04-30 08:41:11: stdedos uploaded file spped-1.png (17.7 KiB)

spped-1.png

totaam commented 4 years ago

Server just dies over and over, it's so frustrating to keep it up

Could it be #2730?

totaam commented 4 years ago

2020-05-04 19:15:31: stdedos commented

Replying to [comment:23 Antoine Martin]:

Server just dies over and over, it's so frustrating to keep it up Could it be #2730?

No, it's the by-default exit-with-client 😕

totaam commented 4 years ago

and definitely not 5fps

Are you sure that you applied the environment variables separately and correctly?

With XPRA_SHADOW_REFRESH_DELAY=200, I am unable to get more than 5fps and I really don't see how it would be possible to do so.

totaam commented 4 years ago

2020-05-05 11:54:57: stdedos commented

What is there more "true" than your own diagnostics? 😕

$ xpra info :0 | grep XPRA_SHADOW_REFRESH_DELAY
env.XPRA_SHADOW_REFRESH_DELAY=200

I can try to check it again. And I'll add --exit-with-client=no for good measure

totaam commented 4 years ago

I can try to check it again.

Please include the full xpra info.

totaam commented 4 years ago

2020-05-15 09:57:50: stdedos commented

Apologies for the tarball, but I did not have the patience to upload 8 different items with this uploader 😕

So, a quick recap, in archive's order:

  • NOT 5fps video (unless I cannot see it, idk)
  • Too high client CPU for 5fps (or in general looks too high, but I think we've discussed you cannot fix this against Xenial Server)
  • Graph says recv 32000 Kb/s. (a) I doubt it, (b) I have 6mbps limit set
  • Graph shows how "server is not responding, drawing spinners over the windows" looks like to me - 0 received data. How does that even work?
  • Latency seems off. On "Server not responding", can it just render blank? I would say 9999, but then that will overshoot y-axis until stabilization
  • Client and redacted xpra info (collected at Xpra-2617-weird-graph_cmd_2020-05-15_11-16-51.png's time)
totaam commented 4 years ago

2020-05-15 09:58:07: stdedos uploaded file xpra-2617-diag-v2.zip (196.4 KiB)

totaam commented 4 years ago

2020-06-04 10:32:49: stdedos commented

Another instance of a seamless server excessively lagging. Sadly, I lost the client (and the log 😕)

[[Image(xpra-lag-200.png)]]

Example would look like this:

"Xpra-Python3-x86_64_4.0-26306\xpra_cmd" attach ssh://user@ip/200 --ssh="plink -ssh -agent" --modal-windows=no --title="@title@ on @@/@server-display@" --opengl=no --bandwidth-limit=6Mbps

2020-06-04 12:41:12,078 Xpra GTK3 client version 4.0-26306 64-bit
2020-06-04 12:41:12,080  running on Microsoft Windows 10
2020-06-04 12:41:12,170 Warning: failed to import opencv:
2020-06-04 12:41:12,171  No module named 'cv2'
2020-06-04 12:41:12,171  webcam forwarding is disabled
2020-06-04 12:41:13,037 GStreamer version 1.16.2 for Python 3.8.2 64-bit
2020-06-04 12:41:13,582 keyboard layout code 0x409
2020-06-04 12:41:13,583 identified as 'United States - English' : us
2020-06-04 12:41:13,939  keyboard settings: layout=us
2020-06-04 12:41:13,944  desktop size is 4160x1440 with 1 screen:
2020-06-04 12:41:13,945   Default (1100x381 mm - DPI: 96x96) workarea: 4160x1400
2020-06-04 12:41:13,946     Generic PnP Monitor 1600x900 at 0x534 (309x174 mm - DPI: 131x131) workarea: 1600x860
2020-06-04 12:41:13,946     C32JG5x 2560x1440 at 1600x0 (697x392 mm - DPI: 93x93) workarea: 2560x1400
2020-06-04 12:41:35,150 enabled remote logging
2020-06-04 12:41:35,154 Xpra GTK3 X11 server version 3.0.9-26132 64-bit
2020-06-04 12:41:35,156  running on Linux Ubuntu 16.04 xenial
2020-06-04 12:41:35,187 Attached to ip:22
2020-06-04 12:41:35,188  (press Control-C to detach)

(xpra_cmd:18576): Pango-WARNING **: 12:41:35.866: couldn't load font "Bitstream Vera Sans Not-Rotated 14.662109375", falling back to "Sans Not-Rotated 14.662109375", expect ugly output.
2020-06-04 12:41:56,477 server is not responding, drawing spinners over the windows
2020-06-04 12:42:00,244 server is OK again
2020-06-04 12:42:05,087 toggling debug on backing gtk3.CairoBacking(<cairo.ImageSurface object at 0x0000000003fe3c50>) for window 286
2020-06-04 12:42:12,778 server is not responding, drawing spinners over the windows
2020-06-04 12:42:17,293 server is OK again
2020-06-04 12:42:53,314 server is not responding, drawing spinners over the windows
2020-06-04 12:43:03,372 server is OK again
2020-06-04 12:44:17,635 unknown string message: 0xc2a9 / 0x0 / 0x0
2020-06-04 12:44:43,498 server is not responding, drawing spinners over the windows
2020-06-04 12:44:51,293 server is OK again
2020-06-04 12:44:59,257 server is not responding, drawing spinners over the windows
2020-06-04 12:45:08,649 unknown string message: 0xc2a9 / 0x0 / 0x0
2020-06-04 12:45:11,012 server is OK again
2020-06-04 12:45:59,852 server is not responding, drawing spinners over the windows
2020-06-04 12:46:14,220 server is OK again
2020-06-04 12:46:47,026 unknown string message: 0xc2a9 / 0x0 / 0x0
2020-06-04 12:49:09,459 server is not responding, drawing spinners over the windows
2020-06-04 12:49:14,252 server is OK again
2020-06-04 12:49:19,664 server is not responding, drawing spinners over the windows
2020-06-04 12:49:51,794 server is OK again
2020-06-04 12:50:16,892 unknown string message: 0xc2a9 / 0x0 / 0x0

... followed by "server connection lost", with version varying in the r26160 - r26503 range

totaam commented 4 years ago

2020-06-04 10:33:03: stdedos uploaded file redact-display-_200-$TIMESTAMP.log.bz2 (378.6 KiB)

totaam commented 4 years ago

2020-06-04 10:33:13: stdedos uploaded file redact-xpra-lag-200.log (169.3 KiB)

totaam commented 4 years ago

2020-06-04 10:33:21: stdedos uploaded file xpra-lag-200.png (38.3 KiB)

xpra-lag-200.png

totaam commented 4 years ago

2020-06-04 10:51:06: antoine commented

Another instance of a seamless server excessively lagging. Sadly, I lost the client (and the log 😕) This ticket is about the shadow server, is this now happening with the seamless server too? Can you grab the xpra info when the client is connected and having problems? 2.5 seconds between each screen update is worse than slideshow!

totaam commented 4 years ago

2020-06-04 11:00:30: stdedos commented

Replying to [comment:31 Antoine Martin]:

Another instance of a seamless server excessively lagging. Sadly, I lost the client (and the log 😕) This ticket is about the shadow server, is this now happening with the seamless server too?

I started mixing them, given that I see a lot less of shadow server action (I gave up trying to make that work as my default), and that issue appears in seamless modes too.

What is more annoying, is that e.g., for every 5 invocations, 2 are broken, 1 is meh, and 1 or 2 are maybe working (instead e.g. for something clearly, constantly broken). In seamless modes, those limitations are changed to 1 broken link every 15 invocations.

Can you grab the xpra info when the client is connected and having problems?

Apologies, redact-xpra-lag-200.log​ is an xpra-info. Added bonus: the timeout messages given when I was collecting xpra info concurrently with the client trying not to choke.

2.5 seconds between each screen update is worse than slideshow!

If you'd plot server timeout values like I asked somewhere (using a magic 9999 number), given wide enough graphs, that graph would look like a cardiograph instead. "Slideshow" is a gross understatement.