secondlife / viewer

🖥️ Second Life's official client
GNU Lesser General Public License v2.1
200 stars 48 forks source link

WebRTC: Stall when crossing parcel boundaries on Windows #1022

Open roxanneskelly opened 3 months ago

roxanneskelly commented 3 months ago

Environment

Second Life Test 7.1.4.8351851435 (64bit) Release Notes

You are at 138.9, 114.0, 33.8 in webRTC1 located at simhost-07f5f7ddf45b02732.aditi SLURL: secondlife://Aditi/secondlife/webRTC1/139/114/34 (global coordinates 259,979.0, 245,362.0, 33.8) WebRTC-Voice 2024-03-15.8291114706 Release Notes

CPU: AMD Ryzen 9 5900HX with Radeon Graphics (3293.81 MHz) Memory: 15792 MB OS Version: Microsoft Windows 10 64-bit (Build 19044.3086) Graphics Card Vendor: NVIDIA Corporation Graphics Card: NVIDIA GeForce RTX 3080 Laptop GPU/PCIe/SSE2

Windows Graphics Driver Version: 30.0.15.1278 OpenGL Version: 4.6.0 NVIDIA 512.78

Window size: 1655x848 Font Size Adjustment: 96pt UI Scaling: 1 Draw distance: 128m Bandwidth: 3000kbit/s LOD factor: 1.75 Render quality: 5 Advanced Lighting Model: Enabled Texture memory: 7273MB Disk cache: Max size 1638.4 MB (99.9% used)

J2C Decoder Version: KDU v7.10.4 Audio Driver Version: FMOD Studio 2.02.13 Dullahan: 1.14.0.202310131404 CEF: 118.4.1+g3dd6078+chromium-118.0.5993.54 Chromium: 118.0.5993.54 LibVLC Version: 3.0.16 Voice Server Version: Secondlife WebRTC Gateway

Packets Lost: 0/298 (0.0%) March 20 2024 10:13:27

Description

Crossing between different voice channels (parcel, region, etc.) stalls the viewer for a short time, resulting in jittering framerate.

This happens during the SetLocalDescription webrtc call, which begins doing things like ICE negotiation. SetLocalDescription may make callbacks into viewer code which in turn are proxied to the main thread, so the obvious thought is that that code is taking too long. However, the code is very small and should complete quickly, and has completed quickly in past measurements.

For the most part, those calls are 'OnIceCandidate' and 'OnIceGatheringState' calls, followed by 'OnAnswerAvailable.' Commenting out those callbacks doesn't fix the problem.

All calls into webrtc are proxied onto the webrtc threads (signaling or worker,) and that is simply putting a call onto the thread queue.

Interesting thing is, this only happens on Windows, not Mac.

I'm wondering if there's some sort of resource contention going on, such as network, which is causing issues.

Reproduction steps

1) set up a parcel with parcel-limited voice. 2) Fly into that parcel from another parcel with parcel or region voice. 3) Observe the stutter in framerate.

or

1) Fly within about 100m of another region. 2) When the viewer connects to that region at that distance, there will be a stutter.

roxanneskelly commented 3 months ago

Interesting snippets from a Tracy profiling session. SwapBuffers appears to be taking a very long time when the stuttering occurs, as opposed to calls into the WebRTC voice subsystem. However, this primarily happens when new WebRTC connections are established. So, there's an interaction somewhere?

Screenshot 2024-03-21 214542 Screenshot 2024-03-21 214609

roxanneskelly commented 3 months ago

Does not happen on the following machine:

Second Life Test 7.1.4.8290752947 (64bit) Release Notes

You are at 196.7, 155.5, 22.9 in webRTC1 located at simhost-0142872bd2be1d5c3.aditi SLURL: secondlife://Aditi/secondlife/webRTC1/197/156/23 (global coordinates 260,037.0, 245,404.0, 22.9) WebRTC-Voice 2024-03-15.8291114706 Release Notes

CPU: 12th Gen Intel(R) Core(TM) i7-12700KF (3609.6 MHz) Memory: 32612 MB OS Version: Microsoft Windows 10/11 64-bit (Build 22631.3155) Graphics Card Vendor: NVIDIA Corporation Graphics Card: NVIDIA GeForce RTX 3060/PCIe/SSE2

Windows Graphics Driver Version: 31.0.15.4601 OpenGL Version: 4.6.0 NVIDIA 546.01

Window size: 2157x1637 Font Size Adjustment: 96pt UI Scaling: 1 Draw distance: 128m Bandwidth: 3000kbit/s LOD factor: 1.75 Render quality: 5 Advanced Lighting Model: Enabled Texture memory: 11361MB Disk cache: Max size 1638.4 MB (12.7% used)

J2C Decoder Version: KDU v7.10.4 Audio Driver Version: FMOD Studio 2.02.13 Dullahan: 1.14.0.202310131404 CEF: 118.4.1+g3dd6078+chromium-118.0.5993.54 Chromium: 118.0.5993.54 LibVLC Version: 3.0.16 Voice Server Version: Secondlife WebRTC Gateway

Packets Lost: 0/4,814 (0.0%) March 22 2024 15:33:35

bridie-linden commented 3 weeks ago

@maestrolinden or @kylelinden - do you see this on your Macs?

Dan-Linden commented 3 weeks ago

Does not happen on the following machine:

Second Life Release 7.1.9.9477703110 (64bit) Release Notes

You are at 147.9, 173.9, 22.9 in webRTC1 located at simhost-07c23d76c83837d55.aditi SLURL: secondlife://Aditi/secondlife/webRTC1/148/174/23 (global coordinates 259,988.0, 245,422.0, 22.9) WebRTC-Voice 2024-06-12.9490472357 Release Notes

CPU: Intel(R) Core(TM) i7-5930K CPU @ 3.50GHz (3491.92 MHz) Memory: 32610 MB OS Version: Microsoft Windows 10 64-bit (Build 19045.4412) Graphics Card Vendor: NVIDIA Corporation Graphics Card: NVIDIA GeForce GTX 1080/PCIe/SSE2

Windows Graphics Driver Version: 31.0.15.3623 OpenGL Version: 4.6.0 NVIDIA 536.23

Window size: 1729x1344 Font Size Adjustment: 96pt UI Scaling: 1 Draw distance: 128m Bandwidth: 3000kbit/s LOD factor: 1.75 Render quality: 5 Texture memory: 7304MB Disk cache: Max size 1638.4 MB (24.3% used)

J2C Decoder Version: KDU v7.10.4 Audio Driver Version: FMOD Studio 2.02.13 Dullahan: 1.14.0.202310131404 CEF: 118.4.1+g3dd6078+chromium-118.0.5993.54 Chromium: 118.0.5993.54 LibVLC Version: 3.0.16 Voice Server Version: Secondlife WebRTC Gateway

Packets Lost: 1/10,915 (0.0%) June 14 2024 15:49:42

maestrolinden commented 3 weeks ago

I cannot reproduce this on my Mac with a recent webRTC build:

Second Life Release 7.1.9.9477703110 (64bit)
Release Notes

You are at 158.5, 173.3, 23.0 in webRTC1 located at simhost-07c23d76c83837d55.aditi
SLURL: secondlife://Aditi/secondlife/webRTC1/159/173/23
(global coordinates 259998.0, 245421.0, 23.0)
WebRTC-Voice 2024-06-12.9490472357
Release Notes

CPU: Apple M1 Pro (2400 MHz)
Memory: 16384 MB
OS Version: Mac OS X 14.5.0 Darwin 23.5.0 Darwin Kernel Version 23.5.0: Wed May  1 20:12:58 PDT 2024; root:xnu-10063.121.3~5/RELEASE_ARM64_T6000 x86_64
Graphics Card Vendor: Apple
Graphics Card: Apple M1 Pro

OpenGL Version: 4.1 Metal - 88.1

Window size: 1086x710
Font Size Adjustment: 96pt
UI Scaling: 1
Draw distance: 128m
Bandwidth: 3000kbit/s
LOD factor: 1.25
Render quality: 2
Texture memory: 10922MB
Disk cache: Max size 1638.4 MB (0.6% used)
HiDPI display mode: 0

J2C Decoder Version: KDU v7.10.4
Audio Driver Version: FMOD Studio 2.02.13
Dullahan: 1.14.0.202310131309
  CEF: 118.4.1+g3dd6078+chromium-118.0.5993.54
  Chromium: 118.0.5993.54
LibVLC Version: 3.0.16
Voice Server Version: Secondlife WebRTC Gateway 

Packets Lost: 2/11946 (0.0%)
June 14 2024 15:50:27

@roxanneskelly I'm happy to close this issue if you're also having difficulty reproducing it with recent builds.

roxanneskelly commented 2 weeks ago

We need to gather data about how prevalent this issues is for end-users, as we no longer have a repro.