search

secondlife / viewer

🖥️ Second Life's official client
GNU Lesser General Public License v2.1
199 stars 46 forks source link

Crash after toggling Voice Echo Cancellation off and on #1732

Open Dan-Linden opened 2 weeks ago

Dan-Linden commented 2 weeks ago

Environment

Second Life Project WebRTC Voice 7.1.4.8947030231 (64bit) Release Notes

You are at 66.7, 39.4, 38.5 in Danger! Danger! located at simhost-04c2df15f77bb17ff.agni SLURL: http://maps.secondlife.com/secondlife/Danger%21%20Danger%21/67/39/39 (global coordinates 261,187.0, 249,639.0, 38.5) Second Life Server 2024-04-13.8669470296 Release Notes

CPU: AMD Ryzen 9 5980HX with Radeon Graphics (3293.82 MHz) Memory: 15775 MB OS Version: Microsoft Windows 10/11 64-bit (Build 22000.2538) Graphics Card Vendor: ATI Technologies Inc. Graphics Card: AMD Radeon RX 6800M

Windows Graphics Driver Version: 30.0.15020.1001 OpenGL Version: 4.6.14800 Core Profile Context 21.50.20.01 30.0.15020.1001

Window size: 1008x766 Font Size Adjustment: 96pt UI Scaling: 1 Draw distance: 128m Bandwidth: 3000kbit/s LOD factor: 1.75 Render quality: 5 Advanced Lighting Model: Enabled Texture memory: 11474MB Disk cache: Max size 1638.4 MB (9.4% used)

J2C Decoder Version: KDU v7.10.4 Audio Driver Version: FMOD Studio 2.02.13 Dullahan: 1.14.0.202310131404 CEF: 118.4.1+g3dd6078+chromium-118.0.5993.54 Chromium: 118.0.5993.54 LibVLC Version: 3.0.16 Voice Server Version: Vivox 4.10.0000.32327.5fc3fe7c.399bd0e

Packets Lost: 0/2,503 (0.0%) June 12 2024 06:16:23

Description

There's no BugSplat report from this crash! Forced crashes (Develop > Force an Error > Force LLError And Crash) from this viewer appear in BugSplat, so there's something unusual about this crash.

Crash after toggling Voice Echo Cancellation off and on.

Reproduction steps

  1. On Windows, replace your C:\Users\USERNAME\AppData\Roaming\SecondLife\user_settings\settings.xml with the file from settings.xml_bad.txt
  2. Log in with a WebRTC viewer (only tested on Windows 11)
  3. Open Preferences > Sound & Media
  4. Toggle the checkbox for Echo Cancellation off then on. See the video for the timing.

Observed: Crash to desktop. Expected: No crash.

https://github.com/secondlife/viewer/assets/119471733/2ba670a2-aaef-467f-9072-edd9ba70b013

7.1.4.8947030231 crash SecondLife.log

maestrolinden commented 2 weeks ago

Testing with Second Life Project Webrtc-voice 7.1.8.9165328470 (64bit) on Win10, which is a newer build than the original report, I can't reproduce this crash when on either Second Life Server 2024-04-13.8669470296 (SLS/vivox-mode simulator) or when on WebRTC-Voice 2024-06-12.9475602142.

Dan-Linden commented 2 weeks ago

This crash reproduced for me on WeRTC 7.1.9.9477703110 Windows 10 / on a Preflight 2024-06-11.9458617693 server.

maestrolinden commented 2 weeks ago

It does not reproduce for me with WebRTC 7.1.9.9477703110 Windows 10 on these server versions:

I'm not sure why I see different results than Dan - perhaps a difference in microphone hardware? My microphone is from a 'Logitech 615 HD WebCam' - connected over USB. In Windows 'Microphone Properties', this microphone is configured to capture at 1 channel, 16 bit, 48kHz.

roxanneskelly commented 2 weeks ago

took a few tries, but...

KernelBase.dll!00007ff9f9208072()
llwebrtc.dll!rtc::webrtc_checks_impl::WriteFatalLog(std::basic_string_view<char,std::char_traits<char>> output) Line 78
    at webrtc\webrtc\src\rtc_base\checks.cc(78)
llwebrtc.dll!rtc::webrtc_checks_impl::WriteFatalLog(const char * file, int line, std::basic_string_view<char,std::char_traits<char>> output) Line 84
    at webrtc\webrtc\src\rtc_base\checks.cc(84)
llwebrtc.dll!rtc::webrtc_checks_impl::FatalLog(const char * file, int line, const char * message, const rtc::webrtc_checks_impl::CheckArgType * fmt, ...) Line 179
    at webrtc\webrtc\src\rtc_base\checks.cc(179)
[Inline Frame] llwebrtc.dll!rtc::webrtc_checks_impl::LogStreamer<>::CallCheckOp(const char * file, const int line, const char * message, const rtc::webrtc_checks_impl::Val<6,unsigned long long> & args, const rtc::webrtc_checks_impl::Val<6,unsigned long long> & args) Line 323
    at webrtc\webrtc\src\rtc_base\checks.h(323)
[Inline Frame] llwebrtc.dll!rtc::webrtc_checks_impl::LogStreamer<rtc::webrtc_checks_impl::Val<6,unsigned long long>>::CallCheckOp(const char * file, const int line, const char * message, const rtc::webrtc_checks_impl::Val<6,unsigned long long> & args) Line 265
    at webrtc\webrtc\src\rtc_base\checks.h(265)
[Inline Frame] llwebrtc.dll!rtc::webrtc_checks_impl::LogStreamer<rtc::webrtc_checks_impl::Val<6,unsigned long long>,rtc::webrtc_checks_impl::Val<6,unsigned long long>>::CallCheckOp(const char * file, const int line, const char * message) Line 265
    at webrtc\webrtc\src\rtc_base\checks.h(265)
[Inline Frame] llwebrtc.dll!rtc::webrtc_checks_impl::FatalLogCall<1>::operator&(const rtc::webrtc_checks_impl::LogStreamer<rtc::webrtc_checks_impl::Val<6,unsigned long long>,rtc::webrtc_checks_impl::Val<6,unsigned long long>> & streamer) Line 345
    at webrtc\webrtc\src\rtc_base\checks.h(345)
llwebrtc.dll!webrtc::EchoCanceller3::RenderWriter::Insert(const webrtc::AudioBuffer & input) Line 700
    at webrtc\webrtc\src\modules\audio_processing\aec3\echo_canceller3.cc(700)
llwebrtc.dll!webrtc::EchoCanceller3::AnalyzeRender(const webrtc::AudioBuffer & render) Line 833
    at webrtc\webrtc\src\modules\audio_processing\aec3\echo_canceller3.cc(833)
llwebrtc.dll!webrtc::EchoCanceller3::AnalyzeRender(webrtc::AudioBuffer * render) Line 105
    at webrtc\webrtc\src\modules\audio_processing\aec3\echo_canceller3.h(105)
llwebrtc.dll!webrtc::AudioProcessingImpl::ProcessRenderStreamLocked() Line 1949
    at webrtc\webrtc\src\modules\audio_processing\audio_processing_impl.cc(1949)
llwebrtc.dll!webrtc::AudioProcessingImpl::ProcessReverseStream(const short * const src, const webrtc::StreamConfig & input_config, const webrtc::StreamConfig & output_config, short * const dest) Line 1914
    at webrtc\webrtc\src\modules\audio_processing\audio_processing_impl.cc(1914)
llwebrtc.dll!webrtc::ProcessReverseAudioFrame(webrtc::AudioProcessing * ap, webrtc::AudioFrame * frame) Line 61
    at webrtc\webrtc\src\modules\audio_processing\include\audio_frame_proxies.cc(61)
llwebrtc.dll!webrtc::AudioTransportImpl::NeedMorePlayData(unsigned __int64 nSamples, unsigned __int64 nBytesPerSample, unsigned __int64 nChannels, unsigned int samplesPerSec, void * audioSamples, unsigned __int64 & nSamplesOut, __int64 * elapsed_time_ms, __int64 * ntp_time_ms) Line 232
    at webrtc\webrtc\src\audio\audio_transport_impl.cc(232)
llwebrtc.dll!webrtc::`anonymous namespace'::ADMWrapper::NeedMorePlayData(const unsigned __int64 nSamples, const unsigned __int64 nBytesPerSample, const unsigned __int64 nChannels, const unsigned int samples_per_sec, void * audioSamples, unsigned __int64 & nSamplesOut, __int64 * elapsed_time_ms, __int64 * ntp_time_ms) Line 111
    at webrtc\webrtc\src\modules\audio_device\audio_device_data_observer.cc(111)
llwebrtc.dll!webrtc::AudioDeviceBuffer::RequestPlayoutData(unsigned __int64 samples_per_channel) Line 338
    at webrtc\webrtc\src\modules\audio_device\audio_device_buffer.cc(338)
llwebrtc.dll!webrtc::AudioDeviceWindowsCore::DoRenderThread() Line 2836
    at webrtc\webrtc\src\modules\audio_device\win\audio_device_core_win.cc(2836)
llwebrtc.dll!webrtc::AudioDeviceWindowsCore::WSAPIRenderThread(void * context) Line 2642
    at webrtc\webrtc\src\modules\audio_device\win\audio_device_core_win.cc(2642)
[External Code]
roxanneskelly commented 2 weeks ago

Repros for me on Vivox regions, but not on WebRTC regions. I do have to repeatedly click the echo checkbox, though.

Dan-Linden commented 2 weeks ago

Reproduced for me on Second Life Release 7.1.9.9477703110 (64bit) // WebRTC-Voice 2024-06-12.9475602142 SLURL: secondlife://Aditi/secondlife/webRTC1/128/128/23 Windows 11

This may be a more reliable repro:

  1. Toggle Preferences > Sound & Media > Echo Cancellation off.
  2. Restart the viewer and log in.
  3. Toggle Preferences > Sound & Media > Echo Cancellation on.

7.1.9.9477703110 crash SecondLife.log

Dan-Linden commented 2 weeks ago

Something in old settings is causing this crash as well as other problems. After deleting my settings.xml, and letting WebRTC generate a new settings.xml I can no longer easily trigger this crash. Also, the Noise Suppression option is no longer grayed out/unavailable.

settings.xml_bad.txt settings.xml_good.txt

maestrolinden commented 2 weeks ago

Something in old settings is causing this crash as well as other problems. After deleting my settings.xml, and letting WebRTC generate a new settings.xml I can no longer easily trigger this crash. Also, the Noise Suppression option is no longer grayed out/unavailable.

settings.xml_bad.txt settings.xml_good.txt

I confirmed that the crash reproduces for me on WebRTC 7.1.9.9477703110 on the same machine that was failing to repro earlier, when I place settings.xml_bad.txt into %APPDATA\SecondLife\user_settings\settings.xml

As with @Dan-Linden 's original findings, I can reproduce the viewer crash when on WebRTC-Voice 2024-06-12.9475602142 (webRTC voice mode), but not when on Second Life Server 2024-04-13.8669470296 (SLS/vivox-mode simulator).

Comparing the 'good' and 'bad' settings.xml files from his comment, the only significant differences are the presence of certain keys:

None of these seems related to the echo cancellation setting. The AudioStreaming settings are both booleans relating to whether streaming media or audio is enabled, and in the "bad" file both of these settings are set to 0 (disabled).

maestrolinden commented 1 week ago

I cannot reproduce this crash when using viewer build 7.1.9.9573202536 roxie/webrtc-voice branch, using the settings.xml_bad.txt attached to this issue.

This viewer also appears to fix #1776 as well (as I commented in that issue).

Mac env

Second Life Test 7.1.9.9573202536 (64bit)
Release Notes

You are at 148.7, 150.5, 23.0 in webRTC1 located at simhost-07c23d76c83837d55.aditi
SLURL: secondlife://Aditi/secondlife/webRTC1/149/150/23
(global coordinates 259989.0, 245398.0, 23.0)
WebRTC-Voice 2024-06-12.9490472357
Release Notes

CPU: Apple M1 Pro (2400 MHz)
Memory: 16384 MB
OS Version: Mac OS X 14.5.0 Darwin 23.5.0 Darwin Kernel Version 23.5.0: Wed May  1 20:12:58 PDT 2024; root:xnu-10063.121.3~5/RELEASE_ARM64_T6000 x86_64
Graphics Card Vendor: Apple
Graphics Card: Apple M1 Pro

OpenGL Version: 4.1 Metal - 88.1

Window size: 2522x1640
Font Size Adjustment: 96pt
UI Scaling: 0.75
Draw distance: 96m
Bandwidth: 10000kbit/s
LOD factor: 1.125
Render quality: 1
Texture memory: 10922MB
Disk cache: Max size 3993.6 MB (3.0% used)
HiDPI display mode: 1

J2C Decoder Version: KDU v7.10.4
Audio Driver Version: FMOD Studio 2.02.13
Dullahan: 1.14.0.202310131309
  CEF: 118.4.1+g3dd6078+chromium-118.0.5993.54
  Chromium: 118.0.5993.54
LibVLC Version: 3.0.16
Voice Server Version: Secondlife WebRTC Gateway 

Packets Lost: 0/3466 (0.0%)
June 18 2024 16:59:40

Windows env

Second Life Test 7.1.9.9573202536 (64bit)
Release Notes

You are at 149.2, 148.9, 23.0 in webRTC1 located at simhost-07c23d76c83837d55.aditi
SLURL: secondlife://Aditi/secondlife/webRTC1/149/149/23
(global coordinates 259,989.0, 245,397.0, 23.0)
WebRTC-Voice 2024-06-12.9490472357
Release Notes

CPU: Intel(R) Core(TM) i7-4930K CPU @ 3.40GHz (3402.14 MHz)
Memory: 32704 MB
OS Version: Microsoft Windows 10 64-bit (Build 19045.4412)
Graphics Card Vendor: NVIDIA Corporation
Graphics Card: NVIDIA GeForce RTX 2080/PCIe/SSE2

Windows Graphics Driver Version: 31.0.15.3623
OpenGL Version: 4.6.0 NVIDIA 536.23

Window size: 1183x743
Font Size Adjustment: 96pt
UI Scaling: 0.75
Draw distance: 128m
Bandwidth: 10000kbit/s
LOD factor: 1.75
Render quality: 5
Texture memory: 7234MB
Disk cache: Max size 3993.6 MB (23.9% used)

J2C Decoder Version: KDU v7.10.4
Audio Driver Version: FMOD Studio 2.02.13
Dullahan: 1.14.0.202310131404
  CEF: 118.4.1+g3dd6078+chromium-118.0.5993.54
  Chromium: 118.0.5993.54
LibVLC Version: 3.0.16
Voice Server Version: Secondlife WebRTC Gateway 

Packets Lost: 0/488 (0.0%)
June 18 2024 17:01:44
maestrolinden commented 1 week ago

I retested this in release/webrtc-voice build 7.1.9.9574256458, which includes the merge with the fix. The crash is also fixed with this build. Passes QA.

DmitrySProductengine commented 1 week ago

Passed QA. Verified on the Second Life Release 7.1.9.9604591078 (64bit) on Win10/Win11/OSX in the scope of IQA #264. Tested on Agni, Product Engine 3 Second Life Server 2024-06-11.9458617693 Vivox 4.10.0000.32327.5fc3fe7c.5942f08