Closed kmille closed 10 months ago
scottnonnenberg-signal
commented
11 months ago An exit code 132 - we haven't seen that one before. This comment seems to suggest that we might be dealing with a SIGILL, so an illegal instruction? https://github.com/chromedp/chromedp/issues/1094#issuecomment-1159745522
Either way, please reach out to the folks who maintain the arch package. That's not an officially-supported platform, and they've made a number of customizations appropriate for Arch.
kmille
commented
11 months ago The problem was caused by linux-hardened. Booting from the regular linux kernel works.
kmille
commented
11 months ago Hello again,
After some research... I have some more information I want to share with you. I created an issue in the Arch Linux bug tracker [0]. They forwarded me to [1]. As far as I understood: Linux 6.4.9 [2] includes commit 553a5c03e90a6087e88f8ff878335ef0621536fb [3] which is: x86/speculation: Add force option to GDS mitigation [4]
And they state in the commit message:
This is a big hammer. It is known to break buggy userspace that uses incomplete, buggy AVX enumeration. Unfortunately, such userspace does exist in the wild
Before signal crashes, it reads /proc/cpuinfo. If I boot linux-hardened, I don't have AVX/AVX2 flags in /proc/cpuinfo. But I don't know why it only crashes on linux-hardened, as my normal linux kernel is also > 6.4.9. Is there a userspace avx enumeration bug?
[0] https://bugs.archlinux.org/task/79444 [1] https://bugs.archlinux.org/task/79828 [2] https://cdn.kernel.org/pub/linux/kernel/v6.x/ChangeLog-6.4.9 [3] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=553a5c03e90a6087e88f8ff878335ef0621536fb [4] https://docs.kernel.org/admin-guide/hw-vuln/gather_data_sampling.html
wansing
commented
11 months ago I think I figured out what's going on here. The Linux kernel disables AVX before the microcode update is loaded:
journalctl -k --grep=microcode -b -1
kernel: SRBDS: Mitigation: Microcode
kernel: GDS: Microcode update needed! Disabling AVX as mitigation.
kernel: GDS: Mitigation: AVX disabled, no microcode
kernel: microcode: Microcode Update Driver: v2.2.Because AVX has been disabled, /proc/cpuinfo does not display it. But apparently Signal-Desktop uses a different method to detect AVX support, assumes that AVX is supported, calls an AVX instruction, and crashes.
Setting the Linux kernel parameter gather_data_sampling=off fixes the issue. It keeps AVX enabled, so you must rely on the microcode update to mitigate the Gather Data Sampling hardware vulnerability. Please make sure that your microcode is enabled and mitigates the vulnerability for your CPU model.
Updating your BIOS might also fix this issue, as the BIOS microcode updates are probably loaded before the Linux kernel.
nikkolade
commented
10 months ago I think I figured out what's going on here. The Linux kernel disables AVX before the microcode update is loaded:
journalctl -k --grep=microcode -b -1 kernel: SRBDS: Mitigation: Microcode kernel: GDS: Microcode update needed! Disabling AVX as mitigation. kernel: GDS: Mitigation: AVX disabled, no microcode kernel: microcode: Microcode Update Driver: v2.2.Because AVX has been disabled,
/proc/cpuinfodoes not display it. But apparently Signal-Desktop uses a different method to detect AVX support, assumes that AVX is supported, calls an AVX instruction, and crashes.Setting the Linux kernel parameter
gather_data_sampling=offfixes the issue. It keeps AVX enabled, so you must rely on the microcode update to mitigate the Gather Data Sampling hardware vulnerability. Please make sure that your microcode is enabled and mitigates the vulnerability for your CPU model.Updating your BIOS might also fix this issue, as the BIOS microcode updates are probably loaded before the Linux kernel.
I've been hunting for the kernel commit that seems to cause the signal-desktop to crash on my old i7-6700K host and found out that this change broke signal-desktop on Debian 12:
commit ca5e8427e20d770b7b2ad0c4eafd17f4f77c7779 Author: Dave Hansen dave.hansen@linux.intel.com Date: Tue Aug 29 08:07:25 2023 -0700
x86/speculation: Mark all Skylake CPUs as vulnerable to GDSThere problem is that this CPU isn't supported anymore by Intel and therefore there is no available microcode updates for this security issue and there also won't be one upcoming anymore.
Please, do check, what can be done for the signal-desktop to be fixed so that it would continue to work also with this CPU with the security mitigation enabled.
kmille
commented
10 months ago Some updates:
TLDR: There is no microkernel update for my CPU. So I can use gather_data_sampling=off and be vulnerable or I can't use signal-desktop. And it seems like you want to have the mitigation turned on.
This page contains more information about how to update the microkernel and some inner details. The intel-ucode package uses the microcode updates provided by intel in this Github Repo.
nikkolade
commented
10 months ago Ok, we seem to be in the same boat together what comes to the available HW support (ie no support whatsoever). I wonder if we have to create a new ticket as this one is closed and if @scottnonnenberg-signal or somebody else is not willing to re-open this ticket anymore.
We do now have solid evidence that this problem is caused by the GDS security mitigation on Linux and it's going to affect all users with current Linux kernels but out of date HW (ie. not supported by Intel anymore).
trevor-signal
commented
10 months ago Thanks for all these investigations. Are you having issues with any other Electron apps (e.g. Discord/Slack) on this hardware, or is it only Signal?
nikkolade
commented
10 months ago Thanks for all these investigations. Are you having issues with any other Electron apps (e.g. Discord/Slack) on this hardware, or is it only Signal?
I haven't tried slack, but discord works just fine. It's only the signal-deskop that is giving me issues with these kernels where the GDS mitigation has been enabled.
trevor-signal
commented
10 months ago Alright, and just to confirm, this is now the case on current linux kernel v6.6 (not just on linux-hardened), right?
nikkolade
commented
10 months ago Thanks for all these investigations. Are you having issues with any other Electron apps (e.g. Discord/Slack) on this hardware, or is it only Signal?
I haven't tried slack, but discord works just fine. It's only the signal-deskop that is giving me issues with these kernels where the GDS mitigation has been enabled.
Ok, I had a chance of rebooting the machine again, with the GDS security mitigation turned on and it seems that I was wrong. The discord application does seem to start, but it does indeed die in a similar way as the signal-desktop application.
discord:
blackbox: 2023-10-30T15:54:46.471Z 20 ✅ webContents.did-finish-load web3
[4192:1030/175446.476080:ERROR:shared_image_factory.cc(575)] Could not find SharedImageBackingFactory with params: usage: Gles2|Raster|DisplayRead|Scanout, format: BGRA_8888, share_between_threads: 0, gmb_type: shared_memory
[4192:1030/175446.477723:ERROR:shared_image_factory.cc(575)] Could not find SharedImageBackingFactory with params: usage: Gles2|Raster|DisplayRead|Scanout, format: BGRA_8888, share_between_threads: 0, gmb_type: shared_memory
[4192:1030/175446.479320:ERROR:shared_image_factory.cc(575)] Could not find SharedImageBackingFactory with params: usage: Gles2|Raster|DisplayRead|Scanout, format: BGRA_8888, share_between_threads: 0, gmb_type: shared_memory
[4192:1030/175446.480740:ERROR:shared_image_factory.cc(575)] Could not find SharedImageBackingFactory with params: usage: Gles2|Raster|DisplayRead|Scanout, format: BGRA_8888, share_between_threads: 0, gmb_type: shared_memory
[4192:1030/175446.482542:ERROR:shared_image_factory.cc(575)] Could not find SharedImageBackingFactory with params: usage: Gles2|Raster|DisplayRead|Scanout, format: BGRA_8888, share_between_threads: 0, gmb_type: shared_memory
[4192:1030/175446.510427:ERROR:shared_image_factory.cc(575)] Could not find SharedImageBackingFactory with params: usage: Gles2|Raster|DisplayRead|Scanout, format: BGRA_8888, share_between_threads: 0, gmb_type: shared_memory
[4192:1030/175446.511909:ERROR:shared_image_factory.cc(575)] Could not find SharedImageBackingFactory with params: usage: Gles2|Raster|DisplayRead|Scanout, format: BGRA_8888, share_between_threads: 0, gmb_type: shared_memory
[WebContents] double crashed (reason: crashed, exitCode: 4)... RIP =(
blackbox: 2023-10-30T15:54:47.456Z 21 ❌ render-process-gone { reason: 'crashed', exitCode: 4 }
blackbox: 2023-10-30T15:54:47.463Z 22 window.close win3signal-desktop:
{"level":30,"time":"2023-10-30T15:55:29.132Z","msg":"spellcheck: setting languages to: [\"en-US\",\"en\"]"}
{"level":40,"time":"2023-10-30T15:55:29.138Z","msg":"MainSQL: Database log code=283: recovered 965 frames from WAL file [REDACTED]/sql/db.sqlite-wal"}
{"level":30,"time":"2023-10-30T15:55:29.138Z","msg":"MainSQL: updateSchema:\n Current user_version: 950;\n Most recent db schema: 960;\n SQLite version: 3.42.0;\n SQLCipher version: 4.5.5 community;\n (deprecated) schema_version: 437;\n"}
{"level":30,"time":"2023-10-30T15:55:29.138Z","msg":"MainSQL: updateToSchemaVersion960: not running, pni is normalized"}
{"level":30,"time":"2023-10-30T15:55:29.138Z","msg":"MainSQL: updateToSchemaVersion960: success!"}
{"level":30,"time":"2023-10-30T15:55:29.139Z","msg":"MainSQL: updateSchema: optimize took 0ms"}
{"level":30,"time":"2023-10-30T15:55:29.477Z","msg":"got fast theme-setting value dark"}
{"level":40,"time":"2023-10-30T15:55:29.822Z","msg":"safeLoadURL: ignoring ERR_FAILED because we are shutting down {\"errno\":-2,\"code\":\"ERR_FAILED\",\"url\":\"file:[REDACTED]/background.html\"}"}
{"level":30,"time":"2023-10-30T15:55:29.822Z","msg":"got fast spellcheck setting true"}
Render process is gone: Error: Reason: crashed, Exit Code: 4
at App.<anonymous> (/opt/Signal/resources/app.asar/app/global_errors.js:88:7)
at App.emit (node:events:525:35)
at WebContents.<anonymous> (node:electron/js2c/browser_init:2:90123)
at WebContents.emit (node:events:513:28)
{"level":50,"time":"2023-10-30T15:55:29.823Z","msg":"Render process is gone: Error: Reason: crashed, Exit Code: 4\n at App.<anonymous> ([REDACTED]/app/global_errors.js:88:7)\n at App.emit (node:events:525:35)\n at WebContents.<anonymous> (node:electron/js2c/browser_init:2:90123)\n at WebContents.emit (node:events:513:28)"}
{"level":30,"time":"2023-10-30T15:55:36.170Z","msg":"System tray service: created"}
{"level":30,"time":"2023-10-30T15:55:36.170Z","msg":"Begin ensuring permissions"}This is just a normal Debian 12 desktop with a kernel.org v6.1.60 amd64 kernel with the following intel CPU:
>cat /proc/cpuinfo
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 94
model name : Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz
stepping : 3
microcode : 0xc2
cpu MHz : 4291.324
cache size : 8192 KB
physical id : 0
siblings : 8
core id : 0
cpu cores : 4
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 22
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb invpcid_single pti ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx rdseed adx smap clflushopt intel_pt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp
vmx flags : vnmi preemption_timer invvpid ept_x_only ept_ad ept_1gb flexpriority tsc_offset vtpr mtf vapic ept vpid unrestricted_guest ple shadow_vmcs pml
bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs taa itlb_multihit srbds mmio_stale_data retbleed gds
bogomips : 7999.96
clflush size : 64
cache_alignment : 64
address sizes : 39 bits physical, 48 bits virtual
power management:It's not a problem to try out the latest mainline, if that is needed.
The application crash can be circumvented either by disabling the GDS Security fix by adding the "gather_data_sampling=off" to the boot parameters, or reverting the below commit:
commit ca5e8427e20d770b7b2ad0c4eafd17f4f77c7779
Author: Dave Hansen <dave.hansen@linux.intel.com>
Date: Tue Aug 29 08:07:25 2023 -0700
x86/speculation: Mark all Skylake CPUs as vulnerable to GDS
[ Upstream commit c9f4c45c8ec3f07f4f083f9750032a1ec3eab6b2 ]
The Gather Data Sampling (GDS) vulnerability is common to all Skylake
processors. However, the "client" Skylakes* are now in this list:
https://www.intel.com/content/www/us/en/support/articles/000022396/processors.html
which means they are no longer included for new vulnerabilities here:
https://www.intel.com/content/www/us/en/developer/topic-technology/software-security-guidance/processors-affected-consolidated-product-cpu-model.html
or in other GDS documentation. Thus, they were not included in the
original GDS mitigation patches.
Mark SKYLAKE and SKYLAKE_L as vulnerable to GDS to match all the
other Skylake CPUs (which include Kaby Lake). Also group the CPUs
so that the ones that share the exact same vulnerabilities are next
to each other.
Last, move SRBDS to the end of each line. This makes it clear at a
glance that SKYLAKE_X is unique. Of the five Skylakes, it is the
only "server" CPU and has a different implementation from the
clients of the "special register" hardware, making it immune to SRBDS.
This makes the diff much harder to read, but the resulting table is
worth it.
I very much appreciate the report from Michael Zhivich about this
issue. Despite what level of support a hardware vendor is providing,
the kernel very much needs an accurate and up-to-date list of
vulnerable CPUs. More reports like this are very welcome.
* Client Skylakes are CPUID 406E3/506E3 which is family 6, models
0x4E and 0x5E, aka INTEL_FAM6_SKYLAKE and INTEL_FAM6_SKYLAKE_L.
Reported-by: Michael Zhivich <mzhivich@akamai.com>
Fixes: 8974eb588283 ("x86/speculation: Add Gather Data Sampling mitigation")
Signed-off-by: Dave Hansen <dave.hansen@linux.intel.com>
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Reviewed-by: Daniel Sneddon <daniel.sneddon@linux.intel.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Sasha Levin <sashal@kernel.org>Neither, which are very good solutions from security point of view.
trevor-signal
commented
10 months ago Anyone willing to get us a crash dump, so we can understand exactly what's responsible? For that, you'll need to either run Signal Beta, or start the app with the --enable-crash-reports flag. Crash reports should live at ~/.config/Signal/Crashpad/pending.
kmille
commented
10 months ago It's a bit strange. If I boot normal linux (6.5.9.arch2-1), signal-desktop still works without an issue. Booting linux-hardened without gather_data_sampling=off leads to a crash.
crash report on linux-hardened: crash-report.zip
nikkolade
commented
10 months ago Here is the crash dump from using kernel v6.1.60 (kernel.org) & Debian 12 (amd64). HW: model name : Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz
trevor-signal
commented
10 months ago Thanks for those crash dumps! Unfortunately it looks like the crash is coming from Electron.
kmille
commented
9 months ago It looks like it's fixed upstream:
The whole thing is a bit confusing. Do you already use these upstream fixes? Then I will test it...
scottnonnenberg-signal
commented
9 months ago Watch for beta 6.41, coming soon - I think it will have the necessary fixes.
hardfalcon
commented
9 months ago Watch for beta 6.41, coming soon - I think it will have the necessary fixes.
It seems that the 6.40.x branch, the 6.41.x branch and the main branch are currently all using Electron 25.9.3:
Apart from the current Electron 25.x release being version 25.9.8, and the Electron 25.x releases after 25.9.3 containing several security fixes (though I don't know how many of them are actually relevant for Signal), the Electron project have made it clear that they do not intend to backport any fixes for AVX-related crashes into their 25-x-y branch. But regardless of this, Electron 25.x is EOL anyway since 2023-12-05.
As far as the AVX-related crashes are concerned, the current Electron >=27.1.3 and Electron >=28.0.0 should be fine because their node version uses a sufficiently new release of its simdutf dependency, and the Electron project has added a backport of the fix for the flawed AVX detection routine in node's base64 dependency.
The only non-EOL branch of Electron that does not contain both fixes seems to be 26-x-y, which only contains the fix for node's base64 dependency but currently still lacks the fix for node's simdutf library (though they seem to be working on this). Since you have to change to another major Electron release anyway, it would probably make the most sense to use their latest release, giving you the longest possible timeframe until the next change to a newer major release of Electron becomes necessary.
By the way, out of curiosity, what was the reason for reverting from Electron 26.x back to 25.x in September?
scottnonnenberg-signal
commented
9 months ago @hardfalcon Electron 26 resulted in crashes on linux. Check out the latest beta - it's on a newer version of Electron. You can install it here: https://support.signal.org/hc/en-us/articles/360007318471-Signal-Beta
hardfalcon
commented
9 months ago Thanks a lot for the explanation and the updates, looking forward to the release of signal-desktop 6.42.0 :)
kmille
commented
8 months ago Using signal-desktop 6.42.0-1 on Arch Linux, the issue is finally resolved.
nikkolade
commented
8 months ago I also confirm that the latest v6.42.0 now works on Debian 12 + kernel.org v6.1.69 just fine
Bug Description
If I start signal-desktop, it crashes.
Steps to Reproduce
Platform Info
Arch Linux x86 with signal-desktop version 6.30.2-1