DemiMarie commented 2 years ago

It turns out that RPM’s keyring and transaction functions are not thread-safe. This showed up as crashes in RPM’s atexit() handler when running the test suite, which were very difficult to debug. Fix the problem by wrapping all of these calls in a global lock. Performance should not be noticibly impacted, as the hot-path is the symmetric cryptography and that can still happen in parallel. rpmcanon is single-threaded and so is not affected.

Tested by running the test suite several times on a multi-core system. Without this patch a substantial fraction of runs will crash with SIGSEGV or SIGABRT. I cannot reproduce the crash with this patch applied.

marmarek commented 2 years ago

PipelineRetry

qubesos-bot commented 2 years ago

OpenQA test summary

Complete test suite and dependencies: https://openqa.qubes-os.org/tests/overview?distri=qubesos&version=4.1&build=2022022706-4.1&flavor=pull-requests

New failures, excluding unstable

Compared to: https://openqa.qubes-os.org/tests/overview?distri=qubesos&version=4.1&build=2022021706-4.1&flavor=update

system_tests_basic_vm_qrexec_gui_ext4
- TC_30_Gui_daemon: test_000_clipboard (failure) self.assertEqual(clipboard_content, ... AssertionError: '' != 'test23'
system_tests_network_updates
- TC_10_QvmTemplate_whonix-gw-16: test_000_template_list (failure) AssertionError: libvirt event impl drain timeout
- TC_11_QvmTemplateMgmtVM_fedora-34: test_010_template_install (failure) AssertionError: libvirt event impl drain timeout

Failed tests

20 failures

* system_tests_suspend@hw1 * suspend: [wait_serial](https://openqa.qubes-os.org/tests/35692#step/suspend/20) (wait serial expected) `# wait_serial expected: qr/p5~T5-\d+-/u...` * suspend: [Failed](https://openqa.qubes-os.org/tests/35692#step/suspend/21) (test died + timed out) `# Test died: command 'true' timed out at qubesos/tests/suspend.pm l...` * suspend: [wait_serial](https://openqa.qubes-os.org/tests/35692#step/suspend/24) (wait serial expected) `# wait_serial expected: "xl info; echo 8Ye1l-\$?-"...` * suspend: [wait_serial](https://openqa.qubes-os.org/tests/35692#step/suspend/25) (wait serial expected) `# wait_serial expected: qr/8Ye1l-\d+-/u...` * suspend: [wait_serial](https://openqa.qubes-os.org/tests/35692#step/suspend/26) (wait serial expected) `# wait_serial expected: "# "...` * suspend: [wait_serial](https://openqa.qubes-os.org/tests/35692#step/suspend/27) (wait serial expected) `# wait_serial expected: "xl list; echo MfjdI-\$?-"...` * suspend: [wait_serial](https://openqa.qubes-os.org/tests/35692#step/suspend/28) (wait serial expected) `# wait_serial expected: qr/MfjdI-\d+-/u...` * suspend: [wait_serial](https://openqa.qubes-os.org/tests/35692#step/suspend/29) (wait serial expected) `# wait_serial expected: "# "...` * suspend: [wait_serial](https://openqa.qubes-os.org/tests/35692#step/suspend/30) (wait serial expected) `# wait_serial expected: "xl dmesg; echo hNz_G-\$?-"...` * suspend: [wait_serial](https://openqa.qubes-os.org/tests/35692#step/suspend/31) (wait serial expected) `# wait_serial expected: qr/hNz_G-\d+-/u...` * suspend: [wait_serial](https://openqa.qubes-os.org/tests/35692#step/suspend/32) (wait serial expected) `# wait_serial expected: "# "...` * suspend: [wait_serial](https://openqa.qubes-os.org/tests/35692#step/suspend/33) (wait serial expected) `# wait_serial expected: "journalctl -b|tail -n 10000; echo Adw2Q-\$...` * suspend: [wait_serial](https://openqa.qubes-os.org/tests/35692#step/suspend/34) (wait serial expected) `# wait_serial expected: qr/Adw2Q-\d+-/u...` * suspend: [wait_serial](https://openqa.qubes-os.org/tests/35692#step/suspend/35) (wait serial expected) `# wait_serial expected: "# "...` * system_tests_pvgrub_salt_storage * [unstable] TC_40_PVGrub_debian-11: [test_010_template_based_vm](https://openqa.qubes-os.org/tests/35701#step/TC_40_PVGrub_debian-11/2) (error + timed out) `qubes.exc.QubesVMShutdownTimeoutError: Domain shutdown timed out: '...` * system_tests_basic_vm_qrexec_gui_ext4 * TC_03_QvmRevertTemplateChanges: [test_000_revert_linux](https://openqa.qubes-os.org/tests/35715#step/TC_03_QvmRevertTemplateChanges/1) (error) `NotImplementedError: FileVolume supports maximum 1 volume revision ...` * TC_30_Gui_daemon: [test_000_clipboard](https://openqa.qubes-os.org/tests/35715#step/TC_30_Gui_daemon/1) (failure) `self.assertEqual(clipboard_content, ... AssertionError: '' != 'test23'` * system_tests_dispvm * [unstable] TC_20_DispVM_whonix-ws-16: [test_100_open_in_dispvm](https://openqa.qubes-os.org/tests/35718#step/TC_20_DispVM_whonix-ws-16/4) (failure + cleanup) `AssertionError: libvirt event impl drain timeout` * system_tests_network_updates * TC_10_QvmTemplate_whonix-gw-16: [test_000_template_list](https://openqa.qubes-os.org/tests/35719#step/TC_10_QvmTemplate_whonix-gw-16/1) (failure) `AssertionError: libvirt event impl drain timeout` * TC_11_QvmTemplateMgmtVM_fedora-34: [test_010_template_install](https://openqa.qubes-os.org/tests/35719#step/TC_11_QvmTemplateMgmtVM_fedora-34/2) (failure) `AssertionError: libvirt event impl drain timeout`

Fixed failures

Compared to: https://openqa.qubes-os.org/tests/35157#dependencies

4 fixed

* system_tests_qrexec * TC_00_Qrexec_debian-11: [test_070_qrexec_vm_simultaneous_write](https://openqa.qubes-os.org/tests/35166#step/TC_00_Qrexec_debian-11/8) (failure) `AssertionError: Timeout, probably deadlock` * system_tests_extra * TC_00_PDFConverter_fedora-34: [test_000_one_page](https://openqa.qubes-os.org/tests/35171#step/TC_00_PDFConverter_fedora-34/1) (failure) `AssertionError: 1 != 0 : qvm-convert-pdf failed: b'Sending file to ...` * system_tests_splitgpg * TC_10_Thunderbird_debian-11: [test_010_send_receive_inline_signed_only](https://openqa.qubes-os.org/tests/35199#step/TC_10_Thunderbird_debian-11/2) (failure) `dogtail.tree.SearchError: descendent of [application | Thunderbird]...` * TC_10_Thunderbird_debian-11: [test_020_send_receive_inline_with_attachment](https://openqa.qubes-os.org/tests/35199#step/TC_10_Thunderbird_debian-11/3) (failure) `Exception: Failed to send message with error 'unknown'`

Unstable tests

* system_tests_whonix@hw1

whonixcheck/ (1/5 times with errors)

- [job 31055](https://openqa.qubes-os.org/tests/31055#step/whonixcheck/21) `None`

whonixcheck/fail (1/5 times with errors)

- [job 31055](https://openqa.qubes-os.org/tests/31055#step/whonixcheck/20) `Whonixcheck for anon-whonix failed...`

* system_tests_basic_vm_qrexec_gui@hw1

TC_30_Gui_daemon/test_000_clipboard (1/5 times with errors)

- [job 30502](https://openqa.qubes-os.org/tests/30502#step/TC_30_Gui_daemon/1) `qubes.exc.QubesMemoryError: Not enough memory to start domain 'test...`

TC_00_Basic/test_140_libvirt_events_reconnect (1/5 times with errors)

- [job 35453](https://openqa.qubes-os.org/tests/35453#step/TC_00_Basic/6) `qubes.exc.QubesVMError: Cannot connect to qrexec agent for 90 secon...`

TC_00_AppVM_whonix-ws-16/test_223_audio_play_hvm (1/5 times with errors)

- [job 31021](https://openqa.qubes-os.org/tests/31021#step/TC_00_AppVM_whonix-ws-16/17) `AssertionError: only silence detected, no useful audio data`

TC_00_AppVM_fedora-34-xfce/test_225_audio_rec_unmuted_hvm (2/5 times with errors)

- [job 30502](https://openqa.qubes-os.org/tests/30502#step/TC_00_AppVM_fedora-34-xfce/19) `AssertionError: only silence detected, no useful audio data` - [job 31021](https://openqa.qubes-os.org/tests/31021#step/TC_00_AppVM_fedora-34-xfce/19) `AssertionError: only silence detected, no useful audio data`

TC_00_AppVM_fedora-34/test_300_bug_1028_gui_memory_pinning (3/5 times with errors)

- [job 30119](https://openqa.qubes-os.org/tests/30119#step/TC_00_AppVM_fedora-34/21) `AssertionError: Dom0 window doesn't match VM window content` - [job 30502](https://openqa.qubes-os.org/tests/30502#step/TC_00_AppVM_fedora-34/21) `AssertionError: Dom0 window doesn't match VM window content` - [job 31021](https://openqa.qubes-os.org/tests/31021#step/TC_00_AppVM_fedora-34/21) `AssertionError: Dom0 window doesn't match VM window content`

TC_00_AppVM_fedora-34-xfce/test_300_bug_1028_gui_memory_pinning (3/5 times with errors)

- [job 30119](https://openqa.qubes-os.org/tests/30119#step/TC_00_AppVM_fedora-34-xfce/21) `AssertionError: Dom0 window doesn't match VM window content` - [job 30502](https://openqa.qubes-os.org/tests/30502#step/TC_00_AppVM_fedora-34-xfce/21) `AssertionError: Dom0 window doesn't match VM window content` - [job 31021](https://openqa.qubes-os.org/tests/31021#step/TC_00_AppVM_fedora-34-xfce/21) `AssertionError: Dom0 window doesn't match VM window content`

TC_00_AppVM_whonix-gw-16/test_300_bug_1028_gui_memory_pinning (1/5 times with errors)

- [job 30119](https://openqa.qubes-os.org/tests/30119#step/TC_00_AppVM_whonix-gw-16/21) `subprocess.CalledProcessError: Command 'qubes.WaitForSession' retur...`

* system_tests_gui_tools@hw1

qubesmanager_vmsettings/ (1/4 times with errors)

- [job 30515](https://openqa.qubes-os.org/tests/30515#step/qubesmanager_vmsettings/26) `None`

qubesmanager_vmsettings/Failed (1/4 times with errors)

- [job 30515](https://openqa.qubes-os.org/tests/30515#step/qubesmanager_vmsettings/27) `# Test died: no candidate needle with tag(s) 'vm-settings-def-dvm-o...`

* system_tests_whonix