search

openbmc / openbmc-build-scripts

Apache License 2.0
19 stars 50 forks source link

sanitizer tests fail on ppc64le #31

Open geissonator opened 3 years ago

geissonator commented 3 years ago

After updating our docker Ubuntu to the latest (hirsute) via https://github.com/openbmc/openbmc-build-scripts/commit/73395159e5729d163fcc1f77cd3a116ddd251b59?, we started to see a hard fail on the sanitizer tests in CI on ppc64le machines. The fail looks like this:

[107/107] Linking target libpldmresponder/test/libpldmresponder_bios_config_test
[1-23/23] 🌑 pldmd_registration_test                                       0/30s
 1/23 libpldm_base_test                                    OK              0.08s
 2/23 libpldm_bios_test                                    OK              0.08s
 3/23 libpldm_bios_table_test                              OK              0.08s
 4/23 libpldm_fru_test                                     OK              0.08s
 5/23 libpldm_pdr_test                                     OK              0.07s
[6-23/23] 🌒 pldmd_registration_test                                       0/30s
 6/23 libpldmresponder_bios_string_attribute_test          FAIL            0.05s   killed by signal 6 SIGABRT
>>> MALLOC_PERTURB_=121 /home/jenkins-op/workspace/CI-MISC/ci-repository-ppc64le/openbmc/pldm/build/libpldmresponder/test/libpldmresponder_bios_string_attribute_test

 7/23 libpldm_platform_test                                OK              0.09s
 8/23 libpldm_utils_test                                   OK              0.08s
 9/23 libpldmresponder_base_test                           FAIL            0.07s   killed by signal 6 SIGABRT
>>> MALLOC_PERTURB_=20 /home/jenkins-op/workspace/CI-MISC/ci-repository-ppc64le/openbmc/pldm/build/libpldmresponder/test/libpldmresponder_base_test

10/23 libpldmresponder_bios_test                           FAIL            0.07s   killed by signal 6 SIGABRT
>>> MALLOC_PERTURB_=2 /home/jenkins-op/workspace/CI-MISC/ci-repository-ppc64le/openbmc/pldm/build/libpldmresponder/test/libpldmresponder_bios_test

11/23 libpldmresponder_bios_attribute_test                 FAIL            0.07s   killed by signal 6 SIGABRT
>>> MALLOC_PERTURB_=191 /home/jenkins-op/workspace/CI-MISC/ci-repository-ppc64le/openbmc/pldm/build/libpldmresponder/test/libpldmresponder_bios_attribute_test

12/23 libpldmresponder_bios_config_test                    FAIL            0.06s   killed by signal 6 SIGABRT
>>> MALLOC_PERTURB_=6 /home/jenkins-op/workspace/CI-MISC/ci-repository-ppc64le/openbmc/pldm/build/libpldmresponder/test/libpldmresponder_bios_config_test

13/23 libpldmresponder_bios_integer_attribute_test         FAIL            0.05s   killed by signal 6 SIGABRT
>>> MALLOC_PERTURB_=19 /home/jenkins-op/workspace/CI-MISC/ci-repository-ppc64le/openbmc/pldm/build/libpldmresponder/test/libpldmresponder_bios_integer_attribute_test

14/23 libpldmresponder_bios_table_test                     FAIL            0.05s   killed by signal 6 SIGABRT
>>> MALLOC_PERTURB_=74 /home/jenkins-op/workspace/CI-MISC/ci-repository-ppc64le/openbmc/pldm/build/libpldmresponder/test/libpldmresponder_bios_table_test

15/23 libpldmresponder_fru_test                            FAIL            0.05s   killed by signal 6 SIGABRT
>>> MALLOC_PERTURB_=158 /home/jenkins-op/workspace/CI-MISC/ci-repository-ppc64le/openbmc/pldm/build/libpldmresponder/test/libpldmresponder_fru_test

16/23 libpldmresponder_bios_enum_attribute_test            FAIL            0.06s   killed by signal 6 SIGABRT
>>> MALLOC_PERTURB_=167 /home/jenkins-op/workspace/CI-MISC/ci-repository-ppc64le/openbmc/pldm/build/libpldmresponder/test/libpldmresponder_bios_enum_attribute_test

[17-23/23] 🌓 pldmd_registration_test                                      0/30s
17/23 pldm_utils_test                                      FAIL            0.03s   killed by signal 6 SIGABRT
>>> MALLOC_PERTURB_=76 /home/jenkins-op/workspace/CI-MISC/ci-repository-ppc64le/openbmc/pldm/build/common/test/pldm_utils_test

[18-23/23] 🌔 pldmd_registration_test                                      0/30s
18/23 pldmd_instanceid_test                                OK              0.03s
19/23 pldmd_registration_test                              OK              0.03s
20/23 libpldmresponder_pdr_effecter_test                   FAIL            0.04s   killed by signal 6 SIGABRT
>>> MALLOC_PERTURB_=61 /home/jenkins-op/workspace/CI-MISC/ci-repository-ppc64le/openbmc/pldm/build/libpldmresponder/test/libpldmresponder_pdr_effecter_test

21/23 dbus_to_host_effecter_test                           FAIL            0.03s   killed by signal 6 SIGABRT
>>> MALLOC_PERTURB_=215 /home/jenkins-op/workspace/CI-MISC/ci-repository-ppc64le/openbmc/pldm/build/host-bmc/test/dbus_to_host_effecter_test

[22-23/23] 🌕 libpldmresponder_pdr_sensor_test                             0/30s
22/23 libpldmresponder_pdr_sensor_test                     FAIL            0.04s   killed by signal 6 SIGABRT
>>> MALLOC_PERTURB_=19 /home/jenkins-op/workspace/CI-MISC/ci-repository-ppc64le/openbmc/pldm/build/libpldmresponder/test/libpldmresponder_pdr_sensor_test

[23/23] 🌖 libpldmresponder_platform_test                                  0/30s
23/23 libpldmresponder_platform_test                       FAIL            0.05s   killed by signal 6 SIGABRT
>>> MALLOC_PERTURB_=60 /home/jenkins-op/workspace/CI-MISC/ci-repository-ppc64le/openbmc/pldm/build/libpldmresponder/test/libpldmresponder_platform_test

Ok:                 9   
Expected Fail:      0   
Fail:               14  
Unexpected Pass:    0   
Skipped:            0   
Timeout:            0

To keep making forward progress, the current plan is to just disable this aspects of the tests on ppc64le.

The ppc64le node is running Ubuntu 18 with the latest docker and runc packages installed for that distro. One thing to try is moving the system up to Ubuntu 20 or even 21.

geissonator commented 3 years ago

Put workaround up at https://gerrit.openbmc-project.xyz/c/openbmc/openbmc-build-scripts/+/42899