search

cockpit-project / bots

*beep* *boop* *beep*
GNU Lesser General Public License v2.1
26 stars 33 forks source link

dnsmasq crashes in lookup_domain #2435

Closed martinpitt closed 2 years ago

martinpitt commented 3 years ago

Downstream report: https://bugzilla.redhat.com/show_bug.cgi?id=2006367

cockpituous commented 3 years ago

fedora-coreos Ooops, it happened again

# ----------------------------------------------------------------------
# testBasic (__main__.TestBonding)
NAME                UUID                                  TYPE      DEVICE      
Wired connection 1  96a5a059-8d97-3cb3-840e-d2c1a9ffe4b2  ethernet  ens14       
cni-podman0         b1ed7cf5-6281-406c-8dae-aa4450d33c7f  bridge    cni-podman0 
mcast1              4bf68e6e-d2d4-345b-aa95-2259ef3bc196  ethernet  ens15       

/usr/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
[1004/133709.839863:ERROR:bus.cc(392)] Failed to connect to the bus: Failed to connect to socket /run/dbus/system_bus_socket: No such file or directory

DevTools listening on ws://127.0.0.1:9485/devtools/browser/0abda827-3b27-426f-9b47-22d326eb5062
[1004/133709.857894:ERROR:gpu_init.cc(441)] Passthrough is not supported, GL is swiftshader
CDP: {"source":"security","level":"error","text":"Refused to apply inline style because it violates the following Content Security Policy directive: \"default-src 'self' http://127.0.0.2:9091\". Either the 'unsafe-inline' keyword, a hash ('sha256-BMIkowAqVEI88ibUJ1NtYB8ByZOuS4MTi73eGoFo2+Q='), or a nonce ('nonce-...') is required to enable inline execution. Note that hashes do not apply to event handlers, style attributes and javascript: navigations unless the 'unsafe-hashes' keyword is present. Note also that 'style-src' was not explicitly set, so 'default-src' is used as a fallback.\n","timestamp":1633354631393.485,"url":"http://127.0.0.2:9091/cockpit/$6ef2ca770d0c3da8e95735f5218488003eb7a376db13acfd024daecc1f23e983/shell/index.js","lineNumber":1,"stackTrace":{"callFrames":[{"functionName":"ke","scriptId":"24","url":"http://127.0.0.2:9091/cockpit/$6ef2ca770d0c3da8e95735f5218488003eb7a376db13acfd024daecc1f23e983/shell/index.js","lineNumber":1,"columnNumber":121721},{"functionName":"Ze","scriptId":"24","url":"http://127.0.0.2:9091/cockpit/$6ef2ca770d0c3da8e95735f5218488003eb7a376db13acfd024daecc1f23e983/shell/index.js","lineNumber":1,"columnNumber":130758},{"functionName":"append","scriptId":"24","url":"http://127.0.0.2:9091/cockpit/$6ef2ca770d0c3da8e95735f5218488003eb7a376db13acfd024daecc1f23e983/shell/index.js","lineNumber":1,"columnNumber":132439},{"functionName":"D","scriptId":"24","url":"http://127.0.0.2:9091/cockpit/$6ef2ca770d0c3da8e95735f5218488003eb7a376db13acfd024daecc1f23e983/shell/index.js","lineNumber":1,"columnNumber":349379},{"functionName":"","scriptId":"24","url":"http://127.0.0.2:9091/cockpit/$6ef2ca770d0c3da8e95735f5218488003eb7a376db13acfd024daecc1f23e983/shell/index.js","lineNumber":1,"columnNumber":358581},{"functionName":"","scriptId":"24","url":"http://127.0.0.2:9091/cockpit/$6ef2ca770d0c3da8e95735f5218488003eb7a376db13acfd024daecc1f23e983/shell/index.js","lineNumber":1,"columnNumber":643366},{"functionName":"","scriptId":"24","url":"http://127.0.0.2:9091/cockpit/$6ef2ca770d0c3da8e95735f5218488003eb7a376db13acfd024daecc1f23e983/shell/index.js","lineNumber":1,"columnNumber":643370}]}}
> error: Modal: Specify at least one of: title, aria-label, aria-labelledby.
> warning: grep: /sys/class/dmi/id/power/autosuspend_delay_ms: Input/output error

> warning: Incomplete 'Connection' object accessed 
> log: Error
    at ca (http://127.0.0.2:9091/cockpit/$6ef2ca770d0c3da8e95735f5218488003eb7a376db13acfd024daecc1f23e983/network/network.js:2:194937)
    at Array.Z.exporters (http://127.0.0.2:9091/cockpit/$6ef2ca770d0c3da8e95735f5218488003eb7a376db13acfd024daecc1f23e983/network/network.js:2:203528)
    at http://127.0.0.2:9091/cockpit/$6ef2ca770d0c3da8e95735f5218488003eb7a376db13acfd024daecc1f23e983/network/network.js:2:196868
    at http://127.0.0.2:9091/cockpit/$6ef2ca770d0c3da8e95735f5218488003eb7a376db13acfd024daecc1f23e983/network/network.js:2:196934
Process 18183 (dnsmasq) of user 985 dumped core.
Stack trace of thread 18183:
#0  0x000055a53ce8d256 lookup_domain (dnsmasq + 0x53256)
#1  0x000055a53ce5da3a forward_query.lto_priv.0 (dnsmasq + 0x23a3a)
#2  0x000055a53ce625d0 check_dns_listeners (dnsmasq + 0x285d0)
#3  0x000055a53ce46b00 main (dnsmasq + 0xcb00)
#4  0x00007fb00529eb75 __libc_start_main (libc.so.6 + 0x27b75)
#5  0x000055a53ce4748e _start (dnsmasq + 0xd48e)
Wrote JS log to TestBonding-testBasic-fedora-coreos-127.0.0.2-2201-FAIL.js.log
Journal extracted to TestBonding-testBasic-fedora-coreos-127.0.0.2-2201-FAIL.log.gz
Core dumps downloaded to /work/bots/make-checkout-workdir/TestBonding-testBasic-fedora-coreos-127.0.0.2-2201-FAIL.core
bash: line 1: kill: (18183) - No such process
> warning: Unknown device "/sys/devices/virtual/net/vethbde8fdd0": No such device

> warning: Unknown device "/sys/devices/virtual/net/vethbde8fdd0": No such device
> warning: 
Traceback (most recent call last):
  File "/work/bots/make-checkout-workdir/test/common/testlib.py", line 1180, in tearDown
    self.check_journal_messages()
  File "/work/bots/make-checkout-workdir/test/common/testlib.py", line 1383, in check_journal_messages
    raise Error(UNEXPECTED_MESSAGE + "journal messages:\n" + first)
testlib.Error: FAIL: Test completed, but found unexpected journal messages:
Process 18183 (dnsmasq) of user 985 dumped core.

Traceback (most recent call last):
  File "/work/bots/make-checkout-workdir/bots/machine/machine_core/ssh_connection.py", line 359, in execute
    raise subprocess.CalledProcessError(proc.returncode, command, output=output)
subprocess.CalledProcessError: Command 'kill 18183' returned non-zero exit status 1.

# Result testBasic (__main__.TestBonding) failed
# 1 TEST FAILED [24s on 2-ci-srv-04]
not ok 289 test/verify/check-networkmanager-bond TestBonding.testBasic [ND@0]

First occurrence: 2021-10-04T13:37:29.711408 | revision 9dff678355ef89d2b42198513b5cb6d58ea22857 Times recorded: 1 Latest occurrences:

# ----------------------------------------------------------------------
# testBasic (__main__.TestBridge)
NAME                UUID                                  TYPE      DEVICE      
Wired connection 1  65f32e3c-b198-34dc-9818-ff5ffd1168c6  ethernet  ens14       
cni-podman0         9b3a233a-97ca-40e9-9da5-2e3310c9b0e1  bridge    cni-podman0 
mcast1              4bf68e6e-d2d4-345b-aa95-2259ef3bc196  ethernet  ens15       

/usr/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
[1013/123046.441671:ERROR:bus.cc(392)] Failed to connect to the bus: Failed to connect to socket /run/dbus/system_bus_socket: No such file or directory

DevTools listening on ws://127.0.0.1:9299/devtools/browser/138abcc1-3a3e-49e1-9c86-03b7b4303d7f
[1013/123046.457247:ERROR:gpu_init.cc(441)] Passthrough is not supported, GL is swiftshader
CDP: {"source":"security","level":"error","text":"Refused to apply inline style because it violates the following Content Security Policy directive: \"default-src 'self' http://127.0.0.2:9091\". Either the 'unsafe-inline' keyword, a hash ('sha256-BMIkowAqVEI88ibUJ1NtYB8ByZOuS4MTi73eGoFo2+Q='), or a nonce ('nonce-...') is required to enable inline execution. Note that hashes do not apply to event handlers, style attributes and javascript: navigations unless the 'unsafe-hashes' keyword is present. Note also that 'style-src' was not explicitly set, so 'default-src' is used as a fallback.\n","timestamp":1634128248012.75,"url":"http://127.0.0.2:9091/cockpit/$cf217e6c1a24a7ac1de27f426f04269a7cb0da706577ff0a1e9d1dfdff64fbcc/shell/index.js","lineNumber":1,"stackTrace":{"callFrames":[{"functionName":"ke","scriptId":"24","url":"http://127.0.0.2:9091/cockpit/$cf217e6c1a24a7ac1de27f426f04269a7cb0da706577ff0a1e9d1dfdff64fbcc/shell/index.js","lineNumber":1,"columnNumber":121721},{"functionName":"Ze","scriptId":"24","url":"http://127.0.0.2:9091/cockpit/$cf217e6c1a24a7ac1de27f426f04269a7cb0da706577ff0a1e9d1dfdff64fbcc/shell/index.js","lineNumber":1,"columnNumber":130758},{"functionName":"append","scriptId":"24","url":"http://127.0.0.2:9091/cockpit/$cf217e6c1a24a7ac1de27f426f04269a7cb0da706577ff0a1e9d1dfdff64fbcc/shell/index.js","lineNumber":1,"columnNumber":132439},{"functionName":"D","scriptId":"24","url":"http://127.0.0.2:9091/cockpit/$cf217e6c1a24a7ac1de27f426f04269a7cb0da706577ff0a1e9d1dfdff64fbcc/shell/index.js","lineNumber":1,"columnNumber":349379},{"functionName":"","scriptId":"24","url":"http://127.0.0.2:9091/cockpit/$cf217e6c1a24a7ac1de27f426f04269a7cb0da706577ff0a1e9d1dfdff64fbcc/shell/index.js","lineNumber":1,"columnNumber":358581},{"functionName":"","scriptId":"24","url":"http://127.0.0.2:9091/cockpit/$cf217e6c1a24a7ac1de27f426f04269a7cb0da706577ff0a1e9d1dfdff64fbcc/shell/index.js","lineNumber":1,"columnNumber":643600},{"functionName":"","scriptId":"24","url":"http://127.0.0.2:9091/cockpit/$cf217e6c1a24a7ac1de27f426f04269a7cb0da706577ff0a1e9d1dfdff64fbcc/shell/index.js","lineNumber":1,"columnNumber":643604}]}}
> error: Modal: Specify at least one of: title, aria-label, aria-labelledby.
> warning: grep: /sys/class/dmi/id/power/autosuspend_delay_ms: Input/output error

> warning: Incomplete 'Connection' object accessed 
> log: Error
    at ra (http://127.0.0.2:9091/cockpit/$cf217e6c1a24a7ac1de27f426f04269a7cb0da706577ff0a1e9d1dfdff64fbcc/network/network.js:2:187149)
    at Array.$.exporters (http://127.0.0.2:9091/cockpit/$cf217e6c1a24a7ac1de27f426f04269a7cb0da706577ff0a1e9d1dfdff64fbcc/network/network.js:2:195740)
    at http://127.0.0.2:9091/cockpit/$cf217e6c1a24a7ac1de27f426f04269a7cb0da706577ff0a1e9d1dfdff64fbcc/network/network.js:2:189080
    at http://127.0.0.2:9091/cockpit/$cf217e6c1a24a7ac1de27f426f04269a7cb0da706577ff0a1e9d1dfdff64fbcc/network/network.js:2:189146
Process 15599 (dnsmasq) of user 985 dumped core.
Stack trace of thread 15599:
#0  0x0000556a32bac256 lookup_domain (dnsmasq + 0x53256)
#1  0x0000556a32b7ca3a forward_query.lto_priv.0 (dnsmasq + 0x23a3a)
#2  0x0000556a32b815d0 check_dns_listeners (dnsmasq + 0x285d0)
#3  0x0000556a32b65b00 main (dnsmasq + 0xcb00)
#4  0x00007ff781685b75 __libc_start_main (libc.so.6 + 0x27b75)
#5  0x0000556a32b6648e _start (dnsmasq + 0xd48e)
Wrote JS log to TestBridge-testBasic-fedora-coreos-127.0.0.2-2201-FAIL.js.log
Journal extracted to TestBridge-testBasic-fedora-coreos-127.0.0.2-2201-FAIL.log.gz
Core dumps downloaded to /work/bots/make-checkout-workdir/TestBridge-testBasic-fedora-coreos-127.0.0.2-2201-FAIL.core
bash: line 1: kill: (15599) - No such process
> warning: Unknown device "/sys/devices/virtual/net/veth73e44edb": No such device

> warning: Unknown device "/sys/devices/virtual/net/veth73e44edb": No such device
> warning: 
Traceback (most recent call last):
  File "/work/bots/make-checkout-workdir/test/common/testlib.py", line 1180, in tearDown
    self.check_journal_messages()
  File "/work/bots/make-checkout-workdir/test/common/testlib.py", line 1383, in check_journal_messages
    raise Error(UNEXPECTED_MESSAGE + "journal messages:\n" + first)
testlib.Error: FAIL: Test completed, but found unexpected journal messages:
Process 15599 (dnsmasq) of user 985 dumped core.

Traceback (most recent call last):
  File "/work/bots/make-checkout-workdir/bots/machine/machine_core/ssh_connection.py", line 359, in execute
    raise subprocess.CalledProcessError(proc.returncode, command, output=output)
subprocess.CalledProcessError: Command 'kill 15599' returned non-zero exit status 1.

# Result testBasic (__main__.TestBridge) failed
# 1 TEST FAILED [25s on 1-ci-srv-02]
not ok 56 test/verify/check-networkmanager-bridge TestBridge.testBasic [ND@0]

First occurrence: 2021-10-13T12:31:07.684857 | revision a3fe0c8b72ad62d2963b60e2b4ec4464bcfd535a Times recorded: 1 Latest occurrences:

# ----------------------------------------------------------------------
# testBasic (__main__.TestBridge)
NAME                UUID                                  TYPE      DEVICE      
Wired connection 1  31053780-0fb3-3954-adfc-ba0901a16a98  ethernet  ens14       
cni-podman0         3ba07799-3827-4523-9880-ce124e2279e1  bridge    cni-podman0 
mcast1              4bf68e6e-d2d4-345b-aa95-2259ef3bc196  ethernet  ens15       

DevTools listening on ws://127.0.0.1:9276/devtools/browser/ac94f3da-f8a1-4b6b-9a39-f456d00080b0
[1015/151912.100892:ERROR:egl_util.cc(74)] Failed to load GLES library: /usr/lib64/chromium-browser/swiftshader/libGLESv2.so: /usr/lib64/chromium-browser/swiftshader/libGLESv2.so: cannot open shared object file: No such file or directory
[1015/151912.125256:ERROR:viz_main_impl.cc(162)] Exiting GPU process due to errors during initialization
[1015/151912.141971:ERROR:gpu_init.cc(441)] Passthrough is not supported, GL is disabled
CDP: {"source":"security","level":"error","text":"Refused to apply inline style because it violates the following Content Security Policy directive: \"default-src 'self' http://127.0.0.2:9091\". Either the 'unsafe-inline' keyword, a hash ('sha256-BMIkowAqVEI88ibUJ1NtYB8ByZOuS4MTi73eGoFo2+Q='), or a nonce ('nonce-...') is required to enable inline execution. Note that hashes do not apply to event handlers, style attributes and javascript: navigations unless the 'unsafe-hashes' keyword is present. Note also that 'style-src' was not explicitly set, so 'default-src' is used as a fallback.\n","timestamp":1634311153711.487,"url":"http://127.0.0.2:9091/cockpit/$cabf5ec154187b86978a5baf0cf9d39ab847193a1c8eb2086ca00af61d5011b3/shell/index.js","lineNumber":1,"stackTrace":{"callFrames":[{"functionName":"Ce","scriptId":"24","url":"http://127.0.0.2:9091/cockpit/$cabf5ec154187b86978a5baf0cf9d39ab847193a1c8eb2086ca00af61d5011b3/shell/index.js","lineNumber":1,"columnNumber":154522},{"functionName":"Ze","scriptId":"24","url":"http://127.0.0.2:9091/cockpit/$cabf5ec154187b86978a5baf0cf9d39ab847193a1c8eb2086ca00af61d5011b3/shell/index.js","lineNumber":1,"columnNumber":163559},{"functionName":"append","scriptId":"24","url":"http://127.0.0.2:9091/cockpit/$cabf5ec154187b86978a5baf0cf9d39ab847193a1c8eb2086ca00af61d5011b3/shell/index.js","lineNumber":1,"columnNumber":165240},{"functionName":"P","scriptId":"24","url":"http://127.0.0.2:9091/cockpit/$cabf5ec154187b86978a5baf0cf9d39ab847193a1c8eb2086ca00af61d5011b3/shell/index.js","lineNumber":1,"columnNumber":400072},{"functionName":"","scriptId":"24","url":"http://127.0.0.2:9091/cockpit/$cabf5ec154187b86978a5baf0cf9d39ab847193a1c8eb2086ca00af61d5011b3/shell/index.js","lineNumber":1,"columnNumber":409274},{"functionName":"","scriptId":"24","url":"http://127.0.0.2:9091/cockpit/$cabf5ec154187b86978a5baf0cf9d39ab847193a1c8eb2086ca00af61d5011b3/shell/index.js","lineNumber":1,"columnNumber":711994},{"functionName":"","scriptId":"24","url":"http://127.0.0.2:9091/cockpit/$cabf5ec154187b86978a5baf0cf9d39ab847193a1c8eb2086ca00af61d5011b3/shell/index.js","lineNumber":1,"columnNumber":711998}]}}
> error: Modal: Specify at least one of: title, aria-label, aria-labelledby.
> warning: grep: /sys/class/dmi/id/power/autosuspend_delay_ms: Input/output error

Process 17264 (dnsmasq) of user 985 dumped core.
Stack trace of thread 17264:
#0  0x000055aa3940d256 lookup_domain (dnsmasq + 0x53256)
#1  0x000055aa393dda3a forward_query.lto_priv.0 (dnsmasq + 0x23a3a)
#2  0x000055aa393e25d0 check_dns_listeners (dnsmasq + 0x285d0)
#3  0x000055aa393c6b00 main (dnsmasq + 0xcb00)
#4  0x00007f39600ccb75 __libc_start_main (libc.so.6 + 0x27b75)
#5  0x000055aa393c748e _start (dnsmasq + 0xd48e)
Wrote JS log to TestBridge-testBasic-fedora-coreos-127.0.0.2-2201-FAIL.js.log
Journal extracted to TestBridge-testBasic-fedora-coreos-127.0.0.2-2201-FAIL.log.gz
Core dumps downloaded to /work/bots/make-checkout-workdir/TestBridge-testBasic-fedora-coreos-127.0.0.2-2201-FAIL.core
bash: line 1: kill: (17264) - No such process
> warning: Unknown device "/sys/devices/virtual/net/veth58aeeeff": No such device

> warning: Unknown device "/sys/devices/virtual/net/veth58aeeeff": No such device
> warning: 
Traceback (most recent call last):
  File "/work/bots/make-checkout-workdir/test/common/testlib.py", line 1180, in tearDown
    self.check_journal_messages()
  File "/work/bots/make-checkout-workdir/test/common/testlib.py", line 1383, in check_journal_messages
    raise Error(UNEXPECTED_MESSAGE + "journal messages:\n" + first)
testlib.Error: FAIL: Test completed, but found unexpected journal messages:
Process 17264 (dnsmasq) of user 985 dumped core.

Traceback (most recent call last):
  File "/work/bots/make-checkout-workdir/bots/machine/machine_core/ssh_connection.py", line 359, in execute
    raise subprocess.CalledProcessError(proc.returncode, command, output=output)
subprocess.CalledProcessError: Command 'kill 17264' returned non-zero exit status 1.

# Result testBasic (__main__.TestBridge) failed
# 1 TEST FAILED [23s on 3-ci-srv-01]
not ok 72 test/verify/check-networkmanager-bridge TestBridge.testBasic [ND@0]

First occurrence: 2021-10-15T15:19:31.065046 | revision 14527cbc3354c5604bf9b48ef10b8609a2e6cb90 Times recorded: 1 Latest occurrences:

# ----------------------------------------------------------------------
# testBasic (__main__.TestBridge)
NAME                UUID                                  TYPE      DEVICE      
Wired connection 1  d086ca06-869d-3f33-88f8-18973e133c42  ethernet  ens14       
cni-podman0         0a90eac2-8c6c-43b4-b40d-7929666976b6  bridge    cni-podman0 
mcast1              4bf68e6e-d2d4-345b-aa95-2259ef3bc196  ethernet  ens15       

DevTools listening on ws://127.0.0.1:9569/devtools/browser/aa00baca-8bdf-4516-8649-f7b01a7c032c
[1102/174419.374154:ERROR:egl_util.cc(74)] Failed to load GLES library: /usr/lib64/chromium-browser/swiftshader/libGLESv2.so: /usr/lib64/chromium-browser/swiftshader/libGLESv2.so: cannot open shared object file: No such file or directory
[1102/174419.376511:ERROR:viz_main_impl.cc(162)] Exiting GPU process due to errors during initialization
[1102/174419.379814:ERROR:gpu_init.cc(441)] Passthrough is not supported, GL is disabled
> error: Modal: Specify at least one of: title, aria-label, aria-labelledby.
> warning: grep: /sys/class/dmi/id/power/autosuspend_delay_ms: Input/output error

Process 35857 (dnsmasq) of user 985 dumped core.
Stack trace of thread 35857:
#0  0x0000559b1b3382de lookup_domain (dnsmasq + 0x532de)
#1  0x0000559b1b308a3a forward_query.lto_priv.0 (dnsmasq + 0x23a3a)
#2  0x0000559b1b30d5d0 check_dns_listeners (dnsmasq + 0x285d0)
#3  0x0000559b1b2f1b00 main (dnsmasq + 0xcb00)
#4  0x00007f231d98ab75 __libc_start_main (libc.so.6 + 0x27b75)
#5  0x0000559b1b2f248e _start (dnsmasq + 0xd48e)
Wrote JS log to TestBridge-testBasic-fedora-coreos-127.0.0.2-2201-FAIL.js.log
Journal extracted to TestBridge-testBasic-fedora-coreos-127.0.0.2-2201-FAIL.log.gz
Core dumps downloaded to /work/bots/make-checkout-workdir/TestBridge-testBasic-fedora-coreos-127.0.0.2-2201-FAIL.core
bash: line 1: kill: (35857) - No such process
> warning: Unknown device "/sys/devices/virtual/net/veth30a3921c": No such device

> warning: Unknown device "/sys/devices/virtual/net/veth30a3921c": No such device
> warning: 
Traceback (most recent call last):
  File "/work/bots/make-checkout-workdir/test/common/testlib.py", line 1183, in tearDown
    self.check_journal_messages()
  File "/work/bots/make-checkout-workdir/test/common/testlib.py", line 1386, in check_journal_messages
    raise Error(UNEXPECTED_MESSAGE + "journal messages:\n" + first)
testlib.Error: FAIL: Test completed, but found unexpected journal messages:
Process 35857 (dnsmasq) of user 985 dumped core.

Traceback (most recent call last):
  File "/work/bots/make-checkout-workdir/bots/machine/machine_core/ssh_connection.py", line 359, in execute
    raise subprocess.CalledProcessError(proc.returncode, command, output=output)
subprocess.CalledProcessError: Command 'kill 35857' returned non-zero exit status 1.

# Result testBasic (__main__.TestBridge) failed
# 1 TEST FAILED [23s on 4-cockpit-9]
not ok 73 test/verify/check-networkmanager-bridge TestBridge.testBasic [ND@0]

First occurrence: 2021-11-02T17:44:38.989882 | revision dbbc13bc6a5658546e7d5aaa7057cf224fa0ae2d Times recorded: 1 Latest occurrences:

pemensik commented 3 years ago

Hello, can I ask if you could attach any coredumps you might have to linked bugzilla?

Do you know what triggers the crash? Can that be change of resolvers or NM connection during the test? Or maybe disabled connection then resumed?

cockpituous commented 3 years ago

fedora-35 Ooops, it happened again

# ----------------------------------------------------------------------
# testBasic (__main__.TestBonding)
NAME         UUID                                  TYPE      DEVICE 
System eth0  5fb06bd0-0bb0-7ffb-45f1-d6edd65f3e03  ethernet  eth0   
virbr0       485f0149-1752-4dc6-9706-1b8d76664c01  bridge    virbr0 
System eth1  9c92fad9-6ecb-3e6c-eb4d-8a47c6f50c04  ethernet  --     

DevTools listening on ws://127.0.0.1:10028/devtools/browser/aebd340a-cb1f-4e8b-96c7-f5ad5c87d4c2
[1102/101126.641765:ERROR:egl_util.cc(74)] Failed to load GLES library: /usr/lib64/chromium-browser/swiftshader/libGLESv2.so: /usr/lib64/chromium-browser/swiftshader/libGLESv2.so: cannot open shared object file: No such file or directory
[1102/101126.644366:ERROR:viz_main_impl.cc(162)] Exiting GPU process due to errors during initialization
[1102/101126.647883:ERROR:gpu_init.cc(441)] Passthrough is not supported, GL is disabled
> error: Modal: Specify at least one of: title, aria-label, aria-labelledby.
> warning: grep: /sys/class/dmi/id/power/autosuspend_delay_ms: Input/output error

> warning: Resolving coreutils failed: {"detail":"cannot update repo 'updates': Cannot prepare internal mirrorlist: Curl error (6): Couldn't resolve host name for https://mirrors.fedoraproject.org/metalink?repo=updates-released-f35&arch=x86_64 [Could not resolve host: mirrors.fedoraproject.org]; Last error: Curl error (6): Couldn't resolve host name for https://mirrors.fedoraproject.org/metalink?repo=updates-released-f35&arch=x86_64 [Could not resolve host: mirrors.fedoraproject.org]","code":64}
> warning: loading available updates failed: {"detail":"cannot update repo 'updates': Cannot prepare internal mirrorlist: Curl error (6): Couldn't resolve host name for https://mirrors.fedoraproject.org/metalink?repo=updates-released-f35&arch=x86_64 [Could not resolve host: mirrors.fedoraproject.org]; Last error: Curl error (6): Couldn't resolve host name for https://mirrors.fedoraproject.org/metalink?repo=updates-released-f35&arch=x86_64 [Could not resolve host: mirrors.fedoraproject.org]","code":64}
Warning: ALREADY_ENABLED: 'dhcp' already in 'public'
Warning: ALREADY_ENABLED: 'dhcp' already in 'public'
> warning: Incomplete 'Connection' object accessed 
> log: Error
    at Si (http://127.0.0.2:9191/cockpit/@localhost/network/network.js:2:212341)
    at Array.Ni.z.exporters (http://127.0.0.2:9191/cockpit/@localhost/network/network.js:2:221031)
    at http://127.0.0.2:9191/cockpit/@localhost/network/network.js:2:214302
    at http://127.0.0.2:9191/cockpit/@localhost/network/network.js:2:214368
Process 88954 (dnsmasq) of user 993 dumped core.
Found module linux-vdso.so.1 with build-id: 940012310595371542638aa7ba497eedc2be9d90
Found module libnss_sss.so.2 with build-id: f8ff93ecdf21e431b595f5ea823fe396d56da42d
Found module libgpg-error.so.0 with build-id: 42587ce26856ad15cf5630134f114599783477ed
Found module libgcc_s.so.1 with build-id: 279c1648ee67377b5de1c45a244740beea48423e
Found module libgcrypt.so.20 with build-id: 711d41580c5a8649a79a8430a985dac3e25b5ba2
Found module libcap.so.2 with build-id: 0214aa9cc6a8646eb9ec27ab7dda6a2219da6500
Found module liblz4.so.1 with build-id: fd02c4542a1ce1ad6c503b958447be1c37a4afee
Found module libzstd.so.1 with build-id: c594c701d8f5ba174a5be306973fa9803d65e327
Found module liblzma.so.5 with build-id: 7fec53ce7cba9489f130c99cfd3ace82e9dde0ee
Found module ld-linux-x86-64.so.2 with build-id: 7d2c06364366496318a96401aee1c204f80ee73a
Found module libunistring.so.2 with build-id: edcae9db236efd11e61995054ddf27a89ec6dc40
Found module libsystemd.so.0 with build-id: ed341b20f80da429e2b509b86da79f8616d5b027
Found module libc.so.6 with build-id: f891252f9069edee265f92cfb9a163880999588b
Found module libgmp.so.10 with build-id: 14dbc5e93375ccb760dc47c5aaecc0cdfc45a06a
Found module libhogweed.so.6 with build-id: 288da1bb6ffaf6600c2f8a126bec247f2e9a4628
Found module libnettle.so.8 with build-id: 62dba2572638edf8336bd4de5017fc70eaf0a567
Found module libidn2.so.0 with build-id: f3ad1b0687f5b426beed2d3d3d18aa301932f7a3
Found module libdbus-1.so.3 with build-id: 58e800f0865bde326c5c9ce510062d5b19902f67
Found module dnsmasq with build-id: bd8761b5091ab2e87fb89cd925ef836c3e0ff06a
Stack trace of thread 88954:
#0  0x000055d991c3f2be lookup_domain (dnsmasq + 0x532be)
#1  0x000055d991c0fa2a forward_query.lto_priv.0 (dnsmasq + 0x23a2a)
#2  0x000055d991c145c0 check_dns_listeners (dnsmasq + 0x285c0)
#3  0x000055d991bf8d71 main (dnsmasq + 0xcd71)
#4  0x00007f47ae2d2560 __libc_start_call_main (libc.so.6 + 0x2d560)
#5  0x00007f47ae2d260c __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x2d60c)
#6  0x000055d991bf9475 _start (dnsmasq + 0xd475)
Wrote JS log to TestBonding-testBasic-fedora-35-127.0.0.2-2301-FAIL.js.log
Journal extracted to TestBonding-testBasic-fedora-35-127.0.0.2-2301-FAIL.log.gz
Core dumps downloaded to /work/bots/make-checkout-workdir/TestBonding-testBasic-fedora-35-127.0.0.2-2301-FAIL.core
bash: line 1: kill: (88954) - No such process
> warning: transport closed: disconnected
Failed to get session path: No session '249' known
Traceback (most recent call last):
  File "/work/bots/make-checkout-workdir/test/common/testlib.py", line 1183, in tearDown
    self.check_journal_messages()
  File "/work/bots/make-checkout-workdir/test/common/testlib.py", line 1386, in check_journal_messages
    raise Error(UNEXPECTED_MESSAGE + "journal messages:\n" + first)
testlib.Error: FAIL: Test completed, but found unexpected journal messages:
Process 88954 (dnsmasq) of user 993 dumped core.

Traceback (most recent call last):
  File "/work/bots/make-checkout-workdir/bots/machine/machine_core/ssh_connection.py", line 359, in execute
    raise subprocess.CalledProcessError(proc.returncode, command, output=output)
subprocess.CalledProcessError: Command 'kill 88954' returned non-zero exit status 1.

# Result testBasic (__main__.TestBonding) failed
# 1 TEST FAILED [23s on 4-ci-srv-06]
not ok 54 test/verify/check-networkmanager-bond TestBonding.testBasic [ND@1]

First occurrence: 2021-11-02T10:11:47.854041 | revision 572c750e7faa95a7be8fde47c10b88a283ffe5f2 Times recorded: 1 Latest occurrences:

pemensik commented 3 years ago

@cockpituous Can you check what version of dnsmasq were installed? F35 should already have in stable dnsmasq-2.86-3.fc35, which already contains candidate fix. If that were this version already, it would help if you can share coredump.

I were unable to find mentioned dnsmasq with build-id: bd8761b5091ab2e87fb89cd925ef836c3e0ff06a in x86_64 f35 builds. Is that different architecture?

martinpitt commented 3 years ago

@pemensik : Indeed our current image has the previous version dnsmasq-2.86-2.fc35.x86_64 (and we only test on that arch, we don't have others available). Our last image refresh was 4 days ago, so the next one is scheduled to be in 3 days. But I manually triggered a refresh now, see #2586 -- then we can see if it still happens from now on. Thanks!