search

cilium / cilium

eBPF-based Networking, Security, and Observability
https://cilium.io
Apache License 2.0
19.87k stars 2.92k forks source link

CI: RuntimePrivilegedUnitTests: panic: test timed out #19093

Closed pchaigno closed 1 year ago

pchaigno commented 2 years ago

Test Name

RuntimePrivilegedUnitTests Run Tests

Failure Output

FAIL: Failed to run privileged unit tests

Stacktrace

Click to show. ```stack-trace /home/jenkins/workspace/Cilium-PR-Runtime-net-next/runtime-gopath/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:527 Failed to run privileged unit tests Expected command: sudo make -C /home/vagrant/go/src/github.com/cilium/cilium/ tests-privileged To succeed, but it failed: Exitcode: 2 Err: Process exited with status 2 Stdout: make: Entering directory '/home/vagrant/go/src/github.com/cilium/cilium' make init-coverage make[1]: Entering directory '/home/vagrant/go/src/github.com/cilium/cilium' echo "mode: count" > coverage-all-tmp.out echo "mode: count" > coverage.out make[1]: Leaving directory '/home/vagrant/go/src/github.com/cilium/cilium' for pkg in github.com/cilium/cilium/cilium/cmd github.com/cilium/cilium/daemon/cmd github.com/cilium/cilium/pkg/bpf github.com/cilium/cilium/pkg/datapath/link github.com/cilium/cilium/pkg/datapath/linux github.com/cilium/cilium/pkg/datapath/linux/config github.com/cilium/cilium/pkg/datapath/linux/ipsec github.com/cilium/cilium/pkg/datapath/linux/probes github.com/cilium/cilium/pkg/datapath/linux/route github.com/cilium/cilium/pkg/datapath/linux/routing github.com/cilium/cilium/pkg/datapath/loader github.com/cilium/cilium/pkg/egressgateway github.com/cilium/cilium/pkg/fqdn/dnsproxy github.com/cilium/cilium/pkg/maps/ctmap github.com/cilium/cilium/pkg/maps/egressmap github.com/cilium/cilium/pkg/maps/eppolicymap github.com/cilium/cilium/pkg/maps/lbmap github.com/cilium/cilium/pkg/maps/policymap github.com/cilium/cilium/pkg/modules github.com/cilium/cilium/pkg/mountinfo github.com/cilium/cilium/pkg/mtu github.com/cilium/cilium/pkg/redirectpolicy github.com/cilium/cilium/pkg/sysctl; do \ PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/vagrant/go/src/github.com/cilium/cilium/bpf go test -mod=vendor -tags=osusergo,privileged_tests -ldflags "-X github.com/cilium/cilium/pkg/kvstore.consulDummyAddress=https://consul:8443 -X github.com/cilium/cilium/pkg/kvstore.etcdDummyAddress=http://etcd:4002 -X github.com/cilium/cilium/pkg/datapath.DatapathSHA256=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" $pkg -test.v -timeout 600s -check.vv -covermode=count -coverprofile=coverage.out -coverpkg ./... \ || exit 1; \ tail -n +2 coverage.out >> coverage-all-tmp.out; \ done === RUN Test START: encrypt_status_test.go:31: EncryptStatusSuite.SetUpSuite PASS: encrypt_status_test.go:31: EncryptStatusSuite.SetUpSuite 0.000s START: encrypt_status_test.go:53: EncryptStatusSuite.TestCountUniqueIPsecKeys PASS: encrypt_status_test.go:53: EncryptStatusSuite.TestCountUniqueIPsecKeys 0.020s START: encrypt_status_test.go:84: EncryptStatusSuite.TestGetXfrmStats PASS: encrypt_status_test.go:84: EncryptStatusSuite.TestGetXfrmStats 0.001s OK: 2 passed --- PASS: Test (0.02s) PASS coverage: 2.1% of statements in ./... ok github.com/cilium/cilium/cilium/cmd 0.135s coverage: 2.1% of statements in ./... === RUN Test START: devices_test.go:38: DevicesSuite.SetUpSuite PASS: devices_test.go:38: DevicesSuite.SetUpSuite 0.000s START: devices_test.go:66: DevicesSuite.TestDetect level=info msg="Detected devices" devices="[]" subsys=daemon level=info msg="Detected devices" devices="[dummy0]" subsys=daemon level=info msg="Detected devices" devices="[dummy0]" subsys=daemon level=info msg="Direct routing device detected" direct-routing-device=dummy1 subsys=daemon level=info msg="Detected devices" devices="[dummy0 dummy1 dummy2]" subsys=daemon level=info msg="Direct routing device detected" direct-routing-device=cilium_foo subsys=daemon level=info msg="IPv6 multicast device detected" ipv6-mcast-device=cilium_foo subsys=daemon level=info msg="Detected devices" devices="[cilium_foo dummy0 dummy1 dummy2]" subsys=daemon level=info msg="Direct routing device detected" direct-routing-device=cilium_foo subsys=daemon level=info msg="Detected devices" devices="[cilium_foo dummy0 dummy1 dummy2]" subsys=daemon level=info msg="Detected devices" devices="[cilium_foo dummy0 dummy1 dummy2 veth0]" subsys=daemon level=info msg="Detected devices" devices="[cilium_foo dummy0 dummy1 dummy2 dummy3 veth0]" subsys=daemon level=info msg="Detected devices" devices="[cilium_foo dummy0 dummy1 dummy2 dummy3 veth0]" subsys=daemon level=info msg="Detected devices" devices="[cilium_foo dummy0 dummy1 dummy2 veth0]" subsys=daemon level=info msg="Detected devices" devices="[bond0 cilium_foo dummy0 dummy1 veth0]" subsys=daemon START: devices_test.go:54: DevicesSuite.TearDownTest PASS: devices_test.go:54: DevicesSuite.TearDownTest 0.000s PASS: devices_test.go:66: DevicesSuite.TestDetect 0.094s START: devices_test.go:160: DevicesSuite.TestExpandDevices START: devices_test.go:54: DevicesSuite.TearDownTest PASS: devices_test.go:54: DevicesSuite.TearDownTest 0.000s PASS: devices_test.go:160: DevicesSuite.TestExpandDevices 0.002s START: devices_test.go:179: DevicesSuite.TestExpandDirectRoutingDevice START: devices_test.go:54: DevicesSuite.TearDownTest PASS: devices_test.go:54: DevicesSuite.TearDownTest 0.000s PASS: devices_test.go:179: DevicesSuite.TestExpandDirectRoutingDevice 0.001s START: nodeport_linux_test.go:41: NodePortSuite.TestCheckNodePortAndEphemeralPortRanges START: nodeport_linux_test.go:25: NodePortSuite.SetUpTest PASS: nodeport_linux_test.go:25: NodePortSuite.SetUpTest 0.000s START: nodeport_linux_test.go:34: NodePortSuite.TearDownTest PASS: nodeport_linux_test.go:34: NodePortSuite.TearDownTest 0.000s PASS: nodeport_linux_test.go:41: NodePortSuite.TestCheckNodePortAndEphemeralPortRanges 0.001s START: sysctl_linux_test.go:24: DaemonPrivilegedSuite.TestEnableIPForwarding PASS: sysctl_linux_test.go:24: DaemonPrivilegedSuite.TestEnableIPForwarding 0.000s OK: 5 passed --- PASS: Test (0.10s) PASS coverage: 2.0% of statements in ./... ok github.com/cilium/cilium/daemon/cmd 0.247s coverage: 2.0% of statements in ./... level=info msg="Detected mounted BPF filesystem at /sys/fs/bpf" subsys=bpf === RUN Test START: map_linux_test.go:249: BPFPrivilegedTestSuite.TestBasicManipulation PASS: map_linux_test.go:249: BPFPrivilegedTestSuite.TestBasicManipulation 0.001s START: map_linux_test.go:536: BPFPrivilegedTestSuite.TestCheckAndUpgrade MapTypeHash: no prealloc to prealloc upgrade level=warning msg="Flags mismatch for BPF map" file-path=/sys/fs/bpf/tc/globals/cilium_test_upgrade new=0 old=1 subsys=bpf level=warning msg="Removing map to allow for property upgrade (expect map data loss)" file-path=/sys/fs/bpf/tc/globals/cilium_test_upgrade subsys=bpf MapTypeLRUHash on 4.9 kernel: no prealloc to no prealloc upgrade PASS: map_linux_test.go:536: BPFPrivilegedTestSuite.TestCheckAndUpgrade 0.001s START: map_linux_test.go:697: BPFPrivilegedTestSuite.TestCreateUnpinned PASS: map_linux_test.go:697: BPFPrivilegedTestSuite.TestCreateUnpinned 0.000s START: map_linux_test.go:496: BPFPrivilegedTestSuite.TestDeleteAll PASS: map_linux_test.go:496: BPFPrivilegedTestSuite.TestDeleteAll 0.000s START: map_linux_test.go:336: BPFPrivilegedTestSuite.TestDump PASS: map_linux_test.go:336: BPFPrivilegedTestSuite.TestDump 0.000s START: map_linux_test.go:407: BPFPrivilegedTestSuite.TestDumpReliablyWithCallback PASS: map_linux_test.go:407: BPFPrivilegedTestSuite.TestDumpReliablyWithCallback 0.142s START: map_linux_test.go:100: BPFPrivilegedTestSuite.TestGetMapInfo PASS: map_linux_test.go:100: BPFPrivilegedTestSuite.TestGetMapInfo 0.000s START: map_linux_test.go:531: BPFPrivilegedTestSuite.TestGetModel PASS: map_linux_test.go:531: BPFPrivilegedTestSuite.TestGetModel 0.000s START: map_linux_test.go:114: BPFPrivilegedTestSuite.TestOpen PASS: map_linux_test.go:114: BPFPrivilegedTestSuite.TestOpen 0.000s START: map_linux_test.go:141: BPFPrivilegedTestSuite.TestOpenMap PASS: map_linux_test.go:141: BPFPrivilegedTestSuite.TestOpenMap 0.000s START: map_linux_test.go:159: BPFPrivilegedTestSuite.TestOpenOrCreate level=warning msg="Flags mismatch for BPF map" file-path=/sys/fs/bpf/tc/globals/cilium_test new=0 old=1 subsys=bpf level=warning msg="Removing map to allow for property upgrade (expect map data loss)" file-path=/sys/fs/bpf/tc/globals/cilium_test subsys=bpf PASS: map_linux_test.go:159: BPFPrivilegedTestSuite.TestOpenOrCreate 0.000s START: map_linux_test.go:199: BPFPrivilegedTestSuite.TestOpenParallel PASS: map_linux_test.go:199: BPFPrivilegedTestSuite.TestOpenParallel 0.000s START: map_linux_test.go:656: BPFPrivilegedTestSuite.TestUnpin PASS: map_linux_test.go:656: BPFPrivilegedTestSuite.TestUnpin 0.000s OK: 13 passed --- PASS: Test (0.15s) PASS coverage: 6.6% of statements in ./... ok github.com/cilium/cilium/pkg/bpf 0.304s coverage: 6.6% of statements in ./... === RUN Test START: link_test.go:24: LinkSuite.TestDeleteByName PASS: link_test.go:24: LinkSuite.TestDeleteByName 0.020s START: link_test.go:62: LinkSuite.TestRename PASS: link_test.go:62: LinkSuite.TestRename 0.052s OK: 2 passed --- PASS: Test (0.07s) PASS coverage: 2.4% of statements in ./... ok github.com/cilium/cilium/pkg/datapath/link 0.101s coverage: 2.4% of statements in ./... === RUN Test START: :1: linuxPrivilegedIPv6OnlyTestSuite.TestAgentRestartOptionChanges START: node_linux_test.go:109: linuxPrivilegedIPv6OnlyTestSuite.SetUpTest PASS: node_linux_test.go:109: linuxPrivilegedIPv6OnlyTestSuite.SetUpTest 0.135s START: node_linux_test.go:131: linuxPrivilegedIPv6OnlyTestSuite.TearDownTest PASS: node_linux_test.go:131: linuxPrivilegedIPv6OnlyTestSuite.TearDownTest 0.100s PASS: :1: linuxPrivilegedIPv6OnlyTestSuite.TestAgentRestartOptionChanges 0.002s START: node_linux_test.go:975: linuxPrivilegedIPv6OnlyTestSuite.TestArpPingHandling START: node_linux_test.go:109: linuxPrivilegedIPv6OnlyTestSuite.SetUpTest PASS: node_linux_test.go:109: linuxPrivilegedIPv6OnlyTestSuite.SetUpTest 0.003s coverage: 3.1% of statements in ./... panic: test timed out after 10m0s goroutine 15 [running]: testing.(*M).startAlarm.func1() /usr/local/go/src/testing/testing.go:1788 +0x8e created by time.goFunc /usr/local/go/src/time/sleep.go:180 +0x31 goroutine 1 [chan receive, 10 minutes]: testing.(*T).Run(0xc0004a64e0, {0x136ba4c, 0x46d993}, 0x140aef8) /usr/local/go/src/testing/testing.go:1307 +0x375 testing.runTests.func1(0xc0005b54d0) /usr/local/go/src/testing/testing.go:1598 +0x6e testing.tRunner(0xc0004a64e0, 0xc0003a1ce0) /usr/local/go/src/testing/testing.go:1259 +0x102 testing.runTests(0xc000454280, {0x1e85de0, 0x1, 0x1}, {0x48cfed, 0x1379bea, 0x1f8b320}) /usr/local/go/src/testing/testing.go:1596 +0x43f testing.(*M).Run(0xc000454280) /usr/local/go/src/testing/testing.go:1504 +0x51d main.main() _testmain.go:1983 +0x20a goroutine 21 [chan receive]: k8s.io/klog/v2.(*loggingT).flushDaemon(0xc0000b4de0) /home/vagrant/go/src/github.com/cilium/cilium/vendor/k8s.io/klog/v2/klog.go:1181 +0x6a created by k8s.io/klog/v2.init.0 /home/vagrant/go/src/github.com/cilium/cilium/vendor/k8s.io/klog/v2/klog.go:420 +0xfb goroutine 22 [select, 10 minutes]: io.(*pipe).Read(0xc0002b7320, {0xc000380000, 0x1000, 0x1}) /usr/local/go/src/io/pipe.go:57 +0xb7 io.(*PipeReader).Read(0x0, {0xc000380000, 0x0, 0x0}) /usr/local/go/src/io/pipe.go:134 +0x25 bufio.(*Scanner).Scan(0xc000386f28) /usr/local/go/src/bufio/scan.go:215 +0x865 github.com/sirupsen/logrus.(*Entry).writerScanner(0x0, 0xc0000b2b40, 0xc00036a900) /home/vagrant/go/src/github.com/cilium/cilium/vendor/github.com/sirupsen/logrus/writer.go:59 +0xa5 created by github.com/sirupsen/logrus.(*Entry).WriterLevel /home/vagrant/go/src/github.com/cilium/cilium/vendor/github.com/sirupsen/logrus/writer.go:51 +0x3d6 goroutine 23 [select, 10 minutes]: io.(*pipe).Read(0xc0002b7380, {0xc000408000, 0x1000, 0x1}) /usr/local/go/src/io/pipe.go:57 +0xb7 io.(*PipeReader).Read(0x0, {0xc000408000, 0x0, 0x0}) /usr/local/go/src/io/pipe.go:134 +0x25 bufio.(*Scanner).Scan(0xc00006df28) /usr/local/go/src/bufio/scan.go:215 +0x865 github.com/sirupsen/logrus.(*Entry).writerScanner(0x0, 0xc0000b2b50, 0xc00036a920) /home/vagrant/go/src/github.com/cilium/cilium/vendor/github.com/sirupsen/logrus/writer.go:59 +0xa5 created by github.com/sirupsen/logrus.(*Entry).WriterLevel /home/vagrant/go/src/github.com/cilium/cilium/vendor/github.com/sirupsen/logrus/writer.go:51 +0x3d6 goroutine 24 [select, 10 minutes]: io.(*pipe).Read(0xc0002b73e0, {0xc000480000, 0x1000, 0x1}) /usr/local/go/src/io/pipe.go:57 +0xb7 io.(*PipeReader).Read(0x0, {0xc000480000, 0x0, 0x0}) /usr/local/go/src/io/pipe.go:134 +0x25 bufio.(*Scanner).Scan(0xc000382f28) /usr/local/go/src/bufio/scan.go:215 +0x865 github.com/sirupsen/logrus.(*Entry).writerScanner(0x0, 0xc0000b2b60, 0xc00036a940) /home/vagrant/go/src/github.com/cilium/cilium/vendor/github.com/sirupsen/logrus/writer.go:59 +0xa5 created by github.com/sirupsen/logrus.(*Entry).WriterLevel /home/vagrant/go/src/github.com/cilium/cilium/vendor/github.com/sirupsen/logrus/writer.go:51 +0x3d6 goroutine 25 [select, 10 minutes]: io.(*pipe).Read(0xc0002b7440, {0xc000409000, 0x1000, 0x1}) /usr/local/go/src/io/pipe.go:57 +0xb7 io.(*PipeReader).Read(0x0, {0xc000409000, 0x0, 0x0}) /usr/local/go/src/io/pipe.go:134 +0x25 bufio.(*Scanner).Scan(0xc00006ef28) /usr/local/go/src/bufio/scan.go:215 +0x865 github.com/sirupsen/logrus.(*Entry).writerScanner(0x0, 0xc0000b2b70, 0xc00036a960) /home/vagrant/go/src/github.com/cilium/cilium/vendor/github.com/sirupsen/logrus/writer.go:59 +0xa5 created by github.com/sirupsen/logrus.(*Entry).WriterLevel /home/vagrant/go/src/github.com/cilium/cilium/vendor/github.com/sirupsen/logrus/writer.go:51 +0x3d6 goroutine 7 [chan receive, 9 minutes]: gopkg.in/check%2ev1.(*suiteRunner).runTest(...) /home/vagrant/go/src/github.com/cilium/cilium/vendor/gopkg.in/check.v1/check.go:813 gopkg.in/check%2ev1.(*suiteRunner).run(0xc000454300) /home/vagrant/go/src/github.com/cilium/cilium/vendor/gopkg.in/check.v1/check.go:618 +0x1be gopkg.in/check%2ev1.Run({0x1329260, 0xc0004a08c0}, 0xc000265f28) /home/vagrant/go/src/github.com/cilium/cilium/vendor/gopkg.in/check.v1/run.go:92 +0x2a gopkg.in/check%2ev1.RunAll(0xc000494ec8) /home/vagrant/go/src/github.com/cilium/cilium/vendor/gopkg.in/check.v1/run.go:84 +0x85 gopkg.in/check%2ev1.TestingT(0xc0004a6680) /home/vagrant/go/src/github.com/cilium/cilium/vendor/gopkg.in/check.v1/run.go:72 +0x278 github.com/cilium/cilium/pkg/datapath/linux.Test(0x0) /home/vagrant/go/src/github.com/cilium/cilium/pkg/datapath/linux/node_linux_test.go:41 +0x19 testing.tRunner(0xc0004a6680, 0x140aef8) /usr/local/go/src/testing/testing.go:1259 +0x102 created by testing.(*T).Run /usr/local/go/src/testing/testing.go:1306 +0x35a goroutine 8 [select, 9 minutes]: gopkg.in/check%2ev1.(*resultTracker)._loopRoutine(0xc000578120) /home/vagrant/go/src/github.com/cilium/cilium/vendor/gopkg.in/check.v1/check.go:464 +0x93 created by gopkg.in/check%2ev1.(*resultTracker).start /home/vagrant/go/src/github.com/cilium/cilium/vendor/gopkg.in/check.v1/check.go:444 +0x5b goroutine 32 [runnable, locked to thread]: github.com/vishvananda/netlink/nl.getNetlinkSocket(0x44fbf4) /home/vagrant/go/src/github.com/cilium/cilium/vendor/github.com/vishvananda/netlink/nl/nl_linux.go:572 +0x45 github.com/vishvananda/netlink/nl.(*NetlinkRequest).Execute(0xc0003adc40, 0x0, 0x1c) /home/vagrant/go/src/github.com/cilium/cilium/vendor/github.com/vishvananda/netlink/nl/nl_linux.go:479 +0x1ac github.com/vishvananda/netlink.(*Handle).NeighListExecute(0x1f83830, {0xa, 0x2c, 0x0, 0x0, 0x0}) /home/vagrant/go/src/github.com/cilium/cilium/vendor/github.com/vishvananda/netlink/neigh_linux.go:251 +0x205 github.com/vishvananda/netlink.(*Handle).NeighList(...) /home/vagrant/go/src/github.com/cilium/cilium/vendor/github.com/vishvananda/netlink/neigh_linux.go:224 github.com/vishvananda/netlink.NeighList(0xc000762df0, 0x0) /home/vagrant/go/src/github.com/cilium/cilium/vendor/github.com/vishvananda/netlink/neigh_linux.go:210 +0x4c github.com/cilium/cilium/pkg/datapath/linux.(*linuxPrivilegedIPv6OnlyTestSuite).TestArpPingHandling(0xc0004a08c0, 0xc00016c5a0) /home/vagrant/go/src/github.com/cilium/cilium/pkg/datapath/linux/node_linux_test.go:1112 +0x118f reflect.Value.call({0x1329260, 0xc0004a08c0, 0x43cec0}, {0x136b7fc, 0x4}, {0xc000068f00, 0x1, 0xe}) /usr/local/go/src/reflect/value.go:556 +0x845 reflect.Value.Call({0x1329260, 0xc0004a08c0, 0xc0000f2900}, {0xc000068f00, 0x1, 0x1}) /usr/local/go/src/reflect/value.go:339 +0xc5 gopkg.in/check%2ev1.(*suiteRunner).forkTest.func1(0xc00016c5a0) /home/vagrant/go/src/github.com/cilium/cilium/vendor/gopkg.in/check.v1/check.go:775 +0x64f gopkg.in/check%2ev1.(*suiteRunner).forkCall.func1() /home/vagrant/go/src/github.com/cilium/cilium/vendor/gopkg.in/check.v1/check.go:669 +0xa2 created by gopkg.in/check%2ev1.(*suiteRunner).forkCall /home/vagrant/go/src/github.com/cilium/cilium/vendor/gopkg.in/check.v1/check.go:666 +0x286 FAIL github.com/cilium/cilium/pkg/datapath/linux 600.078s FAIL make: Leaving directory '/home/vagrant/go/src/github.com/cilium/cilium' Stderr: make: *** [Makefile:138: tests-privileged] Error 1 /home/jenkins/workspace/Cilium-PR-Runtime-net-next/runtime-gopath/src/github.com/cilium/cilium/test/runtime/privileged_tests.go:46 ```

Standard Output

Click to show. ```stack-output ```

Standard Error

Click to show. ```stack-error 22:03:15 STEP: Running BeforeAll block for EntireTestsuite RuntimePrivilegedUnitTests FAIL: Failed to run privileged unit tests Expected command: sudo make -C /home/vagrant/go/src/github.com/cilium/cilium/ tests-privileged To succeed, but it failed: Exitcode: 2 Err: Process exited with status 2 Stdout: make: Entering directory '/home/vagrant/go/src/github.com/cilium/cilium' make init-coverage make[1]: Entering directory '/home/vagrant/go/src/github.com/cilium/cilium' echo "mode: count" > coverage-all-tmp.out echo "mode: count" > coverage.out make[1]: Leaving directory '/home/vagrant/go/src/github.com/cilium/cilium' for pkg in github.com/cilium/cilium/cilium/cmd github.com/cilium/cilium/daemon/cmd github.com/cilium/cilium/pkg/bpf github.com/cilium/cilium/pkg/datapath/link github.com/cilium/cilium/pkg/datapath/linux github.com/cilium/cilium/pkg/datapath/linux/config github.com/cilium/cilium/pkg/datapath/linux/ipsec github.com/cilium/cilium/pkg/datapath/linux/probes github.com/cilium/cilium/pkg/datapath/linux/route github.com/cilium/cilium/pkg/datapath/linux/routing github.com/cilium/cilium/pkg/datapath/loader github.com/cilium/cilium/pkg/egressgateway github.com/cilium/cilium/pkg/fqdn/dnsproxy github.com/cilium/cilium/pkg/maps/ctmap github.com/cilium/cilium/pkg/maps/egressmap github.com/cilium/cilium/pkg/maps/eppolicymap github.com/cilium/cilium/pkg/maps/lbmap github.com/cilium/cilium/pkg/maps/policymap github.com/cilium/cilium/pkg/modules github.com/cilium/cilium/pkg/mountinfo github.com/cilium/cilium/pkg/mtu github.com/cilium/cilium/pkg/redirectpolicy github.com/cilium/cilium/pkg/sysctl; do \ PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/vagrant/go/src/github.com/cilium/cilium/bpf go test -mod=vendor -tags=osusergo,privileged_tests -ldflags "-X github.com/cilium/cilium/pkg/kvstore.consulDummyAddress=https://consul:8443 -X github.com/cilium/cilium/pkg/kvstore.etcdDummyAddress=http://etcd:4002 -X github.com/cilium/cilium/pkg/datapath.DatapathSHA256=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" $pkg -test.v -timeout 600s -check.vv -covermode=count -coverprofile=coverage.out -coverpkg ./... \ || exit 1; \ tail -n +2 coverage.out >> coverage-all-tmp.out; \ done === RUN Test START: encrypt_status_test.go:31: EncryptStatusSuite.SetUpSuite PASS: encrypt_status_test.go:31: EncryptStatusSuite.SetUpSuite 0.000s START: encrypt_status_test.go:53: EncryptStatusSuite.TestCountUniqueIPsecKeys PASS: encrypt_status_test.go:53: EncryptStatusSuite.TestCountUniqueIPsecKeys 0.020s START: encrypt_status_test.go:84: EncryptStatusSuite.TestGetXfrmStats PASS: encrypt_status_test.go:84: EncryptStatusSuite.TestGetXfrmStats 0.001s OK: 2 passed --- PASS: Test (0.02s) PASS coverage: 2.1% of statements in ./... ok github.com/cilium/cilium/cilium/cmd 0.135s coverage: 2.1% of statements in ./... === RUN Test START: devices_test.go:38: DevicesSuite.SetUpSuite PASS: devices_test.go:38: DevicesSuite.SetUpSuite 0.000s START: devices_test.go:66: DevicesSuite.TestDetect level=info msg="Detected devices" devices="[]" subsys=daemon level=info msg="Detected devices" devices="[dummy0]" subsys=daemon level=info msg="Detected devices" devices="[dummy0]" subsys=daemon level=info msg="Direct routing device detected" direct-routing-device=dummy1 subsys=daemon level=info msg="Detected devices" devices="[dummy0 dummy1 dummy2]" subsys=daemon level=info msg="Direct routing device detected" direct-routing-device=cilium_foo subsys=daemon level=info msg="IPv6 multicast device detected" ipv6-mcast-device=cilium_foo subsys=daemon level=info msg="Detected devices" devices="[cilium_foo dummy0 dummy1 dummy2]" subsys=daemon level=info msg="Direct routing device detected" direct-routing-device=cilium_foo subsys=daemon level=info msg="Detected devices" devices="[cilium_foo dummy0 dummy1 dummy2]" subsys=daemon level=info msg="Detected devices" devices="[cilium_foo dummy0 dummy1 dummy2 veth0]" subsys=daemon level=info msg="Detected devices" devices="[cilium_foo dummy0 dummy1 dummy2 dummy3 veth0]" subsys=daemon level=info msg="Detected devices" devices="[cilium_foo dummy0 dummy1 dummy2 dummy3 veth0]" subsys=daemon level=info msg="Detected devices" devices="[cilium_foo dummy0 dummy1 dummy2 veth0]" subsys=daemon level=info msg="Detected devices" devices="[bond0 cilium_foo dummy0 dummy1 veth0]" subsys=daemon START: devices_test.go:54: DevicesSuite.TearDownTest PASS: devices_test.go:54: DevicesSuite.TearDownTest 0.000s PASS: devices_test.go:66: DevicesSuite.TestDetect 0.094s START: devices_test.go:160: DevicesSuite.TestExpandDevices START: devices_test.go:54: DevicesSuite.TearDownTest PASS: devices_test.go:54: DevicesSuite.TearDownTest 0.000s PASS: devices_test.go:160: DevicesSuite.TestExpandDevices 0.002s START: devices_test.go:179: DevicesSuite.TestExpandDirectRoutingDevice START: devices_test.go:54: DevicesSuite.TearDownTest PASS: devices_test.go:54: DevicesSuite.TearDownTest 0.000s PASS: devices_test.go:179: DevicesSuite.TestExpandDirectRoutingDevice 0.001s START: nodeport_linux_test.go:41: NodePortSuite.TestCheckNodePortAndEphemeralPortRanges START: nodeport_linux_test.go:25: NodePortSuite.SetUpTest PASS: nodeport_linux_test.go:25: NodePortSuite.SetUpTest 0.000s START: nodeport_linux_test.go:34: NodePortSuite.TearDownTest PASS: nodeport_linux_test.go:34: NodePortSuite.TearDownTest 0.000s PASS: nodeport_linux_test.go:41: NodePortSuite.TestCheckNodePortAndEphemeralPortRanges 0.001s START: sysctl_linux_test.go:24: DaemonPrivilegedSuite.TestEnableIPForwarding PASS: sysctl_linux_test.go:24: DaemonPrivilegedSuite.TestEnableIPForwarding 0.000s OK: 5 passed --- PASS: Test (0.10s) PASS coverage: 2.0% of statements in ./... ok github.com/cilium/cilium/daemon/cmd 0.247s coverage: 2.0% of statements in ./... level=info msg="Detected mounted BPF filesystem at /sys/fs/bpf" subsys=bpf === RUN Test START: map_linux_test.go:249: BPFPrivilegedTestSuite.TestBasicManipulation PASS: map_linux_test.go:249: BPFPrivilegedTestSuite.TestBasicManipulation 0.001s START: map_linux_test.go:536: BPFPrivilegedTestSuite.TestCheckAndUpgrade MapTypeHash: no prealloc to prealloc upgrade level=warning msg="Flags mismatch for BPF map" file-path=/sys/fs/bpf/tc/globals/cilium_test_upgrade new=0 old=1 subsys=bpf level=warning msg="Removing map to allow for property upgrade (expect map data loss)" file-path=/sys/fs/bpf/tc/globals/cilium_test_upgrade subsys=bpf MapTypeLRUHash on 4.9 kernel: no prealloc to no prealloc upgrade PASS: map_linux_test.go:536: BPFPrivilegedTestSuite.TestCheckAndUpgrade 0.001s START: map_linux_test.go:697: BPFPrivilegedTestSuite.TestCreateUnpinned PASS: map_linux_test.go:697: BPFPrivilegedTestSuite.TestCreateUnpinned 0.000s START: map_linux_test.go:496: BPFPrivilegedTestSuite.TestDeleteAll PASS: map_linux_test.go:496: BPFPrivilegedTestSuite.TestDeleteAll 0.000s START: map_linux_test.go:336: BPFPrivilegedTestSuite.TestDump PASS: map_linux_test.go:336: BPFPrivilegedTestSuite.TestDump 0.000s START: map_linux_test.go:407: BPFPrivilegedTestSuite.TestDumpReliablyWithCallback PASS: map_linux_test.go:407: BPFPrivilegedTestSuite.TestDumpReliablyWithCallback 0.142s START: map_linux_test.go:100: BPFPrivilegedTestSuite.TestGetMapInfo PASS: map_linux_test.go:100: BPFPrivilegedTestSuite.TestGetMapInfo 0.000s START: map_linux_test.go:531: BPFPrivilegedTestSuite.TestGetModel PASS: map_linux_test.go:531: BPFPrivilegedTestSuite.TestGetModel 0.000s START: map_linux_test.go:114: BPFPrivilegedTestSuite.TestOpen PASS: map_linux_test.go:114: BPFPrivilegedTestSuite.TestOpen 0.000s START: map_linux_test.go:141: BPFPrivilegedTestSuite.TestOpenMap PASS: map_linux_test.go:141: BPFPrivilegedTestSuite.TestOpenMap 0.000s START: map_linux_test.go:159: BPFPrivilegedTestSuite.TestOpenOrCreate level=warning msg="Flags mismatch for BPF map" file-path=/sys/fs/bpf/tc/globals/cilium_test new=0 old=1 subsys=bpf level=warning msg="Removing map to allow for property upgrade (expect map data loss)" file-path=/sys/fs/bpf/tc/globals/cilium_test subsys=bpf PASS: map_linux_test.go:159: BPFPrivilegedTestSuite.TestOpenOrCreate 0.000s START: map_linux_test.go:199: BPFPrivilegedTestSuite.TestOpenParallel PASS: map_linux_test.go:199: BPFPrivilegedTestSuite.TestOpenParallel 0.000s START: map_linux_test.go:656: BPFPrivilegedTestSuite.TestUnpin PASS: map_linux_test.go:656: BPFPrivilegedTestSuite.TestUnpin 0.000s OK: 13 passed --- PASS: Test (0.15s) PASS coverage: 6.6% of statements in ./... ok github.com/cilium/cilium/pkg/bpf 0.304s coverage: 6.6% of statements in ./... === RUN Test START: link_test.go:24: LinkSuite.TestDeleteByName PASS: link_test.go:24: LinkSuite.TestDeleteByName 0.020s START: link_test.go:62: LinkSuite.TestRename PASS: link_test.go:62: LinkSuite.TestRename 0.052s OK: 2 passed --- PASS: Test (0.07s) PASS coverage: 2.4% of statements in ./... ok github.com/cilium/cilium/pkg/datapath/link 0.101s coverage: 2.4% of statements in ./... === RUN Test START: :1: linuxPrivilegedIPv6OnlyTestSuite.TestAgentRestartOptionChanges START: node_linux_test.go:109: linuxPrivilegedIPv6OnlyTestSuite.SetUpTest PASS: node_linux_test.go:109: linuxPrivilegedIPv6OnlyTestSuite.SetUpTest 0.135s START: node_linux_test.go:131: linuxPrivilegedIPv6OnlyTestSuite.TearDownTest PASS: node_linux_test.go:131: linuxPrivilegedIPv6OnlyTestSuite.TearDownTest 0.100s PASS: :1: linuxPrivilegedIPv6OnlyTestSuite.TestAgentRestartOptionChanges 0.002s START: node_linux_test.go:975: linuxPrivilegedIPv6OnlyTestSuite.TestArpPingHandling START: node_linux_test.go:109: linuxPrivilegedIPv6OnlyTestSuite.SetUpTest PASS: node_linux_test.go:109: linuxPrivilegedIPv6OnlyTestSuite.SetUpTest 0.003s coverage: 3.1% of statements in ./... panic: test timed out after 10m0s goroutine 15 [running]: testing.(*M).startAlarm.func1() /usr/local/go/src/testing/testing.go:1788 +0x8e created by time.goFunc /usr/local/go/src/time/sleep.go:180 +0x31 goroutine 1 [chan receive, 10 minutes]: testing.(*T).Run(0xc0004a64e0, {0x136ba4c, 0x46d993}, 0x140aef8) /usr/local/go/src/testing/testing.go:1307 +0x375 testing.runTests.func1(0xc0005b54d0) /usr/local/go/src/testing/testing.go:1598 +0x6e testing.tRunner(0xc0004a64e0, 0xc0003a1ce0) /usr/local/go/src/testing/testing.go:1259 +0x102 testing.runTests(0xc000454280, {0x1e85de0, 0x1, 0x1}, {0x48cfed, 0x1379bea, 0x1f8b320}) /usr/local/go/src/testing/testing.go:1596 +0x43f testing.(*M).Run(0xc000454280) /usr/local/go/src/testing/testing.go:1504 +0x51d main.main() _testmain.go:1983 +0x20a goroutine 21 [chan receive]: k8s.io/klog/v2.(*loggingT).flushDaemon(0xc0000b4de0) /home/vagrant/go/src/github.com/cilium/cilium/vendor/k8s.io/klog/v2/klog.go:1181 +0x6a created by k8s.io/klog/v2.init.0 /home/vagrant/go/src/github.com/cilium/cilium/vendor/k8s.io/klog/v2/klog.go:420 +0xfb goroutine 22 [select, 10 minutes]: io.(*pipe).Read(0xc0002b7320, {0xc000380000, 0x1000, 0x1}) /usr/local/go/src/io/pipe.go:57 +0xb7 io.(*PipeReader).Read(0x0, {0xc000380000, 0x0, 0x0}) /usr/local/go/src/io/pipe.go:134 +0x25 bufio.(*Scanner).Scan(0xc000386f28) /usr/local/go/src/bufio/scan.go:215 +0x865 github.com/sirupsen/logrus.(*Entry).writerScanner(0x0, 0xc0000b2b40, 0xc00036a900) /home/vagrant/go/src/github.com/cilium/cilium/vendor/github.com/sirupsen/logrus/writer.go:59 +0xa5 created by github.com/sirupsen/logrus.(*Entry).WriterLevel /home/vagrant/go/src/github.com/cilium/cilium/vendor/github.com/sirupsen/logrus/writer.go:51 +0x3d6 goroutine 23 [select, 10 minutes]: io.(*pipe).Read(0xc0002b7380, {0xc000408000, 0x1000, 0x1}) /usr/local/go/src/io/pipe.go:57 +0xb7 io.(*PipeReader).Read(0x0, {0xc000408000, 0x0, 0x0}) /usr/local/go/src/io/pipe.go:134 +0x25 bufio.(*Scanner).Scan(0xc00006df28) /usr/local/go/src/bufio/scan.go:215 +0x865 github.com/sirupsen/logrus.(*Entry).writerScanner(0x0, 0xc0000b2b50, 0xc00036a920) /home/vagrant/go/src/github.com/cilium/cilium/vendor/github.com/sirupsen/logrus/writer.go:59 +0xa5 created by github.com/sirupsen/logrus.(*Entry).WriterLevel /home/vagrant/go/src/github.com/cilium/cilium/vendor/github.com/sirupsen/logrus/writer.go:51 +0x3d6 goroutine 24 [select, 10 minutes]: io.(*pipe).Read(0xc0002b73e0, {0xc000480000, 0x1000, 0x1}) /usr/local/go/src/io/pipe.go:57 +0xb7 io.(*PipeReader).Read(0x0, {0xc000480000, 0x0, 0x0}) /usr/local/go/src/io/pipe.go:134 +0x25 bufio.(*Scanner).Scan(0xc000382f28) /usr/local/go/src/bufio/scan.go:215 +0x865 github.com/sirupsen/logrus.(*Entry).writerScanner(0x0, 0xc0000b2b60, 0xc00036a940) /home/vagrant/go/src/github.com/cilium/cilium/vendor/github.com/sirupsen/logrus/writer.go:59 +0xa5 created by github.com/sirupsen/logrus.(*Entry).WriterLevel /home/vagrant/go/src/github.com/cilium/cilium/vendor/github.com/sirupsen/logrus/writer.go:51 +0x3d6 goroutine 25 [select, 10 minutes]: io.(*pipe).Read(0xc0002b7440, {0xc000409000, 0x1000, 0x1}) /usr/local/go/src/io/pipe.go:57 +0xb7 io.(*PipeReader).Read(0x0, {0xc000409000, 0x0, 0x0}) /usr/local/go/src/io/pipe.go:134 +0x25 bufio.(*Scanner).Scan(0xc00006ef28) /usr/local/go/src/bufio/scan.go:215 +0x865 github.com/sirupsen/logrus.(*Entry).writerScanner(0x0, 0xc0000b2b70, 0xc00036a960) /home/vagrant/go/src/github.com/cilium/cilium/vendor/github.com/sirupsen/logrus/writer.go:59 +0xa5 created by github.com/sirupsen/logrus.(*Entry).WriterLevel /home/vagrant/go/src/github.com/cilium/cilium/vendor/github.com/sirupsen/logrus/writer.go:51 +0x3d6 goroutine 7 [chan receive, 9 minutes]: gopkg.in/check%2ev1.(*suiteRunner).runTest(...) /home/vagrant/go/src/github.com/cilium/cilium/vendor/gopkg.in/check.v1/check.go:813 gopkg.in/check%2ev1.(*suiteRunner).run(0xc000454300) /home/vagrant/go/src/github.com/cilium/cilium/vendor/gopkg.in/check.v1/check.go:618 +0x1be gopkg.in/check%2ev1.Run({0x1329260, 0xc0004a08c0}, 0xc000265f28) /home/vagrant/go/src/github.com/cilium/cilium/vendor/gopkg.in/check.v1/run.go:92 +0x2a gopkg.in/check%2ev1.RunAll(0xc000494ec8) /home/vagrant/go/src/github.com/cilium/cilium/vendor/gopkg.in/check.v1/run.go:84 +0x85 gopkg.in/check%2ev1.TestingT(0xc0004a6680) /home/vagrant/go/src/github.com/cilium/cilium/vendor/gopkg.in/check.v1/run.go:72 +0x278 github.com/cilium/cilium/pkg/datapath/linux.Test(0x0) /home/vagrant/go/src/github.com/cilium/cilium/pkg/datapath/linux/node_linux_test.go:41 +0x19 testing.tRunner(0xc0004a6680, 0x140aef8) /usr/local/go/src/testing/testing.go:1259 +0x102 created by testing.(*T).Run /usr/local/go/src/testing/testing.go:1306 +0x35a goroutine 8 [select, 9 minutes]: gopkg.in/check%2ev1.(*resultTracker)._loopRoutine(0xc000578120) /home/vagrant/go/src/github.com/cilium/cilium/vendor/gopkg.in/check.v1/check.go:464 +0x93 created by gopkg.in/check%2ev1.(*resultTracker).start /home/vagrant/go/src/github.com/cilium/cilium/vendor/gopkg.in/check.v1/check.go:444 +0x5b goroutine 32 [runnable, locked to thread]: github.com/vishvananda/netlink/nl.getNetlinkSocket(0x44fbf4) /home/vagrant/go/src/github.com/cilium/cilium/vendor/github.com/vishvananda/netlink/nl/nl_linux.go:572 +0x45 github.com/vishvananda/netlink/nl.(*NetlinkRequest).Execute(0xc0003adc40, 0x0, 0x1c) /home/vagrant/go/src/github.com/cilium/cilium/vendor/github.com/vishvananda/netlink/nl/nl_linux.go:479 +0x1ac github.com/vishvananda/netlink.(*Handle).NeighListExecute(0x1f83830, {0xa, 0x2c, 0x0, 0x0, 0x0}) /home/vagrant/go/src/github.com/cilium/cilium/vendor/github.com/vishvananda/netlink/neigh_linux.go:251 +0x205 github.com/vishvananda/netlink.(*Handle).NeighList(...) /home/vagrant/go/src/github.com/cilium/cilium/vendor/github.com/vishvananda/netlink/neigh_linux.go:224 github.com/vishvananda/netlink.NeighList(0xc000762df0, 0x0) /home/vagrant/go/src/github.com/cilium/cilium/vendor/github.com/vishvananda/netlink/neigh_linux.go:210 +0x4c github.com/cilium/cilium/pkg/datapath/linux.(*linuxPrivilegedIPv6OnlyTestSuite).TestArpPingHandling(0xc0004a08c0, 0xc00016c5a0) /home/vagrant/go/src/github.com/cilium/cilium/pkg/datapath/linux/node_linux_test.go:1112 +0x118f reflect.Value.call({0x1329260, 0xc0004a08c0, 0x43cec0}, {0x136b7fc, 0x4}, {0xc000068f00, 0x1, 0xe}) /usr/local/go/src/reflect/value.go:556 +0x845 reflect.Value.Call({0x1329260, 0xc0004a08c0, 0xc0000f2900}, {0xc000068f00, 0x1, 0x1}) /usr/local/go/src/reflect/value.go:339 +0xc5 gopkg.in/check%2ev1.(*suiteRunner).forkTest.func1(0xc00016c5a0) /home/vagrant/go/src/github.com/cilium/cilium/vendor/gopkg.in/check.v1/check.go:775 +0x64f gopkg.in/check%2ev1.(*suiteRunner).forkCall.func1() /home/vagrant/go/src/github.com/cilium/cilium/vendor/gopkg.in/check.v1/check.go:669 +0xa2 created by gopkg.in/check%2ev1.(*suiteRunner).forkCall /home/vagrant/go/src/github.com/cilium/cilium/vendor/gopkg.in/check.v1/check.go:666 +0x286 FAIL github.com/cilium/cilium/pkg/datapath/linux 600.078s FAIL make: Leaving directory '/home/vagrant/go/src/github.com/cilium/cilium' Stderr: make: *** [Makefile:138: tests-privileged] Error 1 === Test Finished at 2022-03-01T22:15:20Z==== 22:15:20 STEP: Running AfterEach for block EntireTestsuite [[ATTACHMENT|0006e2ef_RuntimePrivilegedUnitTests_Run_Tests.zip]] 22:15:20 STEP: Running AfterAll block for EntireTestsuite RuntimePrivilegedUnitTests 22:15:20 STEP: Restarting Cilium ```

ZIP Links:

Click to show. https://jenkins.cilium.io/job/Cilium-PR-Runtime-net-next//1440/artifact/0006e2ef_RuntimePrivilegedUnitTests_Run_Tests.zip https://jenkins.cilium.io/job/Cilium-PR-Runtime-net-next//1440/artifact/test_results_Cilium-PR-Runtime-net-next_1440_BDD-Test-PR.zip

Jenkins URL: https://jenkins.cilium.io/job/Cilium-PR-Runtime-net-next/1440/

If this is a duplicate of an existing flake, comment 'Duplicate of #\' and close this issue.

pchaigno commented 2 years ago

Previously opened at https://github.com/cilium/cilium/issues/19072 but MLH started associating it with arbitrary RuntimePrivilegedUnitTests failures.

tklauser commented 2 years ago

Seen on #19397: https://jenkins.cilium.io/job/Cilium-PR-Runtime-net-next/1880/

joestringer commented 2 years ago

Is this distinct from #18958?

joestringer commented 2 years ago

EDIT: Deleted, in the one I reported there was no panic. Moving to #18958.

github-actions[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

nbusseneau commented 1 year ago

I think this is still happening and was hit in https://jenkins.cilium.io/job/cilium-master-runtime-kernel-net-next/5606/testReport/junit/(root)/Suite-runtime/RuntimeDatapathPrivilegedUnitTests_Run_Tests/ and https://jenkins.cilium.io/job/cilium-master-runtime-kernel-net-next/5603/testReport/junit/(root)/Suite-runtime/RuntimeDatapathPrivilegedUnitTests_Run_Tests/.

Artifacts:

github-actions[bot] commented 1 year ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

github-actions[bot] commented 1 year ago

This issue has not seen any activity since it was marked stale. Closing.