search

jetstack / navigator

Managed Database-as-a-Service (DBaaS) on Kubernetes
Apache License 2.0
271 stars 31 forks source link

Failed to delete test namespace in ElasticSearch E2E tests #157

Closed wallrj closed 6 years ago

wallrj commented 6 years ago

A few errors in a recent 1.7 e2e test run:

See https://jetstack-build-infra.appspot.com/build/jetstack-logs/pr-logs/pull/jetstack_navigator/142/navigator-e2e-v1-7/187/

W1128 14:20:15.049] + echo 'TEST FAILURE: Elasticsearch pilot did not update the document count'
...
I1128 14:25:16.218] TEST FAILURE: Failed to delete test namespace
...
I1128 14:28:01.266] E1128 14:16:59.244123      14 pilot.go:60] Error creating elasticsearch api client: health check timeout: no Elasticsearch node available
I1128 14:28:01.266] E1128 14:17:04.515060      14 pilot.go:60] Error creating elasticsearch api client: health check timeout: no Elasticsearch node available
I1128 14:28:01.267] [2017-11-28T14:17:06,743][INFO ][o.e.n.Node               ] [es-test-mixed-0] initializing ...
I1128 14:28:01.267] [2017-11-28T14:17:08,663][INFO ][o.e.e.NodeEnvironment    ] [es-test-mixed-0] using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/sda1)]], net usable_space [12.6gb], net total_space [16.1gb], spins? [possibly], types [ext4]
I1128 14:28:01.267] [2017-11-28T14:17:08,668][INFO ][o.e.e.NodeEnvironment    ] [es-test-mixed-0] heap size [1.9gb], compressed ordinary object pointers [true]
I1128 14:28:01.267] [2017-11-28T14:17:08,679][INFO ][o.e.n.Node               ] [es-test-mixed-0] node name [es-test-mixed-0], node ID [1D2QZ_KgQHutv6dQzkMsUg]
I1128 14:28:01.267] [2017-11-28T14:17:08,682][INFO ][o.e.n.Node               ] [es-test-mixed-0] version[5.6.2], pid[46], build[57e20f3/2017-09-23T13:16:45.703Z], OS[Linux/4.9.13/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_141/25.141-b16]
I1128 14:28:01.267] [2017-11-28T14:17:08,682][INFO ][o.e.n.Node               ] [es-test-mixed-0] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/usr/share/elasticsearch]
I1128 14:28:01.267] E1128 14:17:10.001616      14 pilot.go:60] Error creating elasticsearch api client: health check timeout: no Elasticsearch node available
I1128 14:28:01.267] I1128 14:17:15.180358      14 reflector.go:286] github.com/jetstack/navigator/pkg/client/informers/externalversions/factory.go:73: forcing resync
I1128 14:28:01.268] I1128 14:17:15.726798      14 logs.go:41] http: panic serving 172.17.0.1:48422: runtime error: invalid memory address or nil pointer dereference
I1128 14:28:01.268] goroutine 168 [running]:
I1128 14:28:01.268] net/http.(*conn).serve.func1(0xc420156d20)
I1128 14:28:01.268]     /usr/local/go/src/net/http/server.go:1697 +0xd0
I1128 14:28:01.268] panic(0x111d920, 0x1afee60)
I1128 14:28:01.268]     /usr/local/go/src/runtime/panic.go:491 +0x283
I1128 14:28:01.268] github.com/jetstack/navigator/vendor/gopkg.in/olivere/elastic%2ev5.(*Client).PerformRequest(0x0, 0x1ab60c0, 0xc4202a4180, 0x1281af9, 0x3, 0x1289e95, 0x10, 0xc42021cfc0, 0x0, 0x0, ...)
I1128 14:28:01.268]     /go/src/github.com/jetstack/navigator/vendor/gopkg.in/olivere/elastic.v5/client.go:1179 +0xe3
I1128 14:28:01.268] github.com/jetstack/navigator/vendor/gopkg.in/olivere/elastic%2ev5.(*ClusterHealthService).Do(0xc42053eb98, 0x1ab60c0, 0xc4202a4180, 0x1b30b78, 0x0, 0x0)
I1128 14:28:01.268]     /go/src/github.com/jetstack/navigator/vendor/gopkg.in/olivere/elastic.v5/cluster_health.go:182 +0xe1
I1128 14:28:01.268] github.com/jetstack/navigator/pkg/pilot/elasticsearch/v5.(*Pilot).localNodeHealth(0xc4202d6780, 0x0, 0x0)
I1128 14:28:01.268]     /go/src/github.com/jetstack/navigator/pkg/pilot/elasticsearch/v5/probes.go:30 +0x176
I1128 14:28:01.269] github.com/jetstack/navigator/pkg/pilot/elasticsearch/v5.(*Pilot).(github.com/jetstack/navigator/pkg/pilot/elasticsearch/v5.localNodeHealth)-fm(0x492d72, 0xc420302280)
I1128 14:28:01.269]     /go/src/github.com/jetstack/navigator/pkg/pilot/elasticsearch/v5/probes.go:14 +0x2a
I1128 14:28:01.269] github.com/jetstack/navigator/pkg/pilot/genericpilot/probe.CombineChecks.func1(0x0, 0xc42053ecf8)
I1128 14:28:01.269]     /go/src/github.com/jetstack/navigator/pkg/pilot/genericpilot/probe/check.go:10 +0x60
I1128 14:28:01.269] github.com/jetstack/navigator/pkg/pilot/genericpilot/probe.(*Listener).Listen.func1(0x1ab5380, 0xc420438000, 0xc4204d3300)
I1128 14:28:01.269]     /go/src/github.com/jetstack/navigator/pkg/pilot/genericpilot/probe/listen.go:18 +0x2e
I1128 14:28:01.269] net/http.HandlerFunc.ServeHTTP(0xc42045e0a0, 0x1ab5380, 0xc420438000, 0xc4204d3300)
I1128 14:28:01.269]     /usr/local/go/src/net/http/server.go:1918 +0x44
I1128 14:28:01.269] net/http.serverHandler.ServeHTTP(0xc420213c70, 0x1ab5380, 0xc420438000, 0xc4204d3300)
I1128 14:28:01.269]     /usr/local/go/src/net/http/server.go:2619 +0xb4
I1128 14:28:01.269] net/http.(*conn).serve(0xc420156d20, 0x1ab6040, 0xc4203255c0)
I1128 14:28:01.269]     /usr/local/go/src/net/http/server.go:1801 +0x71d
I1128 14:28:01.270] created by net/http.(*Server).Serve
I1128 14:28:01.270]     /usr/local/go/src/net/http/server.go:2720 +0x288
I1128 14:28:01.270] E1128 14:17:16.064150      14 pilot.go:60] Error creating elasticsearch api client: health check timeout: no Elasticsearch node available
I1128 14:28:01.270] E1128 14:17:21.417831      14 pilot.go:60] Error creating elasticsearch api client: health check timeout: no Elasticsearch node available
I1128 14:28:01.270] I1128 14:17:24.065445      14 logs.go:41] http: panic serving 172.17.0.1:48462: runtime error: invalid memory address or nil pointer dereference
...
I1128 14:28:01.423] E1128 14:28:00.771603      14 reflector.go:205] github.com/jetstack/navigator/pkg/client/informers/externalversions/factory.go:73: Failed to list *v1alpha1.Pilot: the server has asked for the client to provide credentials (get pilots.navigator.jetstack.io)
...
I1128 14:28:01.423] ==== END logs for container elasticsearch of pod test-elasticsearchcluster-1511878502-24723/es-test-mixed-0 ====

Found while testing https://github.com/jetstack/navigator/pull/142 /kind bug

wallrj commented 6 years ago

Another such test failure: https://jetstack-build-infra.appspot.com/build/jetstack-logs/pr-logs/pull/jetstack_navigator/155/navigator-e2e-v1-8/223/