search

solo-io / gloo

The Feature-rich, Kubernetes-native, Next-Generation API Gateway Built on Envoy
https://docs.solo.io/
Apache License 2.0
4.09k stars 442 forks source link

CI flakes in helm regression test #2859

Closed rickducott closed 2 years ago

rickducott commented 4 years ago

Encountered on https://github.com/solo-io/gloo/pull/2825

[1587577633] Helm Suite - 2/2 specs "Wed, 22 Apr 2020 17:52:46 UTC: /go/pkg/mod/github.com/solo-io/go-utils@v0.14.1/testutils/helper/install.go:219"    found Helm index file at: "/workspace/gloo/_test/index.yaml"
"Wed, 22 Apr 2020 17:52:46 UTC: /go/pkg/mod/github.com/solo-io/go-utils@v0.14.1/testutils/helper/install.go:228"    version of ["gloo"] Helm chart is: "1.4.0-beta4-7-gaa8e15d9"
Checking deployments... Deployment discovery in namespace helm-test-7966-1 is not available! Message: Deployment does not have minimum availability.
Checking deployments... Deployment gateway in namespace helm-test-7966-1 is not available! Message: Deployment does not have minimum availability.
Checking deployments... Deployment gateway in namespace helm-test-7966-1 is not available! Message: Deployment does not have minimum availability.
Checking deployments... Deployment gateway in namespace helm-test-7966-1 is not available! Message: Deployment does not have minimum availability.
Checking deployments... Deployment gateway in namespace helm-test-7966-1 is not available! Message: Deployment does not have minimum availability.
Checking deployments... Deployment gateway in namespace helm-test-7966-1 is not available! Message: Deployment does not have minimum availability.
Checking deployments... Deployment gateway in namespace helm-test-7966-1 is not available! Message: Deployment does not have minimum availability.
Checking deployments... Deployment gateway in namespace helm-test-7966-1 is not available! Message: Deployment does not have minimum availability.
Fail handler msg Timed out after 40.001s.
Expected
    <*eris.rootError | 0xc000b61c00>: {
        msg: "glooctl check detected a problem with the installation",
        stack: [0x25d6499, 0x51eeeb, 0x51e5b4, 0x63866a, 0x639054, 0x638152, 0x25d5a7f, 0x25e6678, 0x25ab6e0, 0x25ab198, 0x25aba33, 0x25bc041, 0x25bb8d0, 0x25bf1f0, 0x25c3499, 0x25c2f7e, 0x25e5a88, 0x5960ca, 0x48a3b1],
    }
to be nil
** Begin Kubernetes Dump ** 
PODS FROM knative-serving:
PODS FROM helm-test-7966-1:
discovery-66bdcd9d9b-4zfnr
gateway-b8bcfc777-dm2jw
gateway-proxy-6574cd99bb-q8fm6
gloo-894fc69b5-rhxrj
STATUS FOR helm-test-7966-1.discovery-66bdcd9d9b-4zfnr: map[running:map[startedAt:2020-04-22T17:52:57Z]]
STATUS FOR helm-test-7966-1.gateway-b8bcfc777-dm2jw: map[running:map[startedAt:2020-04-22T17:52:57Z]]
STATUS FOR helm-test-7966-1.gateway-proxy-6574cd99bb-q8fm6: map[running:map[startedAt:2020-04-22T17:52:57Z]]
STATUS FOR helm-test-7966-1.gloo-894fc69b5-rhxrj: map[running:map[startedAt:2020-04-22T17:52:57Z]]
LOGS FROM helm-test-7966-1.discovery-66bdcd9d9b-4zfnr: {"level":"info","ts":1587577977.4443243,"logger":"uds.v1.event_loop","caller":"v1/setup_event_loop.sk.go:57","msg":"event loop started","version":"1.3.0"} {"level":"info","ts":1587577977.448889,"logger":"fds.v1.event_loop","caller":"v1/setup_event_loop.sk.go:57","msg":"event loop started","version":"1.3.0"} {"level":"info","ts":1587577977.6552222,"logger":"uds.v1.event_loop.uds.kube-uds","caller":"kubernetes/uds.go:31","msg":"started","version":"1.3.0","watchns":["helm-test-7966-1"],"writens":"helm-test-7966-1"} {"level":"info","ts":1587577977.655315,"logger":"uds.v1.event_loop.uds.v1.event_loop","caller":"v1/discovery_event_loop.sk.go:57","msg":"event loop started","version":"1.3.0"} {"level":"info","ts":1587577977.6583722,"logger":"fds.v1.event_loop.fds.v1.event_loop","caller":"v1/discovery_event_loop.sk.go:57","msg":"event loop started","version":"1.3.0"} {"level":"info","ts":1587577977.756171,"logger":"uds.v1.event_loop.uds.v1.event_loop.syncer","caller":"syncer/discovery_syncer.go:34","msg":"begin sync 6708220120071169714 (0 upstreams)","version":"1.3.0"} {"level":"info","ts":1587577977.7590952,"logger":"fds.v1.event_loop.fds.v1.event_loop.syncer","caller":"syncer/discovery_syncer.go:35","msg":"begin sync 6708220120071169714 (0 upstreams)","version":"1.3.0"} {"level":"info","ts":1587577977.7591367,"logger":"fds.v1.event_loop.fds.v1.event_loop.syncer","caller":"syncer/discovery_syncer.go:46","msg":"end sync 6708220120071169714","version":"1.3.0"} {"level":"info","ts":1587577979.759597,"logger":"fds.v1.event_loop.fds.v1.event_loop.syncer","caller":"syncer/discovery_syncer.go:35","msg":"begin sync 16068153935155318811 (4 upstreams)","version":"1.3.0"} {"level":"info","ts":1587577979.759775,"logger":"fds.v1.event_loop.fds.v1.event_loop.syncer","caller":"syncer/discovery_syncer.go:46","msg":"end sync 16068153935155318811","version":"1.3.0"} {"level":"info","ts":1587577980.759346,"logger":"fds.v1.event_loop.fds.v1.event_loop.syncer","caller":"syncer/discovery_syncer.go:35","msg":"begin sync 10790202228258755360 (6 upstreams)","version":"1.3.0"} {"level":"info","ts":1587577980.7594101,"logger":"fds.v1.event_loop.fds.v1.event_loop.syncer","caller":"syncer/discovery_syncer.go:46","msg":"end sync 10790202228258755360","version":"1.3.0"} {"level":"info","ts":1587577981.4564922,"logger":"uds.v1.event_loop.uds","caller":"discovery/discovery.go:143","msg":"reconciled upstreams","version":"1.3.0","discovered_by":"kubernetesplugin","upstreams":9} {"level":"info","ts":1587577981.4586961,"logger":"uds.v1.event_loop.uds.v1.event_loop.syncer","caller":"discovery/discovery.go:143","msg":"reconciled upstreams","version":"1.3.0","discovered_by":"kubernetesplugin","upstreams":9} {"level":"info","ts":1587577981.4587576,"logger":"uds.v1.event_loop.uds.v1.event_loop.syncer","caller":"syncer/discovery_syncer.go:44","msg":"end sync 6708220120071169714","version":"1.3.0"} {"level":"info","ts":1587577981.4588792,"logger":"uds.v1.event_loop.uds.v1.event_loop.syncer","caller":"syncer/discovery_syncer.go:34","msg":"begin sync 6708220120071169714 (0 upstreams)","version":"1.3.0"} {"level":"info","ts":1587577981.4608142,"logger":"uds.v1.event_loop.uds","caller":"discovery/discovery.go:143","msg":"reconciled upstreams","version":"1.3.0","discovered_by":"kubernetesplugin","upstreams":9} {"level":"info","ts":1587577981.4628658,"logger":"uds.v1.event_loop.uds.v1.event_loop.syncer","caller":"discovery/discovery.go:143","msg":"reconciled upstreams","version":"1.3.0","discovered_by":"kubernetesplugin","upstreams":9} {"level":"info","ts":1587577981.462982,"logger":"uds.v1.event_loop.uds.v1.event_loop.syncer","caller":"syncer/discovery_syncer.go:44","msg":"end sync 6708220120071169714","version":"1.3.0"} {"level":"info","ts":1587577981.4644644,"logger":"uds.v1.event_loop.uds","caller":"discovery/discovery.go:143","msg":"reconciled upstreams","version":"1.3.0","discovered_by":"kubernetesplugin","upstreams":9} {"level":"info","ts":1587577981.4661953,"logger":"uds.v1.event_loop.uds","caller":"discovery/discovery.go:143","msg":"reconciled upstreams","version":"1.3.0","discovered_by":"kubernetesplugin","upstreams":9} {"level":"info","ts":1587577981.756828,"logger":"uds.v1.event_loop.uds.v1.event_loop.syncer","caller":"syncer/discovery_syncer.go:34","msg":"begin sync 6734511270705228043 (8 upstreams)","version":"1.3.0"} {"level":"info","ts":1587577981.763783,"logger":"uds.v1.event_loop.uds.v1.event_loop.syncer","caller":"discovery/discovery.go:143","msg":"reconciled upstreams","version":"1.3.0","discovered_by":"kubernetesplugin","upstreams":9} {"level":"info","ts":1587577981.763863,"logger":"uds.v1.event_loop.uds.v1.event_loop.syncer","caller":"syncer/discovery_syncer.go:44","msg":"end sync 6734511270705228043","version":"1.3.0"} {"level":"info","ts":1587577981.7655268,"logger":"fds.v1.event_loop.fds.v1.event_loop.syncer","caller":"syncer/discovery_syncer.go:35","msg":"begin sync 6734511270705228043 (8 upstreams)","version":"1.3.0"} {"level":"info","ts":1587577981.765682,"logger":"fds.v1.event_loop.fds.v1.event_loop.syncer","caller":"syncer/discovery_syncer.go:46","msg":"end sync 6734511270705228043","version":"1.3.0"} {"level":"info","ts":1587577982.6941373,"logger":"uds.v1.event_loop.uds","caller":"discovery/discovery.go:143","msg":"reconciled upstreams","version":"1.3.0","discovered_by":"kubernetesplugin","upstreams":9} {"level":"info","ts":1587577982.7567315,"logger":"uds.v1.event_loop.uds.v1.event_loop.syncer","caller":"syncer/discovery_syncer.go:34","msg":"begin sync 4315775645621696770 (9 upstreams)","version":"1.3.0"} {"level":"info","ts":1587577982.7586253,"logger":"uds.v1.event_loop.uds.v1.event_loop.syncer","caller":"discovery/discovery.go:143","msg":"reconciled upstreams","version":"1.3.0","discovered_by":"kubernetesplugin","upstreams":9} {"level":"info","ts":1587577982.7586682,"logger":"uds.v1.event_loop.uds.v1.event_loop.syncer","caller":"syncer/discovery_syncer.go:44","msg":"end sync 4315775645621696770","version":"1.3.0"} {"level":"info","ts":1587577982.7593758,"logger":"fds.v1.event_loop.fds.v1.event_loop.syncer","caller":"syncer/discovery_syncer.go:35","msg":"begin sync 4315775645621696770 (9 upstreams)","version":"1.3.0"} {"level":"info","ts":1587577982.7594168,"logger":"fds.v1.event_loop.fds.v1.event_loop.syncer","caller":"syncer/discovery_syncer.go:46","msg":"end sync 4315775645621696770","version":"1.3.0"} {"level":"info","ts":1587577987.6382718,"logger":"uds.v1.event_loop.uds","caller":"discovery/discovery.go:143","msg":"reconciled upstreams","version":"1.3.0","discovered_by":"kubernetesplugin","upstreams":9} {"level":"info","ts":1587577992.642142,"logger":"uds.v1.event_loop.uds","caller":"discovery/discovery.go:143","msg":"reconciled upstreams","version":"1.3.0","discovered_by":"kubernetesplugin","upstreams":9} {"level":"info","ts":1587577993.756452,"logger":"uds.v1.event_loop.uds.v1.event_loop.syncer","caller":"syncer/discovery_syncer.go:34","msg":"begin sync 6708220120071169714 (0 upstreams)","version":"1.3.0"} {"level":"info","ts":1587577993.7592,"logger":"fds.v1.event_loop.fds.v1.event_loop.syncer","caller":"syncer/discovery_syncer.go:35","msg":"begin sync 6708220120071169714 (0 upstreams)","version":"1.3.0"} {"level":"info","ts":1587577993.7592497,"logger":"fds.v1.event_loop.fds.v1.event_loop.syncer","caller":"syncer/discovery_syncer.go:46","msg":"end sync 6708220120071169714","version":"1.3.0"} {"level":"error","ts":1587577993.76259,"logger":"uds.v1.event_loop.uds.v1.event_loop.syncer","caller":"discovery/discovery.go:139","msg":"failed reconciling upstreams","version":"1.3.0","discovered_by":"kubernetesplugin","upstreams":9,"error":"reconciling resource helm-test-7966-1-gloo-9977: creating kube resource helm-test-7966-1-gloo-9977: the server could not find the requested resource (post upstreams.gloo.solo.io)","errorVerbose":"the server could not find the requested resource (post upstreams.gloo.solo.io)\ncreating kube resource helm-test-7966-1-gloo-9977\ngithub.com/solo-io/solo-kit/pkg/errors.Wrapf\n\t/workspace/gopath/pkg/mod/github.com/solo-io/solo-kit@v0.11.16/pkg/errors/errors.go:12\ngithub.com/solo-io/solo-kit/pkg/api/v1/clients/kube.(*ResourceClient).Write\n\t/workspace/gopath/pkg/mod/github.com/solo-io/solo-kit@v0.11.16/pkg/api/v1/clients/kube/resource_client.go:242\ngithub.com/solo-io/solo-kit/pkg/api/v1/reconcile.(*reconciler).syncResource\n\t/workspace/gopath/pkg/mod/github.com/solo-io/solo-kit@v0.11.16/pkg/api/v1/reconcile/reconciler.go:92\ngithub.com/solo-io/solo-kit/pkg/api/v1/reconcile.(*reconciler).Reconcile\n\t/workspace/gopath/pkg/mod/github.com/solo-io/solo-kit@v0.11.16/pkg/api/v1/reconcile/reconciler.go:39\ngithub.com/solo-io/gloo/projects/gloo/pkg/api/v1.(*upstreamReconciler).Reconcile\n\t/workspace/gopath/src/github.com/solo-io/gloo/projects/gloo/pkg/api/v1/upstream_reconciler.sk.go:46\ngithub.com/solo-io/gloo/projects/gloo/pkg/discovery.(*UpstreamDiscovery).Resync\n\t/workspace/gopath/src/github.com/solo-io/gloo/projects/gloo/pkg/discovery/discovery.go:135\ngithub.com/solo-io/gloo/projects/discovery/pkg/uds/syncer.(*syncer).Sync\n\t/workspace/gopath/src/github.com/solo-io/gloo/projects/discovery/pkg/uds/syncer/discovery_syncer.go:44\ngithub.com/solo-io/gloo/projects/gloo/pkg/api/v1.(*discoveryEventLoop).Run.func1\n\t/workspace/gopath/src/github.com/solo-io/gloo/projects/gloo/pkg/api/v1/discovery_event_loop.sk.go:84\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357\nreconciling resource helm-test-7966-1-gloo-9977\ngithub.com/solo-io/solo-kit/pkg/errors.Wrapf\n\t/workspace/gopath/pkg/mod/github.com/solo-io/solo-kit@v0.11.16/pkg/errors/errors.go:12\ngithub.com/solo-io/solo-kit/pkg/api/v1/reconcile.(*reconciler).Reconcile\n\t/workspace/gopath/pkg/mod/github.com/solo-io/solo-kit@v0.11.16/pkg/api/v1/reconcile/reconciler.go:40\ngithub.com/solo-io/gloo/projects/gloo/pkg/api/v1.(*upstreamReconciler).Reconcile\n\t/workspace/gopath/src/github.com/solo-io/gloo/projects/gloo/pkg/api/v1/upstream_reconciler.sk.go:46\ngithub.com/solo-io/gloo/projects/gloo/pkg/discovery.(*UpstreamDiscovery).Resync\n\t/workspace/gopath/src/github.com/solo-io/gloo/projects/gloo/pkg/discovery/discovery.go:135\ngithub.com/solo-io/gloo/projects/discovery/pkg/uds/syncer.(*syncer).Sync\n\t/workspace/gopath/src/github.com/solo-io/gloo/projects/discovery/pkg/uds/syncer/discovery_syncer.go:44\ngithub.com/solo-io/gloo/projects/gloo/pkg/api/v1.(*discoveryEventLoop).Run.func1\n\t/workspace/gopath/src/github.com/solo-io/gloo/projects/gloo/pkg/api/v1/discovery_event_loop.sk.go:84\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357","stacktrace":"github.com/solo-io/gloo/projects/gloo/pkg/discovery.(*UpstreamDiscovery).Resync\n\t/workspace/gopath/src/github.com/solo-io/gloo/projects/gloo/pkg/discovery/discovery.go:139\ngithub.com/solo-io/gloo/projects/discovery/pkg/uds/syncer.(*syncer).Sync\n\t/workspace/gopath/src/github.com/solo-io/gloo/projects/discovery/pkg/uds/syncer/discovery_syncer.go:44\ngithub.com/solo-io/gloo/projects/gloo/pkg/api/v1.(*discoveryEventLoop).Run.func1\n\t/workspace/gopath/src/github.com/solo-io/gloo/projects/gloo/pkg/api/v1/discovery_event_loop.sk.go:84"} {"level":"info","ts":1587577993.7628667,"logger":"uds.v1.event_loop.uds.v1.event_loop.syncer","caller":"syncer/discovery_syncer.go:44","msg":"end sync 6708220120071169714","version":"1.3.0"} {"level":"error","ts":1587577993.7629685,"logger":"uds.v1.event_loop.uds","caller":"syncer/setup_syncer.go:99","msg":"error: event_loop.uds: reconciling resource helm-test-7966-1-gloo-9977: creating kube resource helm-test-7966-1-gloo-9977: the server could not find the requested resource (post upstreams.gloo.solo.io)","version":"1.3.0","stacktrace":"github.com/solo-io/gloo/projects/discovery/pkg/uds/syncer.RunUDS.func2\n\t/workspace/gopath/src/github.com/solo-io/gloo/projects/discovery/pkg/uds/syncer/setup_syncer.go:99"} {"level":"error","ts":1587577997.6440556,"logger":"uds.v1.event_loop.uds","caller":"discovery/discovery.go:139","msg":"failed reconciling upstreams","version":"1.3.0","discovered_by":"kubernetesplugin","upstreams":9,"error":"reconciling resource helm-test-7966-1-gloo-9977: creating kube resource helm-test-7966-1-gloo-9977: the server could not find the requested resource (post upstreams.gloo.solo.io)","errorVerbose":"the server could not find the requested resource (post upstreams.gloo.solo.io)\ncreating kube resource helm-test-7966-1-gloo-9977\ngithub.com/solo-io/solo-kit/pkg/errors.Wrapf\n\t/workspace/gopath/pkg/mod/github.com/solo-io/solo-kit@v0.11.16/pkg/errors/errors.go:12\ngithub.com/solo-io/solo-kit/pkg/api/v1/clients/kube.(*ResourceClient).Write\n\t/workspace/gopath/pkg/mod/github.com/solo-io/solo-kit@v0.11.16/pkg/api/v1/clients/kube/resource_client.go:242\ngithub.com/solo-io/solo-kit/pkg/api/v1/reconcile.(*reconciler).syncResource\n\t/workspace/gopath/pkg/mod/github.com/solo-io/solo-kit@v0.11.16/pkg/api/v1/reconcile/reconciler.go:92\ngithub.com/solo-io/solo-kit/pkg/api/v1/reconcile.(*reconciler).Reconcile\n\t/workspace/gopath/pkg/mod/github.com/solo-io/solo-kit@v0.11.16/pkg/api/v1/reconcile/reconciler.go:39\ngithub.com/solo-io/gloo/projects/gloo/pkg/api/v1.(*upstreamReconciler).Reconcile\n\t/workspace/gopath/src/github.com/solo-io/gloo/projects/gloo/pkg/api/v1/upstream_reconciler.sk.go:46\ngithub.com/solo-io/gloo/projects/gloo/pkg/discovery.(*UpstreamDiscovery).Resync\n\t/workspace/gopath/src/github.com/solo-io/gloo/projects/gloo/pkg/discovery/discovery.go:135\ngithub.com/solo-io/gloo/projects/gloo/pkg/discovery.(*UpstreamDiscovery).StartUds.func1\n\t/workspace/gopath/src/github.com/solo-io/gloo/projects/gloo/pkg/discovery/discovery.go:106\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357\nreconciling resource helm-test-7966-1-gloo-9977\ngithub.com/solo-io/solo-kit/pkg/errors.Wrapf\n\t/workspace/gopath/pkg/mod/github.com/solo-io/solo-kit@v0.11.16/pkg/errors/errors.go:12\ngithub.com/solo-io/solo-kit/pkg/api/v1/reconcile.(*reconciler).Reconcile\n\t/workspace/gopath/pkg/mod/github.com/solo-io/solo-kit@v0.11.16/pkg/api/v1/reconcile/reconciler.go:40\ngithub.com/solo-io/gloo/projects/gloo/pkg/api/v1.(*upstreamReconciler).Reconcile\n\t/workspace/gopath/src/github.com/solo-io/gloo/projects/gloo/pkg/api/v1/upstream_reconciler.sk.go:46\ngithub.com/solo-io/gloo/projects/gloo/pkg/discovery.(*UpstreamDiscovery).Resync\n\t/workspace/gopath/src/github.com/solo-io/gloo/projects/gloo/pkg/discovery/discovery.go:135\ngithub.com/solo-io/gloo/projects/gloo/pkg/discovery.(*UpstreamDiscovery).StartUds.func1\n\t/workspace/gopath/src/github.com/solo-io/gloo/projects/gloo/pkg/discovery/discovery.go:106\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357","stacktrace":"github.com/solo-io/gloo/projects/gloo/pkg/discovery.(*UpstreamDiscovery).Resync\n\t/workspace/gopath/src/github.com/solo-io/gloo/projects/gloo/pkg/discovery/discovery.go:139\ngithub.com/solo-io/gloo/projects/gloo/pkg/discovery.(*UpstreamDiscovery).StartUds.func1\n\t/workspace/gopath/src/github.com/solo-io/gloo/projects/gloo/pkg/discovery/discovery.go:106"} {"level":"error","ts":1587577997.6442585,"logger":"uds.v1.event_loop.uds","caller":"syncer/setup_syncer.go:99","msg":"error: event_loop.uds: error in uds plugin : reconciling resource helm-test-7966-1-gloo-9977: creating kube resource helm-test-7966-1-gloo-9977: the server could not find the requested resource (post upstreams.gloo.solo.io)","version":"1.3.0","stacktrace":"github.com/solo-io/gloo/projects/discovery/pkg/uds/syncer.RunUDS.func2\n\t/workspace/gopath/src/github.com/solo-io/gloo/projects/discovery/pkg/uds/syncer/setup_syncer.go:99"} {"level":"error","ts":1587578002.644694,"logger":"uds.v1.event_loop.uds","caller":"discovery/discovery.go:139","msg":"failed reconciling upstreams","version":"1.3.0","discovered_by":"kubernetesplugin","upstreams":9,"error":"reconciling resource helm-test-7966-1-gloo-9977: creating kube resource helm-test-7966-1-gloo-9977: the server could not find the requested resource (post upstreams.gloo.solo.io)","errorVerbose":"the server could not find the requested resource (post upstreams.gloo.solo.io)\ncreating kube resource helm-test-7966-1-gloo-9977\ngithub.com/solo-io/solo-kit/pkg/errors.Wrapf\n\t/workspace/gopath/pkg/mod/github.com/solo-io/solo-kit@v0.11.16/pkg/errors/errors.go:12\ngithub.com/solo-io/solo-kit/pkg/api/v1/clients/kube.(*ResourceClient).Write\n\t/workspace/gopath/pkg/mod/github.com/solo-io/solo-kit@v0.11.16/pkg/api/v1/clients/kube/resource_client.go:242\ngithub.com/solo-io/solo-kit/pkg/api/v1/reconcile.(*reconciler).syncResource\n\t/workspace/gopath/pkg/mod/github.com/solo-io/solo-kit@v0.11.16/pkg/api/v1/reconcile/reconciler.go:92\ngithub.com/solo-io/solo-kit/pkg/api/v1/reconcile.(*reconciler).Reconcile\n\t/workspace/gopath/pkg/mod/github.com/solo-io/solo-kit@v0.11.16/pkg/api/v1/reconcile/reconciler.go:39\ngithub.com/solo-io/gloo/projects/gloo/pkg/api/v1.(*upstreamReconciler).Reconcile\n\t/workspace/gopath/src/github.com/solo-io/gloo/projects/gloo/pkg/api/v1/upstream_reconciler.sk.go:46\ngithub.com/solo-io/gloo/projects/gloo/pkg/discovery.(*UpstreamDiscovery).Resync\n\t/workspace/gopath/src/github.com/solo-io/gloo/projects/gloo/pkg/discovery/discovery.go:135\ngithub.com/solo-io/gloo/projects/gloo/pkg/discovery.(*UpstreamDiscovery).StartUds.func1\n\t/workspace/gopath/src/github.com/solo-io/gloo/projects/gloo/pkg/discovery/discovery.go:106\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357\nreconciling resource helm-test-7966-1-gloo-9977\ngithub.com/solo-io/solo-kit/pkg/errors.Wrapf\n\t/workspace/gopath/pkg/mod/github.com/solo-io/solo-kit@v0.11.16/pkg/errors/errors.go:12\ngithub.com/solo-io/solo-kit/pkg/api/v1/reconcile.(*reconciler).Reconcile\n\t/workspace/gopath/pkg/mod/github.com/solo-io/solo-kit@v0.11.16/pkg/api/v1/reconcile/reconciler.go:40\ngithub.com/solo-io/gloo/projects/gloo/pkg/api/v1.(*upstreamReconciler).Reconcile\n\t/workspace/gopath/src/github.com/solo-io/gloo/projects/gloo/pkg/api/v1/upstream_reconciler.sk.go:46\ngithub.com/solo-io/gloo/projects/gloo/pkg/discovery.(*UpstreamDiscovery).Resync\n\t/workspace/gopath/src/github.com/solo-io/gloo/projects/gloo/pkg/discovery/discovery.go:135\ngithub.com/solo-io/gloo/projects/gloo/pkg/discovery.(*UpstreamDiscovery).StartUds.func1\n\t/workspace/gopath/src/github.com/solo-io/gloo/projects/gloo/pkg/discovery/discovery.go:106\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357","stacktrace":"github.com/solo-io/gloo/projects/gloo/pkg/discovery.(*UpstreamDiscovery).Resync\n\t/workspace/gopath/src/github.com/solo-io/gloo/projects/gloo/pkg/discovery/discovery.go:139\ngithub.com/solo-io/gloo/projects/gloo/pkg/discovery.(*UpstreamDiscovery).StartUds.func1\n\t/workspace/gopath/src/github.com/solo-io/gloo/projects/gloo/pkg/discovery/discovery.go:106"} {"level":"error","ts":1587578002.645075,"logger":"uds.v1.event_loop.uds","caller":"syncer/setup_syncer.go:99","msg":"error: event_loop.uds: error in uds plugin : reconciling resource helm-test-7966-1-gloo-9977: creating kube resource helm-test-7966-1-gloo-9977: the server could not find the requested resource (post upstreams.gloo.solo.io)","version":"1.3.0","stacktrace":"github.com/solo-io/gloo/projects/discovery/pkg/uds/syncer.RunUDS.func2\n\t/workspace/gopath/src/github.com/solo-io/gloo/projects/discovery/pkg/uds/syncer/setup_syncer.go:99"} {"level":"error","ts":1587578007.644539,"logger":"uds.v1.event_loop.uds","caller":"discovery/discovery.go:139","msg":"failed reconciling upstreams","version":"1.3.0","discovered_by":"kubernetesplugin","upstreams":9,"error":"reconciling resource helm-test-7966-1-gloo-9977: creating kube resource helm-test-7966-1-gloo-9977: the server could not find the requested resource (post upstreams.gloo.solo.io)","errorVerbose":"the server could not find the requested resource (post upstreams.gloo.solo.io)\ncreating kube resource helm-test-7966-1-gloo-9977\ngithub.com/solo-io/solo-kit/pkg/errors.Wrapf\n\t/workspace/gopath/pkg/mod/github.com/solo-io/solo-kit@v0.11.16/pkg/errors/errors.go:12\ngithub.com/solo-io/solo-kit/pkg/api/v1/clients/kube.(*ResourceClient).Write\n\t/workspace/gopath/pkg/mod/github.com/solo-io/solo-kit@v0.11.16/pkg/api/v1/clients/kube/resource_client.go:242\ngithub.com/solo-io/solo-kit/pkg/api/v1/reconcile.(*reconciler).syncResource\n\t/workspace/gopath/pkg/mod/github.com/solo-io/solo-kit@v0.11.16/pkg/api/v1/reconcile/reconciler.go:92\ngithub.com/solo-io/solo-kit/pkg/api/v1/reconcile.(*reconciler).Reconcile\n\t/workspace/gopath/pkg/mod/github.com/solo-io/solo-kit@v0.11.16/pkg/api/v1/reconcile/reconciler.go:39\ngithub.com/solo-io/gloo/projects/gloo/pkg/api/v1.(*upstreamReconciler).Reconcile\n\t/workspace/gopath/src/github.com/solo-io/gloo/projects/gloo/pkg/api/v1/upstream_reconciler.sk.go:46\ngithub.com/solo-io/gloo/projects/gloo/pkg/discovery.(*UpstreamDiscovery).Resync\n\t/workspace/gopath/src/github.com/solo-io/gloo/projects/gloo/pkg/discovery/discovery.go:135\ngithub.com/solo-io/gloo/projects/gloo/pkg/discovery.(*UpstreamDiscovery).StartUds.func1\n\t/workspace/gopath/src/github.com/solo-io/gloo/projects/gloo/pkg/discovery/discovery.go:106\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357\nreconciling resource helm-test-7966-1-gloo-9977\ngithub.com/solo-io/solo-kit/pkg/errors.Wrapf\n\t/workspace/gopath/pkg/mod/github.com/solo-io/solo-kit@v0.11.16/pkg/errors/errors.go:12\ngithub.com/solo-io/solo-kit/pkg/api/v1/reconcile.(*reconciler).Reconcile\n\t/workspace/gopath/pkg/mod/github.com/solo-io/solo-kit@v0.11.16/pkg/api/v1/reconcile/reconciler.go:40\ngithub.com/solo-io/gloo/projects/gloo/pkg/api/v1.(*upstreamReconciler).Reconcile\n\t/workspace/gopath/src/github.com/solo-io/gloo/projects/gloo/pkg/api/v1/upstream_reconciler.sk.go:46\ngithub.com/solo-io/gloo/projects/gloo/pkg/discovery.(*UpstreamDiscovery).Resync\n\t/workspace/gopath/src/github.com/solo-io/gloo/projects/gloo/pkg/discovery/discovery.go:135\ngithub.com/solo-io/gloo/projects/gloo/pkg/discovery.(*UpstreamDiscovery).StartUds.func1\n\t/workspace/gopath/src/github.com/solo-io/gloo/projects/gloo/pkg/discovery/discovery.go:106\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357","stacktrace":"github.com/solo-io/gloo/projects/gloo/pkg/discovery.(*UpstreamDiscovery).Resync\n\t/workspace/gopath/src/github.com/solo-io/gloo/projects/gloo/pkg/discovery/discovery.go:139\ngithub.com/solo-io/gloo/projects/gloo/pkg/discovery.(*UpstreamDiscovery).StartUds.func1\n\t/workspace/gopath/src/github.com/solo-io/gloo/projects/gloo/pkg/discovery/discovery.go:106"} {"level":"error","ts":1587578007.6451287,"logger":"uds.v1.event_loop.uds","caller":"syncer/setup_syncer.go:99","msg":"error: event_loop.uds: error in uds plugin : reconciling resource helm-test-7966-1-gloo-9977: creating kube resource helm-test-7966-1-gloo-9977: the server could not find the requested resource (post upstreams.gloo.solo.io)","version":"1.3.0","stacktrace":"github.com/solo-io/gloo/projects/discovery/pkg/uds/syncer.RunUDS.func2\n\t/workspace/gopath/src/github.com/solo-io/gloo/projects/discovery/pkg/uds/syncer/setup_syncer.go:99"} {"level":"error","ts":1587578012.6514795,"logger":"uds.v1.event_loop.uds","caller":"discovery/discovery.go:139","msg":"failed reconciling upstreams","version":"1.3.0","discovered_by":"kubernetesplugin","upstreams":9,"error":"reconciling resource helm-test-7966-1-gloo-9977: creating kube resource helm-test-7966-1-gloo-9977: the server could not find the requested resource (post upstreams.gloo.solo.io)","errorVerbose":"the server could not find the requested resource (post upstreams.gloo.solo.io)\ncreating kube resource helm-test-7966-1-gloo-9977\ngithub.com/solo-io/solo-kit/pkg/errors.Wrapf\n\t/workspace/gopath/pkg/mod/github.com/solo-io/solo-kit@v0.11.16/pkg/errors/errors.go:12\ngithub.com/solo-io/solo-kit/pkg/api/v1/clients/kube.(*ResourceClient).Write\n\t/workspace/gopath/pkg/mod/github.com/solo-io/solo-kit@v0.11.16/pkg/api/v1/clients/kube/resource_client.go:242\ngithub.com/solo-io/solo-kit/pkg/api/v1/reconcile.(*reconciler).syncResource\n\t/workspace/gopath/pkg/mod/github.com/solo-io/solo-kit@v0.11.16/pkg/api/v1/reconcile/reconciler.go:92\ngithub.com/solo-io/solo-kit/pkg/api/v1/reconcile.(*reconciler).Reconcile\n\t/workspace/gopath/pkg/mod/github.com/solo-io/solo-kit@v0.11.16/pkg/api/v1/reconcile/reconciler.go:39\ngithub.com/solo-io/gloo/projects/gloo/pkg/api/v1.(*upstreamReconciler).Reconcile\n\t/workspace/gopath/src/github.com/solo-io/gloo/projects/gloo/pkg/api/v1/upstream_reconciler.sk.go:46\ngithub.com/solo-io/gloo/projects/gloo/pkg/discovery.(*UpstreamDiscovery).Resync\n\t/workspace/gopath/src/github.com/solo-io/gloo/projects/gloo/pkg/discovery/discovery.go:135\ngithub.com/solo-io/gloo/projects/gloo/pkg/discovery.(*UpstreamDiscovery).StartUds.func1\n\t/workspace/gopath/src/github.com/solo-io/gloo/projects/gloo/pkg/discovery/discovery.go:106\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357\nreconciling resource helm-test-7966-1-gloo-9977\ngithub.com/solo-io/solo-kit/pkg/errors.Wrapf\n\t/workspace/gopath/pkg/mod/github.com/solo-io/solo-kit@v0.11.16/pkg/errors/errors.go:12\ngithub.com/solo-io/solo-kit/pkg/api/v1/reconcile.(*reconciler).Reconcile\n\t/workspace/gopath/pkg/mod/github.com/solo-io/solo-kit@v0.11.16/pkg/api/v1/reconcile/reconciler.go:40\ngithub.com/solo-io/gloo/projects/gloo/pkg/api/v1.(*upstreamReconciler).Reconcile\n\t/workspace/gopath/src/github.com/solo-io/gloo/projects/gloo/pkg/api/v1/upstream_reconciler.sk.go:46\ngithub.com/solo-io/gloo/projects/gloo/pkg/discovery.(*UpstreamDiscovery).Resync\n\t/workspace/gopath/src/github.com/solo-io/gloo/projects/gloo/pkg/discovery/discovery.go:135\ngithub.com/solo-io/gloo/projects/gloo/pkg/discovery.(*UpstreamDiscovery).StartUds.func1\n\t/workspace/gopath/src/github.com/solo-io/gloo/projects/gloo/pkg/discovery/discovery.go:106\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357","stacktrace":"github.com/solo-io/gloo/projects/gloo/pkg/discovery.(*UpstreamDiscovery).Resync\n\t/workspace/gopath/src/github.com/solo-io/gloo/projects/gloo/pkg/discovery/discovery.go:139\ngithub.com/solo-io/gloo/projects/gloo/pkg/discovery.(*UpstreamDiscovery).StartUds.func1\n\t/workspace/gopath/src/github.com/solo-io/gloo/projects/gloo/pkg/discovery/discovery.go:106"} {"level":"error","ts":1587578012.6517844,"logger":"uds.v1.event_loop.uds","caller":"syncer/setup_syncer.go:99","msg":"error: event_loop.uds: error in uds plugin : reconciling resource helm-test-7966-1-gloo-9977: creating kube resource helm-test-7966-1-gloo-9977: the server could not find the requested resource (post upstreams.gloo.solo.io)","version":"1.3.0","stacktrace":"github.com/solo-io/gloo/projects/discovery/pkg/uds/syncer.RunUDS.func2\n\t/workspace/gopath/src/github.com/solo-io/gloo/projects/discovery/pkg/uds/syncer/setup_syncer.go:99"} {"level":"error","ts":1587578017.6474948,"logger":"uds.v1.event_loop.uds","caller":"discovery/discovery.go:139","msg":"failed reconciling upstreams","version":"1.3.0","discovered_by":"kubernetesplugin","upstreams":9,"error":"reconciling resource helm-test-7966-1-gloo-9977: creating kube resource helm-test-7966-1-gloo-9977: the server could not find the requested resource (post upstreams.gloo.solo.io)","errorVerbose":"the server could not find the requested resource (post upstreams.gloo.solo.io)\ncreating kube resource helm-test-7966-1-gloo-9977\ngithub.com/solo-io/solo-kit/pkg/errors.Wrapf\n\t/workspace/gopath/pkg/mod/github.com/solo-io/solo-kit@v0.11.16/pkg/errors/errors.go:12\ngithub.com/solo-io/solo-kit/pkg/api/v1/clients/kube.(*ResourceClient).Write\n\t/workspace/gopath/pkg/mod/github.com/solo-io/solo-kit@v0.11.16/pkg/api/v1/clients/kube/resource_client.go:242\ngithub.com/solo-io/solo-kit/pkg/api/v1/reconcile.(*reconciler).syncResource\n\t/workspace/gopath/pkg/mod/github.com/solo-io/solo-kit@v0.11.16/pkg/api/v1/reconcile/reconciler.go:92\ngithub.com/solo-io/solo-kit/pkg/api/v1/reconcile.(*reconciler).Reconcile\n\t/workspace/gopath/pkg/mod/github.com/solo-io/solo-kit@v0.11.16/pkg/api/v1/reconcile/reconciler.go:39\ngithub.com/solo-io/gloo/projects/gloo/pkg/api/v1.(*upstreamReconciler).Reconcile\n\t/workspace/gopath/src/github.com/solo-io/gloo/projects/gloo/pkg/api/v1/upstream_reconciler.sk.go:46\ngithub.com/solo-io/gloo/projects/gloo/pkg/discovery.(*UpstreamDiscovery).Resync\n\t/workspace/gopath/src/github.com/solo-io/gloo/projects/gloo/pkg/discovery/discovery.go:135\ngithub.com/solo-io/gloo/projects/gloo/pkg/discovery.(*UpstreamDiscovery).StartUds.func1\n\t/workspace/gopath/src/github.com/solo-io/gloo/projects/gloo/pkg/discovery/discovery.go:106\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357\nreconciling resource helm-test-7966-1-gloo-9977\ngithub.com/solo-io/solo-kit/pkg/errors.Wrapf\n\t/workspace/gopath/pkg/mod/github.com/solo-io/solo-kit@v0.11.16/pkg/errors/errors.go:12\ngithub.com/solo-io/solo-kit/pkg/api/v1/reconcile.(*reconciler).Reconcile\n\t/workspace/gopath/pkg/mod/github.com/solo-io/solo-kit@v0.11.16/pkg/api/v1/reconcile/reconciler.go:40\ngithub.com/solo-io/gloo/projects/gloo/pkg/api/v1.(*upstreamReconciler).Reconcile\n\t/workspace/gopath/src/github.com/solo-io/gloo/projects/gloo/pkg/api/v1/upstream_reconciler.sk.go:46\ngithub.com/solo-io/gloo/projects/gloo/pkg/discovery.(*UpstreamDiscovery).Resync\n\t/workspace/gopath/src/github.com/solo-io/gloo/projects/gloo/pkg/discovery/discovery.go:135\ngithub.com/solo-io/gloo/projects/gloo/pkg/discovery.(*UpstreamDiscovery).StartUds.func1\n\t/workspace/gopath/src/github.com/solo-io/gloo/projects/gloo/pkg/discovery/discovery.go:106\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357","stacktrace":"github.com/solo-io/gloo/projects/gloo/pkg/discovery.(*UpstreamDiscovery).Resync\n\t/workspace/gopath/src/github.com/solo-io/gloo/projects/gloo/pkg/discovery/discovery.go:139\ngithub.com/solo-io/gloo/projects/gloo/pkg/discovery.(*UpstreamDiscovery).StartUds.func1\n\t/workspace/gopath/src/github.com/solo-io/gloo/projects/gloo/pkg/discovery/discovery.go:106"} {"level":"error","ts":1587578017.6477034,"logger":"uds.v1.event_loop.uds","caller":"syncer/setup_syncer.go:99","msg":"error: event_loop.uds: error in uds plugin : reconciling resource helm-test-7966-1-gloo-9977: creating kube resource helm-test-7966-1-gloo-9977: the server could not find the requested resource (post upstreams.gloo.solo.io)","version":"1.3.0","stacktrace":"github.com/solo-io/gloo/projects/discovery/pkg/uds/syncer.RunUDS.func2\n\t/workspace/gopath/src/github.com/solo-io/gloo/projects/discovery/pkg/uds/syncer/setup_syncer.go:99"}
LOGS FROM helm-test-7966-1.gateway-b8bcfc777-dm2jw: {"level":"info","ts":1587577977.536671,"logger":"gateway.v1.event_loop","caller":"v1/setup_event_loop.sk.go:57","msg":"event loop started","version":"1.3.0"} {"level":"info","ts":1587577977.677023,"logger":"gateway.v1.event_loop.gateway","caller":"validation/robust_client.go:34","msg":"starting proxy validation client... this may take a moment","version":"1.3.0","validation_server":"gloo:9988"} {"level":"info","ts":1587577998.68299,"logger":"gateway.v1.event_loop.gateway.v1.event_loop","caller":"v1/api_event_loop.sk.go:57","msg":"event loop started","version":"1.3.0"}
LOGS FROM helm-test-7966-1.gateway-proxy-6574cd99bb-q8fm6: [2020-04-22 17:52:57.604][6][info][main] [external/envoy/source/server/server.cc:251] initializing epoch 0 (hot restart version=disabled) [2020-04-22 17:52:57.604][6][info][main] [external/envoy/source/server/server.cc:253] statically linked extensions: [2020-04-22 17:52:57.604][6][info][main] [external/envoy/source/server/server.cc:255] access_loggers: envoy.file_access_log, envoy.http_grpc_access_log, envoy.tcp_grpc_access_log [2020-04-22 17:52:57.604][6][info][main] [external/envoy/source/server/server.cc:255] clusters: envoy.cluster.eds, envoy.cluster.logical_dns, envoy.cluster.original_dst, envoy.cluster.static, envoy.cluster.strict_dns, envoy.clusters.aggregate, envoy.clusters.dynamic_forward_proxy, envoy.clusters.redis [2020-04-22 17:52:57.604][6][info][main] [external/envoy/source/server/server.cc:255] dubbo_proxy.filters: envoy.filters.dubbo.router [2020-04-22 17:52:57.604][6][info][main] [external/envoy/source/server/server.cc:255] dubbo_proxy.protocols: dubbo [2020-04-22 17:52:57.604][6][info][main] [external/envoy/source/server/server.cc:255] dubbo_proxy.route_matchers: default [2020-04-22 17:52:57.604][6][info][main] [external/envoy/source/server/server.cc:255] dubbo_proxy.serializers: dubbo.hessian2 [2020-04-22 17:52:57.604][6][info][main] [external/envoy/source/server/server.cc:255] filters.http: envoy.buffer, envoy.cors, envoy.csrf, envoy.ext_authz, envoy.fault, envoy.filters.http.adaptive_concurrency, envoy.filters.http.dynamic_forward_proxy, envoy.filters.http.grpc_http1_reverse_bridge, envoy.filters.http.grpc_stats, envoy.filters.http.header_to_metadata, envoy.filters.http.jwt_authn, envoy.filters.http.original_src, envoy.filters.http.rbac, envoy.filters.http.tap, envoy.grpc_http1_bridge, envoy.grpc_json_transcoder, envoy.grpc_web, envoy.gzip, envoy.health_check, envoy.http_dynamo_filter, envoy.ip_tagging, envoy.lua, envoy.rate_limit, envoy.router, envoy.squash, io.solo.aws_lambda, io.solo.nats_streaming, io.solo.transformation [2020-04-22 17:52:57.604][6][info][main] [external/envoy/source/server/server.cc:255] filters.listener: envoy.listener.http_inspector, envoy.listener.original_dst, envoy.listener.original_src, envoy.listener.proxy_protocol, envoy.listener.tls_inspector [2020-04-22 17:52:57.604][6][info][main] [external/envoy/source/server/server.cc:255] filters.network: envoy.client_ssl_auth, envoy.echo, envoy.ext_authz, envoy.filters.network.dubbo_proxy, envoy.filters.network.mysql_proxy, envoy.filters.network.rbac, envoy.filters.network.sni_cluster, envoy.filters.network.thrift_proxy, envoy.filters.network.zookeeper_proxy, envoy.http_connection_manager, envoy.mongo_proxy, envoy.ratelimit, envoy.redis_proxy, envoy.tcp_proxy [2020-04-22 17:52:57.604][6][info][main] [external/envoy/source/server/server.cc:255] filters.udp_listener: envoy.filters.udp_listener.udp_proxy [2020-04-22 17:52:57.605][6][info][main] [external/envoy/source/server/server.cc:255] grpc_credentials: envoy.grpc_credentials.aws_iam, envoy.grpc_credentials.default, envoy.grpc_credentials.file_based_metadata [2020-04-22 17:52:57.605][6][info][main] [external/envoy/source/server/server.cc:255] health_checkers: envoy.health_checkers.redis [2020-04-22 17:52:57.605][6][info][main] [external/envoy/source/server/server.cc:255] resolvers: envoy.ip [2020-04-22 17:52:57.605][6][info][main] [external/envoy/source/server/server.cc:255] resource_monitors: envoy.resource_monitors.fixed_heap, envoy.resource_monitors.injected_resource [2020-04-22 17:52:57.605][6][info][main] [external/envoy/source/server/server.cc:255] retry_host_predicates: envoy.retry_host_predicates.omit_canary_hosts, envoy.retry_host_predicates.previous_hosts [2020-04-22 17:52:57.605][6][info][main] [external/envoy/source/server/server.cc:255] retry_priorities: envoy.retry_priorities.previous_priorities [2020-04-22 17:52:57.605][6][info][main] [external/envoy/source/server/server.cc:255] stats_sinks: envoy.dog_statsd, envoy.metrics_service, envoy.stat_sinks.hystrix, envoy.statsd [2020-04-22 17:52:57.605][6][info][main] [external/envoy/source/server/server.cc:255] thrift_proxy.filters: envoy.filters.thrift.rate_limit, envoy.filters.thrift.router [2020-04-22 17:52:57.605][6][info][main] [external/envoy/source/server/server.cc:255] thrift_proxy.protocols: auto, binary, binary/non-strict, compact, twitter [2020-04-22 17:52:57.605][6][info][main] [external/envoy/source/server/server.cc:255] thrift_proxy.transports: auto, framed, header, unframed [2020-04-22 17:52:57.605][6][info][main] [external/envoy/source/server/server.cc:255] tracers: envoy.dynamic.ot, envoy.lightstep, envoy.tracers.datadog, envoy.tracers.opencensus, envoy.tracers.xray, envoy.zipkin [2020-04-22 17:52:57.605][6][info][main] [external/envoy/source/server/server.cc:255] transport_sockets.downstream: envoy.transport_sockets.alts, envoy.transport_sockets.raw_buffer, envoy.transport_sockets.tap, envoy.transport_sockets.tls, raw_buffer, tls [2020-04-22 17:52:57.605][6][info][main] [external/envoy/source/server/server.cc:255] transport_sockets.upstream: envoy.transport_sockets.alts, envoy.transport_sockets.raw_buffer, envoy.transport_sockets.tap, envoy.transport_sockets.tls, raw_buffer, tls [2020-04-22 17:52:57.605][6][info][main] [external/envoy/source/server/server.cc:255] udp_listeners: raw_udp_listener [2020-04-22 17:52:57.608][6][warning][misc] [external/envoy/source/common/protobuf/utility.cc:356] Using deprecated option 'envoy.api.v2.listener.Filter.config' from file listener.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/intro/deprecated for details. [2020-04-22 17:52:57.609][6][warning][misc] [external/envoy/source/common/protobuf/utility.cc:356] Using deprecated option 'envoy.config.metrics.v2.StatsSink.config' from file stats.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/intro/deprecated for details. [2020-04-22 17:52:57.609][6][info][main] [external/envoy/source/server/server.cc:322] admin address: 127.0.0.1:19000 [2020-04-22 17:52:57.610][6][info][main] [external/envoy/source/server/server.cc:441] runtime: layers: - name: base static_layer: {} - name: admin admin_layer: {} [2020-04-22 17:52:57.610][6][info][config] [external/envoy/source/server/configuration_impl.cc:60] loading 0 static secret(s) [2020-04-22 17:52:57.610][6][info][config] [external/envoy/source/server/configuration_impl.cc:66] loading 4 cluster(s) [2020-04-22 17:52:57.615][6][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:91] gRPC config stream closed: 14, no healthy upstream [2020-04-22 17:52:57.615][6][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:54] Unable to establish new stream [2020-04-22 17:52:57.615][6][info][config] [external/envoy/source/server/configuration_impl.cc:70] loading 1 listener(s) [2020-04-22 17:52:57.617][6][warning][misc] [external/envoy/source/common/protobuf/utility.cc:356] Using deprecated option 'envoy.config.filter.network.http_connection_manager.v2.HttpFilter.config' from file http_connection_manager.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/intro/deprecated for details. [2020-04-22 17:52:57.619][6][info][config] [external/envoy/source/server/configuration_impl.cc:95] loading tracing configuration [2020-04-22 17:52:57.619][6][info][config] [external/envoy/source/server/configuration_impl.cc:115] loading stats sink configuration [2020-04-22 17:52:57.619][6][info][main] [external/envoy/source/server/server.cc:532] starting main dispatch loop [2020-04-22 17:52:57.640][6][info][upstream] [external/envoy/source/common/upstream/cluster_manager_impl.cc:163] cm init: initializing cds [2020-04-22 17:53:03.814][6][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:91] gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers. reset reason: connection failure [2020-04-22 17:53:12.641][6][info][upstream] [external/envoy/source/common/upstream/cluster_manager_impl.cc:167] cm init: all clusters initialized [2020-04-22 17:53:12.641][6][info][main] [external/envoy/source/server/server.cc:511] all clusters initialized. initializing init manager [2020-04-22 17:53:27.642][6][info][config] [external/envoy/source/server/listener_manager_impl.cc:661] all dependencies initialized. starting workers
LOGS FROM helm-test-7966-1.gloo-894fc69b5-rhxrj: {"level":"info","ts":1587577977.5150902,"logger":"gloo.v1.event_loop","caller":"v1/setup_event_loop.sk.go:57","msg":"event loop started","version":"1.3.0"} {"level":"info","ts":1587577977.734311,"logger":"gloo.v1.event_loop.setup.v1.event_loop","caller":"v1/eds_event_loop.sk.go:57","msg":"event loop started","version":"1.3.0"} {"level":"info","ts":1587577977.8375268,"logger":"gloo.v1.event_loop.setup.v1.event_loop","caller":"v1/api_event_loop.sk.go:57","msg":"event loop started","version":"1.3.0"} {"level":"info","ts":1587577977.8383248,"logger":"gloo.v1.event_loop.setup.v1.event_loop.syncer","caller":"discovery/run.go:35","msg":"begin sync 7705956483892295836 (10 upstreams)","version":"1.3.0"} {"level":"info","ts":1587577977.8385916,"logger":"gloo.v1.event_loop.setup.v1.event_loop.syncer","caller":"discovery/discovery.go:184","msg":"Received first EDS update from plugin: *ec2.plugin","version":"1.3.0"} {"level":"info","ts":1587577977.8418639,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer","caller":"syncer/envoy_translator_syncer.go:60","msg":"begin sync 2176305009050195658 (0 proxies, 10 upstreams, 0 endpoints, 1 secrets, 2 artifacts, 0 auth configs)","version":"1.3.0"} {"level":"info","ts":1587577977.8435874,"logger":"gloo.v1.event_loop.gloo.metrics","caller":"runner/run.go:50","msg":"Starting metrics server","version":"1.3.0"} {"level":"info","ts":1587577977.8436363,"logger":"gloo.v1.event_loop.gloo.metrics","caller":"runner/run.go:56","msg":"metrics server running in [gRPC] mode, listening at [:9966]","version":"1.3.0"} {"level":"info","ts":1587577977.9257476,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer","caller":"syncer/envoy_translator_syncer.go:149","msg":"end sync 2176305009050195658","version":"1.3.0"} {"level":"info","ts":1587577977.938711,"logger":"gloo.v1.event_loop.setup.v1.event_loop.syncer","caller":"discovery/run.go:65","msg":"end sync 7705956483892295836","version":"1.3.0"} {"level":"info","ts":1587577977.9388862,"logger":"gloo.v1.event_loop.setup.v1.event_loop.syncer","caller":"discovery/discovery.go:184","msg":"Received first EDS update from plugin: *kubernetes.plugin","version":"1.3.0"} {"level":"info","ts":1587577978.8416677,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer","caller":"syncer/envoy_translator_syncer.go:60","msg":"begin sync 11271630931848560383 (0 proxies, 10 upstreams, 2 endpoints, 1 secrets, 2 artifacts, 0 auth configs)","version":"1.3.0"} {"level":"info","ts":1587577979.8393118,"logger":"gloo.v1.event_loop.setup.v1.event_loop.syncer","caller":"discovery/run.go:35","msg":"begin sync 6378318002041217438 (11 upstreams)","version":"1.3.0"} {"level":"info","ts":1587577979.839687,"logger":"gloo.v1.event_loop.setup.v1.event_loop.syncer","caller":"discovery/discovery.go:184","msg":"Received first EDS update from plugin: *ec2.plugin","version":"1.3.0"} {"level":"info","ts":1587577979.940194,"logger":"gloo.v1.event_loop.setup.v1.event_loop.syncer","caller":"discovery/run.go:65","msg":"end sync 6378318002041217438","version":"1.3.0"} {"level":"info","ts":1587577979.940627,"logger":"gloo.v1.event_loop.setup.v1.event_loop.syncer","caller":"discovery/discovery.go:184","msg":"Received first EDS update from plugin: *kubernetes.plugin","version":"1.3.0"} {"level":"info","ts":1587577980.838951,"logger":"gloo.v1.event_loop.setup.v1.event_loop.syncer","caller":"discovery/run.go:35","msg":"begin sync 4780744194403547645 (13 upstreams)","version":"1.3.0"} {"level":"info","ts":1587577980.8391368,"logger":"gloo.v1.event_loop.setup.v1.event_loop.syncer","caller":"discovery/discovery.go:184","msg":"Received first EDS update from plugin: *ec2.plugin","version":"1.3.0"} {"level":"info","ts":1587577980.9393432,"logger":"gloo.v1.event_loop.setup.v1.event_loop.syncer","caller":"discovery/run.go:65","msg":"end sync 4780744194403547645","version":"1.3.0"} {"level":"info","ts":1587577980.9399145,"logger":"gloo.v1.event_loop.setup.v1.event_loop.syncer","caller":"discovery/discovery.go:184","msg":"Received first EDS update from plugin: *kubernetes.plugin","version":"1.3.0"} {"level":"info","ts":1587577981.8391263,"logger":"gloo.v1.event_loop.setup.v1.event_loop.syncer","caller":"discovery/run.go:35","msg":"begin sync 18131329834649021166 (15 upstreams)","version":"1.3.0"} {"level":"info","ts":1587577981.8393414,"logger":"gloo.v1.event_loop.setup.v1.event_loop.syncer","caller":"discovery/discovery.go:184","msg":"Received first EDS update from plugin: *ec2.plugin","version":"1.3.0"} {"level":"info","ts":1587577981.9395437,"logger":"gloo.v1.event_loop.setup.v1.event_loop.syncer","caller":"discovery/run.go:65","msg":"end sync 18131329834649021166","version":"1.3.0"} {"level":"info","ts":1587577981.9401364,"logger":"gloo.v1.event_loop.setup.v1.event_loop.syncer","caller":"discovery/discovery.go:184","msg":"Received first EDS update from plugin: *kubernetes.plugin","version":"1.3.0"} {"level":"info","ts":1587577982.1255631,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer","caller":"syncer/envoy_translator_syncer.go:149","msg":"end sync 11271630931848560383","version":"1.3.0"} {"level":"info","ts":1587577982.8392763,"logger":"gloo.v1.event_loop.setup.v1.event_loop.syncer","caller":"discovery/run.go:35","msg":"begin sync 9350366103665025439 (16 upstreams)","version":"1.3.0"} {"level":"info","ts":1587577982.8397212,"logger":"gloo.v1.event_loop.setup.v1.event_loop.syncer","caller":"discovery/discovery.go:184","msg":"Received first EDS update from plugin: *ec2.plugin","version":"1.3.0"} {"level":"info","ts":1587577982.841638,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer","caller":"syncer/envoy_translator_syncer.go:60","msg":"begin sync 6240644959319228173 (0 proxies, 16 upstreams, 0 endpoints, 1 secrets, 2 artifacts, 0 auth configs)","version":"1.3.0"} {"level":"info","ts":1587577982.9400156,"logger":"gloo.v1.event_loop.setup.v1.event_loop.syncer","caller":"discovery/run.go:65","msg":"end sync 9350366103665025439","version":"1.3.0"} {"level":"info","ts":1587577982.9406989,"logger":"gloo.v1.event_loop.setup.v1.event_loop.syncer","caller":"discovery/discovery.go:184","msg":"Received first EDS update from plugin: *kubernetes.plugin","version":"1.3.0"} {"level":"info","ts":1587577985.7277615,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer","caller":"syncer/envoy_translator_syncer.go:149","msg":"end sync 6240644959319228173","version":"1.3.0"} {"level":"info","ts":1587577985.8421388,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer","caller":"syncer/envoy_translator_syncer.go:60","msg":"begin sync 2787567119256544471 (0 proxies, 16 upstreams, 14 endpoints, 1 secrets, 2 artifacts, 0 auth configs)","version":"1.3.0"} {"level":"info","ts":1587577986.6483293,"caller":"cache/simple.go:224","msg":"open watch 1 for type.googleapis.com/envoy.api.v2.Cluster[] from nodeID \"helm-test-7966-1~gateway-proxy\", version \"\""} {"level":"info","ts":1587577989.9259503,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer","caller":"syncer/envoy_translator_syncer.go:149","msg":"end sync 2787567119256544471","version":"1.3.0"} {"level":"info","ts":1587577990.841848,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer","caller":"syncer/envoy_translator_syncer.go:60","msg":"begin sync 18328969164848882140 (0 proxies, 16 upstreams, 14 endpoints, 1 secrets, 2 artifacts, 0 auth configs)","version":"1.3.0"} {"level":"info","ts":1587577990.8422587,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer","caller":"syncer/envoy_translator_syncer.go:149","msg":"end sync 18328969164848882140","version":"1.3.0"} {"level":"info","ts":1587577992.6416273,"caller":"cache/simple.go:224","msg":"open watch 2 for type.googleapis.com/envoy.api.v2.Listener[] from nodeID \"helm-test-7966-1~gateway-proxy\", version \"\""} {"level":"info","ts":1587577992.8421838,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer","caller":"syncer/envoy_translator_syncer.go:60","msg":"begin sync 16461492759679082258 (0 proxies, 16 upstreams, 14 endpoints, 1 secrets, 2 artifacts, 0 auth configs)","version":"1.3.0"} {"level":"info","ts":1587577992.8427558,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer","caller":"syncer/envoy_translator_syncer.go:149","msg":"end sync 16461492759679082258","version":"1.3.0"} {"level":"info","ts":1587577993.839014,"logger":"gloo.v1.event_loop.setup.v1.event_loop.syncer","caller":"discovery/run.go:35","msg":"begin sync 1940954903023608175 (7 upstreams)","version":"1.3.0"} {"level":"info","ts":1587577993.8392925,"logger":"gloo.v1.event_loop.setup.v1.event_loop.syncer","caller":"discovery/discovery.go:184","msg":"Received first EDS update from plugin: *ec2.plugin","version":"1.3.0"} {"level":"info","ts":1587577993.8412657,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer","caller":"syncer/envoy_translator_syncer.go:60","msg":"begin sync 12461253091720208254 (0 proxies, 7 upstreams, 0 endpoints, 1 secrets, 2 artifacts, 0 auth configs)","version":"1.3.0"} {"level":"info","ts":1587577993.8415275,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer","caller":"syncer/envoy_translator_syncer.go:149","msg":"end sync 12461253091720208254","version":"1.3.0"} {"level":"info","ts":1587577993.9395752,"logger":"gloo.v1.event_loop.setup.v1.event_loop.syncer","caller":"discovery/run.go:65","msg":"end sync 1940954903023608175","version":"1.3.0"} {"level":"info","ts":1587577993.939941,"logger":"gloo.v1.event_loop.setup.v1.event_loop.syncer","caller":"discovery/discovery.go:184","msg":"Received first EDS update from plugin: *kubernetes.plugin","version":"1.3.0"} {"level":"info","ts":1587577994.8415513,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer","caller":"syncer/envoy_translator_syncer.go:60","msg":"begin sync 9728870973041699273 (0 proxies, 7 upstreams, 6 endpoints, 1 secrets, 2 artifacts, 0 auth configs)","version":"1.3.0"} {"level":"info","ts":1587577994.841775,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer","caller":"syncer/envoy_translator_syncer.go:149","msg":"end sync 9728870973041699273","version":"1.3.0"} {"level":"info","ts":1587577997.8432045,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer","caller":"syncer/envoy_translator_syncer.go:60","msg":"begin sync 4578984388128283038 (0 proxies, 7 upstreams, 6 endpoints, 1 secrets, 2 artifacts, 0 auth configs)","version":"1.3.0"} {"level":"info","ts":1587577997.8440397,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer","caller":"syncer/envoy_translator_syncer.go:149","msg":"end sync 4578984388128283038","version":"1.3.0"} {"level":"info","ts":1587578002.841483,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer","caller":"syncer/envoy_translator_syncer.go:60","msg":"begin sync 12781170036903047195 (0 proxies, 7 upstreams, 6 endpoints, 1 secrets, 2 artifacts, 0 auth configs)","version":"1.3.0"} {"level":"info","ts":1587578002.8418171,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer","caller":"syncer/envoy_translator_syncer.go:149","msg":"end sync 12781170036903047195","version":"1.3.0"} {"level":"info","ts":1587578007.8417006,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer","caller":"syncer/envoy_translator_syncer.go:60","msg":"begin sync 5873025908811858708 (0 proxies, 7 upstreams, 6 endpoints, 1 secrets, 2 artifacts, 0 auth configs)","version":"1.3.0"} {"level":"info","ts":1587578007.8420057,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer","caller":"syncer/envoy_translator_syncer.go:149","msg":"end sync 5873025908811858708","version":"1.3.0"} {"level":"info","ts":1587578012.8415399,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer","caller":"syncer/envoy_translator_syncer.go:60","msg":"begin sync 16192120747850968388 (0 proxies, 7 upstreams, 6 endpoints, 1 secrets, 2 artifacts, 0 auth configs)","version":"1.3.0"} {"level":"info","ts":1587578012.8418176,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer","caller":"syncer/envoy_translator_syncer.go:149","msg":"end sync 16192120747850968388","version":"1.3.0"} {"level":"info","ts":1587578017.8414893,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer","caller":"syncer/envoy_translator_syncer.go:60","msg":"begin sync 9734057434643395442 (0 proxies, 7 upstreams, 6 endpoints, 1 secrets, 2 artifacts, 0 auth configs)","version":"1.3.0"} {"level":"info","ts":1587578017.8418117,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer","caller":"syncer/envoy_translator_syncer.go:149","msg":"end sync 9734057434643395442","version":"1.3.0"}
** End Kubernetes Dump ** 

------------------------------
Failure [54.279 seconds]
[BeforeSuite] BeforeSuite 
/workspace/gloo/test/kube2e/helm/helm_suite_test.go:44

  Timed out after 40.001s.
  Expected
      <*eris.rootError | 0xc000b61c00>: {
          msg: "glooctl check detected a problem with the installation",
          stack: [0x25d6499, 0x51eeeb, 0x51e5b4, 0x63866a, 0x639054, 0x638152, 0x25d5a7f, 0x25e6678, 0x25ab6e0, 0x25ab198, 0x25aba33, 0x25bc041, 0x25bb8d0, 0x25bf1f0, 0x25c3499, 0x25c2f7e, 0x25e5a88, 0x5960ca, 0x48a3b1],
      }
  to be nil

  /workspace/gloo/test/kube2e/util.go:46

  Full Stack Trace
  github.com/solo-io/gloo/test/kube2e.GlooctlCheckEventuallyHealthy(0xc0009f7100, 0x2ad09bc, 0x3)
    /workspace/gloo/test/kube2e/util.go:46 +0x15f
  github.com/solo-io/gloo/test/kube2e/helm_test.StartTestHelper()
    /workspace/gloo/test/kube2e/helm/helm_suite_test.go:76 +0xa78
  github.com/onsi/ginkgo/internal/leafnodes.(*runner).runSync(0xc0004409c0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /go/pkg/mod/github.com/onsi/ginkgo@v1.11.0/internal/leafnodes/runner.go:113 +0xe0
  github.com/onsi/ginkgo/internal/leafnodes.(*runner).run(0xc0004409c0, 0xf690de4, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /go/pkg/mod/github.com/onsi/ginkgo@v1.11.0/internal/leafnodes/runner.go:64 +0x108
  github.com/onsi/ginkgo/internal/leafnodes.(*simpleSuiteNode).Run(0xc0002226e0, 0x1, 0x1, 0x0, 0x0, 0x3)
    /go/pkg/mod/github.com/onsi/ginkgo@v1.11.0/internal/leafnodes/suite_nodes.go:25 +0xb3
  github.com/onsi/ginkgo/internal/specrunner.(*SpecRunner).runBeforeSuite(0xc0001f6140, 0x0)
    /go/pkg/mod/github.com/onsi/ginkgo@v1.11.0/internal/specrunner/spec_runner.go:123 +0x161
  github.com/onsi/ginkgo/internal/specrunner.(*SpecRunner).Run(0xc0001f6140, 0x8)
    /go/pkg/mod/github.com/onsi/ginkgo@v1.11.0/internal/specrunner/spec_runner.go:63 +0xd0
  github.com/onsi/ginkgo/internal/suite.(*Suite).Run(0xc0002a5b80, 0x7f868d731390, 0xc00005ae00, 0x2ad9686, 0xa, 0xc0009cb130, 0x1, 0x1, 0x2ff2700, 0xc0000f9440, ...)
    /go/pkg/mod/github.com/onsi/ginkgo@v1.11.0/internal/suite/suite.go:62 +0x6a0
  github.com/onsi/ginkgo.RunSpecsWithCustomReporters(0x2f84960, 0xc00005ae00, 0x2ad9686, 0xa, 0xc0000cbe98, 0x1, 0x1, 0xc0009cb120)
    /go/pkg/mod/github.com/onsi/ginkgo@v1.11.0/ginkgo_dsl.go:226 +0x339
  github.com/onsi/ginkgo.RunSpecs(0x2f84960, 0xc00005ae00, 0x2ad9686, 0xa, 0x1)
    /go/pkg/mod/github.com/onsi/ginkgo@v1.11.0/ginkgo_dsl.go:207 +0x24e
  github.com/solo-io/gloo/test/kube2e/helm_test.TestHelm(0xc00005ae00)
    /workspace/gloo/test/kube2e/helm/helm_suite_test.go:39 +0x2e8
  testing.tRunner(0xc00005ae00, 0x2bab7b8)
    /usr/local/go/src/testing/testing.go:909 +0x19a
  created by testing.(*T).Run
    /usr/local/go/src/testing/testing.go:960 +0x652
------------------------------
"Wed, 22 Apr 2020 17:53:40 UTC: /go/pkg/mod/github.com/solo-io/go-utils@v0.14.1/testutils/helper/install.go:196"    WARNING:
error terminating "testrunner"

"Wed, 22 Apr 2020 17:53:40 UTC: /go/pkg/mod/github.com/solo-io/go-utils@v0.14.1/testutils/helper/install.go:200"    uninstalling gloo...

Ran 2 of 0 Specs in 75.663 seconds
FAIL! -- 0 Passed | 2 Failed | 0 Pending | 0 Skipped
--- FAIL: TestHelm (75.66s)
FAIL

Ginkgo ran 2 suites in 6m48.529974346s
Test Suite Failed
jenshu commented 2 years ago

helm test suite has been rewritten; haven't seen this recently