search

meshery / meshery-operator

Meshery Operator is a Kubernetes Operator that deploys and manages the lifecycle of two Meshery components critical to Meshery's operations of Kubernetes clusters
https://meshery.io
Apache License 2.0
108 stars 80 forks source link

Operator failing to deprovision; crashes Meshery #146

Closed leecalcote closed 3 years ago

leecalcote commented 3 years ago

Description Switching off the Operator in Settings results in an error.

Environment:

Logs

~ docker logs meshery_meshery_1
time="2021-03-23T00:19:40Z" level=info msg="Registered Meshery local Capabilities"
time="2021-03-23T00:19:40Z" level=info msg="Meshery server current channel: edge"
time="2021-03-23T00:19:40Z" level=info msg="Using '/home/appuser/.meshery/config' to store user data"
time="2021-03-23T00:19:40Z" level=info msg="Using '/home/appuser/.kube' as the folder to look for kubeconfig file"
time="2021-03-23T00:19:40Z" level=info msg="Log level: info"
taskq: 2021/03/23 00:19:40 consumer_config.go:249: taskq: system does not have enough free resources
time="2021-03-23T00:19:41Z" level=info msg="Starting Server listening on :8080"
time="2021-03-23T00:20:49Z" level=error msg="error in getting the token, http: named cookie not present"
time="2021-03-23T00:20:49Z" level=info msg="session not found"
time="2021-03-23T00:21:01Z" level=error msg="Unable to read data from bitcask store: error: key not found"
time="2021-03-23T00:21:01Z" level=info msg="retrieved user: {lcalcote   http://pbs.twimg.com/profile_images/880205475643441152/V_vhfnzb_normal.jpg}"
time="2021-03-23T00:21:01Z" level=info msg="retrieved user: {lcalcote   http://pbs.twimg.com/profile_images/880205475643441152/V_vhfnzb_normal.jpg}"
time="2021-03-23T00:21:01Z" level=info msg="retrieved user: {lcalcote   http://pbs.twimg.com/profile_images/880205475643441152/V_vhfnzb_normal.jpg}"
time="2021-03-23T00:21:01Z" level=error msg="failed to compare latest and current version of Meshery: failed to parse edge-latest, Malformed version: edge-latest"
time="2021-03-23T00:21:01Z" level=info msg="retrieved user: {lcalcote   http://pbs.twimg.com/profile_images/880205475643441152/V_vhfnzb_normal.jpg}"
time="2021-03-23T00:21:01Z" level=info msg="retrieved user: {lcalcote   http://pbs.twimg.com/profile_images/880205475643441152/V_vhfnzb_normal.jpg}"
time="2021-03-23T00:21:01Z" level=info msg="retrieved user: {lcalcote   http://pbs.twimg.com/profile_images/880205475643441152/V_vhfnzb_normal.jpg}"
time="2021-03-23T00:21:01Z" level=info msg="MeshSync subscription started" app=meshery
time="2021-03-23T00:21:01Z" level=info msg="retrieved user: {lcalcote   http://pbs.twimg.com/profile_images/880205475643441152/V_vhfnzb_normal.jpg}"
time="2021-03-23T00:21:01Z" level=info msg="retrieved user: {lcalcote   http://pbs.twimg.com/profile_images/880205475643441152/V_vhfnzb_normal.jpg}"
time="2021-03-23T00:21:01Z" level=info msg="retrieved user: {lcalcote   http://pbs.twimg.com/profile_images/880205475643441152/V_vhfnzb_normal.jpg}"
time="2021-03-23T00:21:01Z" level=error msg="Error creating a mesh client.: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp: lookup meshery-octarine on 127.0.0.11:53: no such host\""
time="2021-03-23T00:21:01Z" level=info msg="ControlPlane subscription started" app=meshery
time="2021-03-23T00:21:01Z" level=info msg="Operator subscription started" app=meshery
time="2021-03-23T00:21:02Z" level=info msg="retrieved user: {lcalcote   http://pbs.twimg.com/profile_images/880205475643441152/V_vhfnzb_normal.jpg}"
time="2021-03-23T00:21:05Z" level=info msg="retrieved user: {lcalcote   http://pbs.twimg.com/profile_images/880205475643441152/V_vhfnzb_normal.jpg}"
time="2021-03-23T00:21:05Z" level=info msg="retrieved user: {lcalcote   http://pbs.twimg.com/profile_images/880205475643441152/V_vhfnzb_normal.jpg}"
time="2021-03-23T00:21:21Z" level=error msg="failed to compare latest and current version of Meshery: failed to parse edge-latest, Malformed version: edge-latest"
time="2021-03-23T00:21:21Z" level=info msg="retrieved user: {lcalcote   http://pbs.twimg.com/profile_images/880205475643441152/V_vhfnzb_normal.jpg}"
time="2021-03-23T00:21:21Z" level=info msg="retrieved user: {lcalcote   http://pbs.twimg.com/profile_images/880205475643441152/V_vhfnzb_normal.jpg}"
time="2021-03-23T00:21:21Z" level=info msg="retrieved user: {lcalcote   http://pbs.twimg.com/profile_images/880205475643441152/V_vhfnzb_normal.jpg}"
time="2021-03-23T00:21:21Z" level=info msg="Operator subscription started" app=meshery
time="2021-03-23T00:21:21Z" level=info msg="MeshSync subscription started" app=meshery
time="2021-03-23T00:21:21Z" level=info msg="Installing Operator" app=meshery
time="2021-03-23T00:21:21Z" level=info msg="retrieved user: {lcalcote   http://pbs.twimg.com/profile_images/880205475643441152/V_vhfnzb_normal.jpg}"
time="2021-03-23T00:21:21Z" level=info msg="ControlPlane subscription started" app=meshery
time="2021-03-23T00:21:31Z" level=info msg="Operator operation executed" app=meshery
time="2021-03-23T00:21:32Z" level=info msg="Connected to broker at:host.docker.internal:4222" app=meshery
time="2021-03-23T00:21:32Z" level=info msg="Incoming object: eac074af.layer5.io, kind: ConfigMap" app=meshery
time="2021-03-23T00:21:32Z" level=info msg="Installing Operator" app=meshery
time="2021-03-23T00:21:32Z" level=info msg="Meshsync operation executed" app=meshery
time="2021-03-23T00:21:34Z" level=info msg="Incoming object: eac074af.layer5.io, kind: ConfigMap" app=meshery
time="2021-03-23T00:21:36Z" level=info msg="Incoming object: eac074af.layer5.io, kind: ConfigMap" app=meshery
time="2021-03-23T00:21:37Z" level=info msg="Incoming object: docker-desktop, kind: Node" app=meshery
time="2021-03-23T00:21:38Z" level=info msg="Incoming object: eac074af.layer5.io, kind: ConfigMap" app=meshery
time="2021-03-23T00:21:40Z" level=info msg="Incoming object: eac074af.layer5.io, kind: ConfigMap" app=meshery
time="2021-03-23T00:21:42Z" level=info msg="Incoming object: eac074af.layer5.io, kind: ConfigMap" app=meshery
time="2021-03-23T00:21:43Z" level=info msg="Incoming object: meshery-operator, kind: Deployment" app=meshery
time="2021-03-23T00:21:43Z" level=info msg="Incoming object: meshery-operator, kind: Deployment" app=meshery
time="2021-03-23T00:21:43Z" level=info msg="Incoming object: meshery-operator, kind: Deployment" app=meshery
time="2021-03-23T00:21:44Z" level=info msg="Operator operation executed" app=meshery
time="2021-03-23T00:21:44Z" level=info msg="Connected to broker at:host.docker.internal:4222" app=meshery
time="2021-03-23T00:21:44Z" level=info msg="Incoming object: eac074af.layer5.io, kind: ConfigMap" app=meshery
time="2021-03-23T00:21:45Z" level=info msg="Meshsync operation executed" app=meshery
time="2021-03-23T00:21:46Z" level=info msg="Incoming object: eac074af.layer5.io, kind: ConfigMap" app=meshery
time="2021-03-23T00:21:48Z" level=info msg="retrieved user: {lcalcote   http://pbs.twimg.com/profile_images/880205475643441152/V_vhfnzb_normal.jpg}"
time="2021-03-23T00:21:48Z" level=info msg="retrieved user: {lcalcote   http://pbs.twimg.com/profile_images/880205475643441152/V_vhfnzb_normal.jpg}"
time="2021-03-23T00:21:48Z" level=info msg="retrieved user: {lcalcote   http://pbs.twimg.com/profile_images/880205475643441152/V_vhfnzb_normal.jpg}"
time="2021-03-23T00:21:48Z" level=info msg="retrieved user: {lcalcote   http://pbs.twimg.com/profile_images/880205475643441152/V_vhfnzb_normal.jpg}"
time="2021-03-23T00:21:48Z" level=info msg="Operator subscription started" app=meshery
time="2021-03-23T00:21:48Z" level=info msg="MeshSync subscription started" app=meshery
time="2021-03-23T00:21:48Z" level=info msg="retrieved user: {lcalcote   http://pbs.twimg.com/profile_images/880205475643441152/V_vhfnzb_normal.jpg}"
time="2021-03-23T00:21:48Z" level=info msg="Incoming object: eac074af.layer5.io, kind: ConfigMap" app=meshery
time="2021-03-23T00:21:50Z" level=info msg="Incoming object: eac074af.layer5.io, kind: ConfigMap" app=meshery
time="2021-03-23T00:21:52Z" level=info msg="Incoming object: eac074af.layer5.io, kind: ConfigMap" app=meshery
time="2021-03-23T00:21:54Z" level=info msg="Incoming object: eac074af.layer5.io, kind: ConfigMap" app=meshery
time="2021-03-23T00:21:56Z" level=info msg="Incoming object: eac074af.layer5.io, kind: ConfigMap" app=meshery
time="2021-03-23T00:21:58Z" level=info msg="Incoming object: eac074af.layer5.io, kind: ConfigMap" app=meshery
time="2021-03-23T00:22:00Z" level=info msg="Incoming object: eac074af.layer5.io, kind: ConfigMap" app=meshery
time="2021-03-23T00:22:02Z" level=info msg="Incoming object: eac074af.layer5.io, kind: ConfigMap" app=meshery
time="2021-03-23T00:22:04Z" level=info msg="Incoming object: grafana-core-7548464597-96s87, kind: Pod" app=meshery
time="2021-03-23T00:22:04Z" level=info msg="Incoming object: jaeger-88f8559f8-gfp4m, kind: Pod" app=meshery
time="2021-03-23T00:22:04Z" level=info msg="Incoming object: prometheus-core-8557bc7d97-s2v2r, kind: Pod" app=meshery
time="2021-03-23T00:22:04Z" level=info msg="Incoming object: traefik-mesh-controller-fc6754485-fnd5f, kind: Pod" app=meshery
time="2021-03-23T00:22:04Z" level=info msg="Incoming object: traefik-mesh-proxy-plg5g, kind: Pod" app=meshery
time="2021-03-23T00:22:04Z" level=info msg="Incoming object: grafana-784c89f4cf-cl7qw, kind: Pod" app=meshery
time="2021-03-23T00:22:04Z" level=info msg="Incoming object: jaeger-7f78b6fb65-fx95f, kind: Pod" app=meshery
time="2021-03-23T00:22:04Z" level=info msg="Incoming object: kiali-dc84967d9-bfcrw, kind: Pod" app=meshery
time="2021-03-23T00:22:04Z" level=info msg="Incoming object: prometheus-7bfddb8dbf-brng6, kind: Pod" app=meshery
time="2021-03-23T00:22:04Z" level=info msg="Incoming object: coredns-5748cf776c-d5dzf, kind: Pod" app=meshery
time="2021-03-23T00:22:04Z" level=info msg="Incoming object: zipkin-7fcd647cf9-6n44d, kind: Pod" app=meshery
time="2021-03-23T00:22:04Z" level=info msg="Incoming object: coredns-5748cf776c-svfzp, kind: Pod" app=meshery
time="2021-03-23T00:22:04Z" level=info msg="Incoming object: etcd-docker-desktop, kind: Pod" app=meshery
time="2021-03-23T00:22:04Z" level=info msg="Incoming object: kube-apiserver-docker-desktop, kind: Pod" app=meshery
time="2021-03-23T00:22:04Z" level=info msg="Incoming object: kube-controller-manager-docker-desktop, kind: Pod" app=meshery
time="2021-03-23T00:22:04Z" level=info msg="Incoming object: kube-proxy-v6npq, kind: Pod" app=meshery
time="2021-03-23T00:22:04Z" level=info msg="Incoming object: kube-scheduler-docker-desktop, kind: Pod" app=meshery
time="2021-03-23T00:22:04Z" level=info msg="Incoming object: storage-provisioner, kind: Pod" app=meshery
time="2021-03-23T00:22:04Z" level=info msg="Incoming object: vpnkit-controller, kind: Pod" app=meshery
time="2021-03-23T00:22:04Z" level=info msg="Incoming object: meshery-74745fb8c9-4bvg2, kind: Pod" app=meshery
time="2021-03-23T00:22:04Z" level=info msg="Incoming object: meshery-istio-5977448cb8-6krpl, kind: Pod" app=meshery
time="2021-03-23T00:22:04Z" level=info msg="Incoming object: meshery-consul-6c77dcd486-2kvdq, kind: Pod" app=meshery
time="2021-03-23T00:22:04Z" level=info msg="Incoming object: meshery-cpx-5bc6f8dddf-2h78b, kind: Pod" app=meshery
time="2021-03-23T00:22:04Z" level=info msg="Incoming object: meshery-broker-0, kind: Pod" app=meshery
time="2021-03-23T00:22:04Z" level=info msg="Incoming object: meshery-kuma-9c4f9879c-rj5zg, kind: Pod" app=meshery
time="2021-03-23T00:22:04Z" level=info msg="Incoming object: meshery-nsm-6bb69cc498-ckjzk, kind: Pod" app=meshery
time="2021-03-23T00:22:04Z" level=info msg="Incoming object: meshery-linkerd-f4c6db56d-mkls6, kind: Pod" app=meshery
time="2021-03-23T00:22:04Z" level=info msg="Incoming object: meshery-meshsync-645d598d4b-89c8w, kind: Pod" app=meshery
time="2021-03-23T00:22:04Z" level=info msg="Incoming object: meshery-operator-549bf67565-htwdn, kind: Pod" app=meshery
time="2021-03-23T00:22:04Z" level=info msg="Incoming object: meshery-osm-f4f7b47c6-dw49f, kind: Pod" app=meshery
time="2021-03-23T00:22:04Z" level=info msg="Incoming object: eac074af.layer5.io, kind: ConfigMap" app=meshery
time="2021-03-23T00:22:06Z" level=info msg="Incoming object: eac074af.layer5.io, kind: ConfigMap" app=meshery
time="2021-03-23T00:22:07Z" level=info msg="Incoming object: docker-desktop, kind: Node" app=meshery
time="2021-03-23T00:22:08Z" level=info msg="Incoming object: eac074af.layer5.io, kind: ConfigMap" app=meshery
time="2021-03-23T00:22:10Z" level=info msg="Incoming object: eac074af.layer5.io, kind: ConfigMap" app=meshery
time="2021-03-23T00:22:12Z" level=info msg="Incoming object: eac074af.layer5.io, kind: ConfigMap" app=meshery
time="2021-03-23T00:22:14Z" level=info msg="Incoming object: eac074af.layer5.io, kind: ConfigMap" app=meshery
time="2021-03-23T00:22:17Z" level=info msg="Incoming object: eac074af.layer5.io, kind: ConfigMap" app=meshery
time="2021-03-23T00:22:19Z" level=info msg="Incoming object: eac074af.layer5.io, kind: ConfigMap" app=meshery
time="2021-03-23T00:22:21Z" level=info msg="Incoming object: eac074af.layer5.io, kind: ConfigMap" app=meshery
time="2021-03-23T00:22:23Z" level=info msg="Incoming object: eac074af.layer5.io, kind: ConfigMap" app=meshery
time="2021-03-23T00:22:25Z" level=info msg="Incoming object: eac074af.layer5.io, kind: ConfigMap" app=meshery
time="2021-03-23T00:22:27Z" level=info msg="Incoming object: eac074af.layer5.io, kind: ConfigMap" app=meshery
time="2021-03-23T00:22:29Z" level=info msg="Incoming object: eac074af.layer5.io, kind: ConfigMap" app=meshery
time="2021-03-23T00:22:31Z" level=info msg="Incoming object: eac074af.layer5.io, kind: ConfigMap" app=meshery
time="2021-03-23T00:22:33Z" level=info msg="Incoming object: eac074af.layer5.io, kind: ConfigMap" app=meshery
time="2021-03-23T00:22:35Z" level=info msg="Incoming object: eac074af.layer5.io, kind: ConfigMap" app=meshery
time="2021-03-23T00:22:37Z" level=info msg="Incoming object: eac074af.layer5.io, kind: ConfigMap" app=meshery
time="2021-03-23T00:22:37Z" level=info msg="Incoming object: docker-desktop, kind: Node" app=meshery
time="2021-03-23T00:22:39Z" level=info msg="Incoming object: eac074af.layer5.io, kind: ConfigMap" app=meshery
time="2021-03-23T00:22:41Z" level=info msg="Incoming object: eac074af.layer5.io, kind: ConfigMap" app=meshery
time="2021-03-23T00:22:43Z" level=info msg="Incoming object: eac074af.layer5.io, kind: ConfigMap" app=meshery
time="2021-03-23T00:22:45Z" level=info msg="Incoming object: eac074af.layer5.io, kind: ConfigMap" app=meshery
time="2021-03-23T00:22:47Z" level=info msg="Incoming object: eac074af.layer5.io, kind: ConfigMap" app=meshery
time="2021-03-23T00:22:49Z" level=info msg="Incoming object: eac074af.layer5.io, kind: ConfigMap" app=meshery
time="2021-03-23T00:22:51Z" level=info msg="Incoming object: eac074af.layer5.io, kind: ConfigMap" app=meshery
time="2021-03-23T00:22:53Z" level=info msg="Incoming object: eac074af.layer5.io, kind: ConfigMap" app=meshery
time="2021-03-23T00:22:54Z" level=info msg="retrieved user: {lcalcote   http://pbs.twimg.com/profile_images/880205475643441152/V_vhfnzb_normal.jpg}"
time="2021-03-23T00:22:54Z" level=info msg="retrieved user: {lcalcote   http://pbs.twimg.com/profile_images/880205475643441152/V_vhfnzb_normal.jpg}"
time="2021-03-23T00:22:54Z" level=info msg="retrieved user: {lcalcote   http://pbs.twimg.com/profile_images/880205475643441152/V_vhfnzb_normal.jpg}"
time="2021-03-23T00:22:54Z" level=info msg="retrieved user: {lcalcote   http://pbs.twimg.com/profile_images/880205475643441152/V_vhfnzb_normal.jpg}"
time="2021-03-23T00:22:54Z" level=info msg="MeshSync subscription started" app=meshery
time="2021-03-23T00:22:54Z" level=info msg="Operator subscription started" app=meshery
time="2021-03-23T00:22:55Z" level=info msg="Incoming object: eac074af.layer5.io, kind: ConfigMap" app=meshery
time="2021-03-23T00:22:57Z" level=info msg="Incoming object: eac074af.layer5.io, kind: ConfigMap" app=meshery
time="2021-03-23T00:22:59Z" level=info msg="Incoming object: eac074af.layer5.io, kind: ConfigMap" app=meshery
time="2021-03-23T00:23:01Z" level=info msg="Incoming object: eac074af.layer5.io, kind: ConfigMap" app=meshery
time="2021-03-23T00:23:03Z" level=info msg="Incoming object: eac074af.layer5.io, kind: ConfigMap" app=meshery
time="2021-03-23T00:23:05Z" level=info msg="Incoming object: eac074af.layer5.io, kind: ConfigMap" app=meshery
time="2021-03-23T00:23:07Z" level=info msg="Incoming object: eac074af.layer5.io, kind: ConfigMap" app=meshery
time="2021-03-23T00:23:07Z" level=info msg="Incoming object: docker-desktop, kind: Node" app=meshery
time="2021-03-23T00:23:09Z" level=info msg="Incoming object: eac074af.layer5.io, kind: ConfigMap" app=meshery
time="2021-03-23T00:23:11Z" level=info msg="Incoming object: eac074af.layer5.io, kind: ConfigMap" app=meshery
time="2021-03-23T00:23:13Z" level=info msg="Incoming object: eac074af.layer5.io, kind: ConfigMap" app=meshery
time="2021-03-23T00:23:13Z" level=info msg="retrieved user: {lcalcote   http://pbs.twimg.com/profile_images/880205475643441152/V_vhfnzb_normal.jpg}"
time="2021-03-23T00:23:13Z" level=info msg="Removing Operator" app=meshery
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x30 pc=0x232107d]

goroutine 4631 [running]:
github.com/layer5io/meshsync/pkg/model.SetID(0xc0010ce780)
    /go/pkg/mod/github.com/layer5io/meshsync@v0.1.17/pkg/model/model_converter.go:92 +0x9d
github.com/layer5io/meshsync/pkg/model.(*Object).BeforeDelete(0xc0010ce780, 0xc00145e090, 0xc0010ce780, 0x7f8ad8335478)
    /go/pkg/mod/github.com/layer5io/meshsync@v0.1.17/pkg/model/model.go:78 +0x2b
gorm.io/gorm/callbacks.BeforeDelete.func1(0x2a4b1a0, 0xc0010ce780, 0xc00145e090, 0x2a4b101)
    /go/pkg/mod/gorm.io/gorm@v1.20.10/callbacks/delete.go:16 +0x73
gorm.io/gorm/callbacks.callMethod(0xc00121bf20, 0xc00104bce0)
    /go/pkg/mod/gorm.io/gorm@v1.20.10/callbacks/callmethod.go:20 +0x2d1
gorm.io/gorm/callbacks.BeforeDelete(0xc00121bf20)
    /go/pkg/mod/gorm.io/gorm@v1.20.10/callbacks/delete.go:14 +0x81
gorm.io/gorm.(*processor).Execute(0xc00017fdc0, 0xc00121bf20)
    /go/pkg/mod/gorm.io/gorm@v1.20.10/callbacks.go:105 +0x224
gorm.io/gorm.(*DB).Delete(0xc0004d4270, 0x2a4b1a0, 0xc0010ce780, 0x0, 0x0, 0x0, 0x0)
    /go/pkg/mod/gorm.io/gorm@v1.20.10/finisher_api.go:350 +0xb8
github.com/layer5io/meshery/internal/graphql/resolver.recordMeshSyncData(0x2cd9c30, 0x7, 0xc0004886d0, 0xc0010ce780, 0x0, 0x0)
    /github.com/layer5io/meshery/internal/graphql/resolver/meshsync.go:85 +0x12d
github.com/layer5io/meshery/internal/graphql/resolver.(*Resolver).cleanEntries(0xc0001fd560, 0x1, 0xc000d06a80, 0x0)
    /github.com/layer5io/meshery/internal/graphql/resolver/operator.go:386 +0x15a
github.com/layer5io/meshery/internal/graphql/resolver.(*Resolver).changeOperatorStatus.func1(0xc0001fd560, 0xc001215b28, 0x8, 0x41b8701, 0xc0005fc6c0)
    /github.com/layer5io/meshery/internal/graphql/resolver/operator.go:50 +0x4c
created by github.com/layer5io/meshery/internal/graphql/resolver.(*Resolver).changeOperatorStatus
    /github.com/layer5io/meshery/internal/graphql/resolver/operator.go:49 +0xa3
leecalcote commented 3 years ago

@kumarabd I assume that this was addressed prior to v0.5.0 release. Is that right? Please update.

kumarabd commented 3 years ago

Yes this was addressed.