search

vitessio / vitess

Vitess is a database clustering system for horizontal scaling of MySQL.
http://vitess.io
Apache License 2.0
18.19k stars 2.06k forks source link

Runtime panic in vtctld when aborting workflow #4789

Open derekperkins opened 5 years ago

derekperkins commented 5 years ago

image

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x5e7e26]

goroutine 172927 [running]:
context.propagateCancel(0x0, 0x0, 0x125c3a0, 0xc0005ec2c0)
    /usr/local/go/src/context/context.go:243 +0x26
context.WithCancel(0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/context/context.go:232 +0x9b
vitess.io/vitess/vendor/golang.org/x/net/context.WithCancel(0x0, 0x0, 0x14, 0x0, 0x0)
    /vt/src/vitess.io/vitess/vendor/golang.org/x/net/context/go17.go:33 +0x35
vitess.io/vitess/vendor/google.golang.org/grpc.newClientStream(0x0, 0x0, 0x1c9da60, 0xc000240c00, 0x10f7f2b, 0x14, 0x0, 0x0, 0x0, 0x0, ...)
    /vt/src/vitess.io/vitess/vendor/google.golang.org/grpc/stream.go:177 +0x15b
vitess.io/vitess/vendor/google.golang.org/grpc.invoke(0x0, 0x0, 0x10f7f2b, 0x14, 0x1063d40, 0xc0004b25f0, 0x1063e40, 0xc0003beed0, 0xc000240c00, 0x0, ...)
    /vt/src/vitess.io/vitess/vendor/google.golang.org/grpc/call.go:66 +0x9c
vitess.io/vitess/vendor/google.golang.org/grpc.(*ClientConn).Invoke(0xc000240c00, 0x0, 0x0, 0x10f7f2b, 0x14, 0x1063d40, 0xc0004b25f0, 0x1063e40, 0xc0003beed0, 0x0, ...)
    /vt/src/vitess.io/vitess/vendor/google.golang.org/grpc/call.go:37 +0x1b3
vitess.io/vitess/vendor/google.golang.org/grpc.Invoke(0x0, 0x0, 0x10f7f2b, 0x14, 0x1063d40, 0xc0004b25f0, 0x1063e40, 0xc0003beed0, 0xc000240c00, 0x0, ...)
    /vt/src/vitess.io/vitess/vendor/google.golang.org/grpc/call.go:60 +0xc1
vitess.io/vitess/vendor/github.com/coreos/etcd/etcdserver/etcdserverpb.(*kVClient).Txn(0xc00000c900, 0x0, 0x0, 0xc0004b25f0, 0x0, 0x0, 0x0, 0x3f, 0x203000, 0x203000)
    /vt/src/vitess.io/vitess/vendor/github.com/coreos/etcd/etcdserver/etcdserverpb/rpc.pb.go:3080 +0xd2
vitess.io/vitess/vendor/github.com/coreos/etcd/clientv3.(*retryWriteKVClient).Txn.func1(0x0, 0x0, 0x40bb8f, 0xc0005ec080)
    /vt/src/vitess.io/vitess/vendor/github.com/coreos/etcd/clientv3/retry.go:121 +0x80
vitess.io/vitess/vendor/github.com/coreos/etcd/clientv3.(*Client).newRetryWrapper.func1(0x0, 0x0, 0xc0005ec080, 0x0, 0xc00039c230)
    /vt/src/vitess.io/vitess/vendor/github.com/coreos/etcd/clientv3/retry.go:31 +0x1b4
vitess.io/vitess/vendor/github.com/coreos/etcd/clientv3.(*retryWriteKVClient).Txn(0xc0002550e0, 0x0, 0x0, 0xc0004b25f0, 0x0, 0x0, 0x0, 0x0, 0xc00008ae00, 0x48)
    /vt/src/vitess.io/vitess/vendor/github.com/coreos/etcd/clientv3/retry.go:120 +0xee
vitess.io/vitess/vendor/github.com/coreos/etcd/clientv3.(*retryWriteKVClient).Txn.func1(0x0, 0x0, 0xc0029f9990, 0x40c418)
    /vt/src/vitess.io/vitess/vendor/github.com/coreos/etcd/clientv3/retry.go:121 +0x80
vitess.io/vitess/vendor/github.com/coreos/etcd/clientv3.(*Client).newAuthRetryWrapper.func1(0x0, 0x0, 0xc0005ec040, 0x50, 0xff28a0)
    /vt/src/vitess.io/vitess/vendor/github.com/coreos/etcd/clientv3/retry.go:61 +0x47
vitess.io/vitess/vendor/github.com/coreos/etcd/clientv3.(*retryWriteKVClient).Txn(0xc000255100, 0x0, 0x0, 0xc0004b25f0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /vt/src/vitess.io/vitess/vendor/github.com/coreos/etcd/clientv3/retry.go:120 +0xee
vitess.io/vitess/vendor/github.com/coreos/etcd/clientv3.(*txn).commit(0xc000f765b0, 0x114d9e0, 0xc000f765c8, 0x0)
    /vt/src/vitess.io/vitess/vendor/github.com/coreos/etcd/clientv3/txn.go:159 +0xf7
vitess.io/vitess/vendor/github.com/coreos/etcd/clientv3.(*txn).Commit(0xc000f765b0, 0x0, 0x0, 0x0)
    /vt/src/vitess.io/vitess/vendor/github.com/coreos/etcd/clientv3/txn.go:139 +0x83
vitess.io/vitess/go/vt/topo/etcd2topo.(*Server).Update(0xc0002551a0, 0x0, 0x0, 0xc00045a000, 0x37, 0xc000286000, 0x4fa, 0x4fa, 0x125ab00, 0xc0001a4ba0, ...)
    /vt/src/vitess.io/vitess/go/vt/topo/etcd2topo/file.go:57 +0x4a4
vitess.io/vitess/go/vt/topo.(*StatsConn).Update(0xc0002551c0, 0x0, 0x0, 0xc00045a000, 0x37, 0xc000286000, 0x4fa, 0x4fa, 0x125ab00, 0xc0001a4ba0, ...)
    /vt/src/vitess.io/vitess/go/vt/topo/stats_conn.go:83 +0x17b
vitess.io/vitess/go/vt/topo.(*Server).SaveWorkflow(0xc0002fc600, 0x0, 0x0, 0xc0007aa0e0, 0x3, 0x0)
    /vt/src/vitess.io/vitess/go/vt/topo/workflow.go:113 +0x126
vitess.io/vitess/go/vt/workflow.(*Manager).executeWorkflowRun(0xc0002fc640, 0x126b2e0, 0xc0005885c0, 0xc000737580)
    /vt/src/vitess.io/vitess/go/vt/workflow/manager.go:406 +0x44c
derekperkins commented 5 years ago

On further investigation, this looks very similar to #4518.