Open BondAnthony opened 5 years ago
I know when using etcd as a backing datastore a panic can occur when wirey is unable to communicate to etcd. Wirey should handle the timeout correctly and retry after a period of time.
panic: Unexpected error from context packet: context deadline exceeded goroutine 1 [running]: github.com/coreos/etcd/Godeps/_workspace/src/google.golang.org/grpc/transport.ContextErr(0xa48240, 0xd05620, 0xd05620, 0x0, 0xc42002e088) /home/fntlnz/go/src/github.com/coreos/etcd/Godeps/_workspace/src/google.golang.org/grpc/transport/transport.go:468 +0x2ae github.com/coreos/etcd/Godeps/_workspace/src/google.golang.org/grpc/transport.(*Stream).Header(0xc42024c1c0, 0x0, 0x0, 0x1d) /home/fntlnz/go/src/github.com/coreos/etcd/Godeps/_workspace/src/google.golang.org/grpc/transport/transport.go:223 +0x175 github.com/coreos/etcd/Godeps/_workspace/src/google.golang.org/grpc.recvResponse(0xa4dcc0, 0xd058a8, 0x0, 0x0, 0xa4f780, 0xc42017dfe0, 0x101, 0x0, 0x0, 0x0, ...) /home/fntlnz/go/src/github.com/coreos/etcd/Godeps/_workspace/src/google.golang.org/grpc/call.go:54 +0x32 github.com/coreos/etcd/Godeps/_workspace/src/google.golang.org/grpc.Invoke(0x7f41f890e050, 0xc420219620, 0x9e5494, 0x16, 0x984b80, 0xc4202337c0, 0x9915a0, 0xc4202560c0, 0xc420198090, 0x0, ...) /home/fntlnz/go/src/github.com/coreos/etcd/Godeps/_workspace/src/google.golang.org/grpc/call.go:181 +0x4d2 github.com/coreos/etcd/etcdserver/etcdserverpb.(*kVClient).Range(0xc42000eae0, 0x7f41f890e050, 0xc420219620, 0xc4202337c0, 0x0, 0x0, 0x0, 0xc42021c9a0, 0xc4201414e0, 0x75f51d) /home/fntlnz/go/src/github.com/coreos/etcd/etcdserver/etcdserverpb/rpc.pb.go:1217 +0xd2 github.com/coreos/etcd/clientv3.(*kv).do(0xc420256090, 0x7f41f890e050, 0xc420219620, 0x1, 0xc4202541f0, 0xa, 0x10, 0xc420254200, 0xa, 0xa, ...) /home/fntlnz/go/src/github.com/coreos/etcd/clientv3/kv.go:138 +0x68c github.com/coreos/etcd/clientv3.(*kv).Get(0xc420256090, 0x7f41f890e050, 0xc420219620, 0xc4202541e0, 0xa, 0xc42000eae8, 0x1, 0x1, 0xcec040, 0x100000001, ...) /home/fntlnz/go/src/github.com/coreos/etcd/clientv3/kv.go:88 +0xfc github.com/influxdata/wirey/backend.(*EtcdBackend).GetPeers(0xc42000e510, 0x9d9f93, 0x3, 0xc42022fd01, 0x40, 0x4, 0x0, 0x0) /home/fntlnz/go/src/github.com/influxdata/wirey/backend/etcd.go:52 +0x267 github.com/influxdata/wirey/backend.(*Interface).Connect(0xc4200d7480, 0xc42000e510, 0x9d9f93) /home/fntlnz/go/src/github.com/influxdata/wirey/backend/plumber.go:190 +0x10b main.glob..func1(0xcde9e0, 0xc42006a660, 0x0, 0x6) /home/fntlnz/go/src/github.com/influxdata/wirey/cmd/wirey/root.go:57 +0x43c github.com/spf13/cobra.(*Command).execute(0xcde9e0, 0xc42001e080, 0x6, 0x6, 0xcde9e0, 0xc42001e080) /home/fntlnz/go/src/github.com/spf13/cobra/command.go:766 +0x2c1 github.com/spf13/cobra.(*Command).ExecuteC(0xcde9e0, 0xc42000e488, 0x0, 0x1) /home/fntlnz/go/src/github.com/spf13/cobra/command.go:852 +0x30a github.com/spf13/cobra.(*Command).Execute(0xcde9e0, 0xc420074058, 0x0) /home/fntlnz/go/src/github.com/spf13/cobra/command.go:800 +0x2b main.Execute() /home/fntlnz/go/src/github.com/influxdata/wirey/cmd/wirey/root.go:99 +0x2d main.main() /home/fntlnz/go/src/github.com/influxdata/wirey/cmd/wirey/main.go:4 +0x20
I know when using etcd as a backing datastore a panic can occur when wirey is unable to communicate to etcd. Wirey should handle the timeout correctly and retry after a period of time.