Plugin falls over when one endpoint does not respond.

[MatthewHink]

We saw this at Wrigley yesterday. Prod repro is consistent.

if one EGauge (modbus TCP device) is not configured correctly (or does not respond), they all bomb since the plugin falls over.

I was hoping to have a simplified repro of this today but failed.

I suspect, but don't know, that we need the plugin, synse, and a client hitting synse to reproduce.

goroutine 554 [running]:
github.com/vapor-ware/synse-modbus-ip-plugin/vendor/github.com/vapor-ware/synse-sdk/sdk.(*Reading).encode(0x0, 0xc000229c18)
        /go/src/github.com/vapor-ware/synse-modbus-ip-plugin/vendor/github.com/vapor-ware/synse-sdk/sdk/models.go:50 +0x4a
github.com/vapor-ware/synse-modbus-ip-plugin/vendor/github.com/vapor-ware/synse-sdk/sdk.(*server).ReadCached(0xc00015e210, 0xc00000b540, 0xa2a060, 0xc00053c670, 0xc00015e210, 0x75b6eb)
        /go/src/github.com/vapor-ware/synse-modbus-ip-plugin/vendor/github.com/vapor-ware/synse-sdk/sdk/server.go:380 +0x138
github.com/vapor-ware/synse-modbus-ip-plugin/vendor/github.com/vapor-ware/synse-server-grpc/go._Plugin_ReadCached_Handler(0x965200, 0xc00015e210, 0xa29ac0, 0xc0006840c0, 0xdf8b30, 0xc000285710)
        /go/src/github.com/vapor-ware/synse-modbus-ip-plugin/vendor/github.com/vapor-ware/synse-server-grpc/go/synse.pb.go:1693 +0x109
github.com/vapor-ware/synse-modbus-ip-plugin/vendor/google.golang.org/grpc.(*Server).processStreamingRPC(0xc0004bad80, 0xa2ade0, 0xc000498a80, 0xc000354100, 0xc0004dfaa0, 0xdd0760, 0x0, 0x0, 0x0)
        /go/src/github.com/vapor-ware/synse-modbus-ip-plugin/vendor/google.golang.org/grpc/server.go:1170 +0xac7
github.com/vapor-ware/synse-modbus-ip-plugin/vendor/google.golang.org/grpc.(*Server).handleStream(0xc0004bad80, 0xa2ade0, 0xc000498a80, 0xc000354100, 0x0)
        /go/src/github.com/vapor-ware/synse-modbus-ip-plugin/vendor/google.golang.org/grpc/server.go:1249 +0xcfa
github.com/vapor-ware/synse-modbus-ip-plugin/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc000468130, 0xc0004bad80, 0xa2ade0, 0xc000498a80, 0xc000354100)
        /go/src/github.com/vapor-ware/synse-modbus-ip-plugin/vendor/google.golang.org/grpc/server.go:685 +0x9f
created by github.com/vapor-ware/synse-modbus-ip-plugin/vendor/google.golang.org/grpc.(*Server).serveStreams.func1
        /go/src/github.com/vapor-ware/synse-modbus-ip-plugin/vendor/google.golang.org/grpc/server.go:683 +0xa1

[MatthewHink]

I also feel like we should be able to hit this with just the plugin, just haven't made that happen yet.

[edaniszewski]

do we have any additional logs captured? it may be good to see whats going on prior to this error to help narrow things down a bit.

just looking at this trace, it seems like the actual error is happening when a reading is being encoded to its grpc model for transport in the ReadCached call. specifically when the unit of a reading is being encoded. this makes me thing that the Unit may be nil, but I'm not totally sure how that would happen right now since the Unit field of a reading is not a pointer, so we should have a reference to it, even if it is just the zero value?

https://github.com/vapor-ware/synse-sdk/blob/d90486afd9d7a0639db2f7c60e325975e81440cb/sdk/models.go#L21-L22

that said, thats just based on an initial trawl through and I'm still waking up. i'll continue to investigate/trace through and see if I come up with anything/an idea of how to reproduce locally

[MatthewHink]

It was happening during a bulk read. Digging for more logs, but not sure I have them. We should be able to misconfigure a VEM-150 and get another repro.

[MatthewHink]

I'm not seeing this happen with modbus plugin 2.0.1. I have a vsm controller endpoint configured but the vsm controller is not stood up and therefore not responding. Other endpoints give me readings.