mschoch
commented
1 year ago Here was the original panic I was able to find by adding the code to recover and log it locally:
interface conversion: interface {} is nil, not string
goroutine 24 [running]:
runtime/debug.Stack()
/Users/mschoch/Documents/goroot/src/runtime/debug/stack.go:24 +0x64
github.com/axiomhq/axiom-grafana/pkg/plugin.(*Datasource).QueryData.func1()
/Users/mschoch/clients/axiom-grafana/pkg/plugin/datasource.go:114 +0x114
panic({0x7daec0, 0x4000420b10})
/Users/mschoch/Documents/goroot/src/runtime/panic.go:884 +0x1f4
github.com/grafana/grafana-plugin-sdk-go/data.(*stringVector).Append(0x8e797b?, {0x0?, 0x0?})
/Users/mschoch/go/pkg/mod/github.com/grafana/grafana-plugin-sdk-go@v0.161.0/data/vector.gen.go:754 +0xf4
github.com/grafana/grafana-plugin-sdk-go/data.(*Frame).AppendRow(...)
/Users/mschoch/go/pkg/mod/github.com/grafana/grafana-plugin-sdk-go@v0.161.0/data/frame.go:103
github.com/axiomhq/axiom-grafana/pkg/plugin.buildFrameMatches(0x400011c900)
/Users/mschoch/clients/axiom-grafana/pkg/plugin/datasource.go:324 +0x818
github.com/axiomhq/axiom-grafana/pkg/plugin.(*Datasource).query(0x1?, {0xa04748, 0x4000301440}, {0x40000aec80?, 0x0?}, {0x1, {0x40002be1f8, 0x18}, 0x40000aec80, 0x0, ...}, ...)
/Users/mschoch/clients/axiom-grafana/pkg/plugin/datasource.go:176 +0x218
github.com/axiomhq/axiom-grafana/pkg/plugin.(*Datasource).QueryData(0x400009a210, {0xa04748, 0x4000301440}, 0x4000433720)
/Users/mschoch/clients/axiom-grafana/pkg/plugin/datasource.go:123 +0x19c
github.com/grafana/grafana-plugin-sdk-go/internal/automanagement.(*Manager).QueryData(0xa04748?, {0xa04748, 0x4000301440}, 0x4000433720)
/Users/mschoch/go/pkg/mod/github.com/grafana/grafana-plugin-sdk-go@v0.161.0/internal/automanagement/manager.go:33 +0x184
github.com/grafana/grafana-plugin-sdk-go/backend.(*dataSDKAdapter).QueryData(0x4000431d40, {0xa04748, 0x4000301440}, 0x0?)
/Users/mschoch/go/pkg/mod/github.com/grafana/grafana-plugin-sdk-go@v0.161.0/backend/data_adapter.go:49 +0x7c
github.com/grafana/grafana-plugin-sdk-go/backend/grpcplugin.(*dataGRPCServer).QueryData(0x8ea137?, {0xa04748?, 0x4000301440?}, 0x4000301410?)
/Users/mschoch/go/pkg/mod/github.com/grafana/grafana-plugin-sdk-go@v0.161.0/backend/grpcplugin/grpc_data.go:47 +0x30
github.com/grafana/grafana-plugin-sdk-go/genproto/pluginv2._Data_QueryData_Handler.func1({0xa04748, 0x4000301440}, {0x8648c0?, 0x40004335e0})
/Users/mschoch/go/pkg/mod/github.com/grafana/grafana-plugin-sdk-go@v0.161.0/genproto/pluginv2/backend_grpc.pb.go:195 +0x74
github.com/grpc-ecosystem/go-grpc-prometheus.(*ServerMetrics).UnaryServerInterceptor.func1({0xa04748, 0x4000301440}, {0x8648c0, 0x40004335e0}, 0x40005bd5f8?, 0x400000ccd8)
/Users/mschoch/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-prometheus@v1.2.0/server_metrics.go:107 +0x74
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func2.1({0xa04748?, 0x4000301440?}, {0x8648c0?, 0x40004335e0?})
/Users/mschoch/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.4.0/chain.go:48 +0x50
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryServerInterceptor.func1({0xa04748, 0x4000301350}, {0x8648c0, 0x40004335e0}, 0x40000ea3e0, 0x40000aeb80)
/Users/mschoch/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc@v0.40.0/interceptor.go:342 +0x3d0
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func2({0xa04748, 0x4000301350}, {0x8648c0, 0x40004335e0}, 0x40000ea3e0, 0x5f187c?)
/Users/mschoch/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.4.0/chain.go:53 +0x158
github.com/grafana/grafana-plugin-sdk-go/genproto/pluginv2._Data_QueryData_Handler({0x7b66c0?, 0x40005920d0}, {0xa04748, 0x4000301350}, 0x40003a2c40, 0x400056a8d0)
/Users/mschoch/go/pkg/mod/github.com/grafana/grafana-plugin-sdk-go@v0.161.0/genproto/pluginv2/backend_grpc.pb.go:197 +0x12c
google.golang.org/grpc.(*Server).processUnaryRPC(0x4000590000, {0xa09798, 0x400059e1a0}, 0x400037f320, 0x400056afc0, 0xfc3620, 0x0)
/Users/mschoch/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:1345 +0xc28
google.golang.org/grpc.(*Server).handleStream(0x4000590000, {0xa09798, 0x400059e1a0}, 0x400037f320, 0x0)
/Users/mschoch/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:1722 +0x80c
google.golang.org/grpc.(*Server).serveStreams.func1.2()
/Users/mschoch/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:966 +0x84
created by google.golang.org/grpc.(*Server).serveStreams.func1
/Users/mschoch/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:964 +0x278
When converting data from the legacy result format into an intermediate format, field names containing dots were not properly escaped. This resulted in the values not being found, and that caused us to insert a nil where a string was expected. This causes the plugin to panic.
Two fixes:
The data-structure used for the intermediate structure is now map[string]string which means the zero-value of anything not found will be the empty string (not nil). This means if we fail to find the value for any other reason in the future, it will be "" and not crash.
I have fixed the field name handling to properly handle dot when it occurs inside a field name by escaping it the same way.
NOTE: I also decided to leave the panic handler in this time. I don't know if they're going to object or not, but I have to add it back myself every time, and without it debugging is significantly harder.