Parsing errors due to inconsistent data types in JSON

[ivanahuckova]

Malformed or inconsistent JSON can cause parsing errors. When values have inconsistent data types, the parser may fail—for instance, if a certain key is expected to be a number but occasionally appears as a string. This type mismatch can lead to type conversion issues and cause the parser to break.

Related to this.

[yesoreyeram]

Hi Ivana.. in such cases, the quick workaround is to set the field type as string. ( If we know that the field is of such mixed types ). Even though this doesn't solve the problem, it will help to skip the error and not ideal in all the situations. I know the team is now having limited bandwidth. Once I return to work, will see anything we can gracefully handle such situations automatically in the parser. Let's keep the ticket open for investigation. Also if possible let's add some more context such as what type of parser being used and what's the field/column types were used. It may be common with all the parsing options but it is good to document the information we have.

Thanks, Sriram

[josiahg]

Also if possible let's add some more context such as what type of parser being used and what's the field/column types were used.

This URL is the one that brought this to our attention. It works with the frontend parser in Infinity, but fails with the backend parser. The user needs to use the backend parser so that they can configure alerting.

On my local instance, attempting to parse that URL with the backend parser results in:

ERROR[08-21|13:43:32] panic triggered                          logger=plugin.yesoreyeram-infinity-datasource error="interface conversion: interface {} is *string, not *float64" stack="goroutine 75 [running]:\nruntime/debug.Stack()\n\t/usr/local/go/src/runtime/debug/stack.go:24 +0x64\ngithub.com/grafana/grafana-plugin-sdk-go/backend.handlePanic({0x106311580, 0x140012c0d80})\n\t/go/pkg/mod/github.com/grafana/grafana-plugin-sdk-go@v0.241.0/backend/serve.go:103 +0x24\ngithub.com/grafana/grafana-plugin-sdk-go/backend.defaultGRPCMiddlewares.WithRecoveryHandler.func4.1({0x10?, 0xf?}, {0x106311580?, 0x140012c0d80?})\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.1.0/interceptors/recovery/options.go:36 +0x34\ngithub.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors/recovery.recoverFrom({0x10650a978?, 0x140008f6090?}, {0x106311580?, 0x140012c0d80?}, 0x140009bb628?)\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.1.0/interceptors/recovery/interceptors.go:54 +0xe4\ngithub.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors/recovery.UnaryServerInterceptor.func1.1()\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.1.0/interceptors/recovery/interceptors.go:30 +0x64\npanic({0x106311580?, 0x140012c0d80?})\n\t/usr/local/go/src/runtime/panic.go:770 +0x124\ngithub.com/grafana/grafana-plugin-sdk-go/data.(*nullableFloat64Vector).Set(0x10627a660?, 0x1400093d0f0?, {0x1062435c0?, 0x140008fcb30?})\n\t/go/pkg/mod/github.com/grafana/grafana-plugin-sdk-go@v0.241.0/data/nullable_vector.gen.go:849 +0xd4\ngithub.com/grafana/grafana-plugin-sdk-go/data.(*Field).Set(...)\n\t/go/pkg/mod/github.com/grafana/grafana-plugin-sdk-go@v0.241.0/data/field.go:145\ngithub.com/grafana/infinity-libs/lib/go/gframer.sliceToFrame({0x106f96788, 0x1}, {0x140011e2008, 0x2eb, 0x3ff}, {{0x106f96788, 0x1}, {0x0, 0x0}, {0x1071a3e80, ...}, ...})\n\t/go/pkg/mod/github.com/grafana/infinity-libs/lib/go/gframer@v1.0.0/gframer.go:192 +0x9e4\ngithub.com/grafana/infinity-libs/lib/go/gframer.ToDataFrame({0x106263c00, 0x14001244840}, {{0x106f96788, 0x1}, {0x0, 0x0}, {0x1071a3e80, 0x0, 0x0}, {0x1071a3e80, ...}})\n\t/go/pkg/mod/github.com/grafana/infinity-libs/lib/go/gframer@v1.0.0/gframer.go:33 +0xec\ngithub.com/grafana/infinity-libs/lib/go/jsonframer.getFrameFromResponseString({0x14000dec000, 0x75a7f}, {{0x0, 0x0}, {0x106f96788, 0x1}, {0x0, 0x0}, {0x1071a3e80, 0x0, ...}, ...})\n\t/go/pkg/mod/github.com/grafana/infinity-libs/lib/go/jsonframer@v1.1.4/jsonframer.go:248 +0x62c\ngithub.com/grafana/infinity-libs/lib/go/jsonframer.ToFrame({0x14000dec000, 0x75a7f}, {{0x0, 0x0}, {0x106f96788, 0x1}, {0x0, 0x0}, {0x1071a3e80, 0x0, ...}, ...})\n\t/go/pkg/mod/github.com/grafana/infinity-libs/lib/go/jsonframer@v1.1.4/jsonframer.go:151 +0xe4\ngithub.com/grafana/grafana-infinity-datasource/pkg/infinity.GetJSONBackendResponse({_, _}, {_, _}, {{0x106f96788, 0x1}, {0x1400074937c, 0x4}, {0x14000749358, 0x5}, ...})\n\t/drone/src/pkg/infinity/jsonBackend.go:37 +0x684\ngithub.com/grafana/grafana-infinity-datasource/pkg/infinity.GetFrameForURLSourcesWithPostProcessing({_, _}, {{0x106f96788, 0x1}, {0x1400074937c, 0x4}, {0x14000749358, 0x5}, {0x14000749377, 0x3}, ...}, ...)\n\t/drone/src/pkg/infinity/remote.go:168 +0x544\ngithub.com/grafana/grafana-infinity-datasource/pkg/infinity.GetFrameForURLSources({_, _}, {{0x106f96788, 0x1}, {0x1400074937c, 0x4}, {0x14000749358, 0x5}, {0x14000749377, 0x3}, ...}, ...)\n\t/drone/src/pkg/infinity/remote.go:25 +0x1fc\ngithub.com/grafana/grafana-infinity-datasource/pkg/pluginhost.QueryDataQuery({_, _}, {{0x106f96788, 0x1}, {0x1400074937c, 0x4}, {0x14000749358, 0x5}, {0x14000749377, 0x3}, ...}, ...)\n\t/drone/src/pkg/pluginhost/handler_querydata.go:139 +0x19ac\ngithub.com/grafana/grafana-infinity-datasource/pkg/pluginhost.(*DataSource).QueryData(0x140008ac0f8, {0x10650a978, 0x140008f6330}, 0x140008ea600)\n\t/drone/src/pkg/pluginhost/handler_querydata.go:49 +0x45c\ngithub.com/grafana/grafana-plugin-sdk-go/internal/automanagement.(*Manager).QueryData(0x10650a978?, {0x10650a978, 0x140008f6330}, 0x140008ea600)\n\t/go/pkg/mod/github.com/grafana/grafana-plugin-sdk-go@v0.241.0/internal/automanagement/manager.go:33 +0x1c4\ngithub.com/grafana/grafana-plugin-sdk-go/backend.(*dataSDKAdapter).QueryData.func1({0x10650a978, 0x140008f62d0})\n\t/go/pkg/mod/github.com/grafana/grafana-plugin-sdk-go@v0.241.0/backend/data_adapter.go:30 +0x94\ngithub.com/grafana/grafana-plugin-sdk-go/backend.wrapHandler.errorWrapper.func1({0x10650a978, 0x140008f62d0})\n\t/go/pkg/mod/github.com/grafana/grafana-plugin-sdk-go@v0.241.0/backend/adapter_utils.go:49 +0x2c\ngithub.com/grafana/grafana-plugin-sdk-go/backend.wrapHandler.logWrapper.func2({0x10650a978, 0x140008f62d0})\n\t/go/pkg/mod/github.com/grafana/grafana-plugin-sdk-go@v0.241.0/backend/adapter_utils.go:123 +0x54\ngithub.com/grafana/grafana-plugin-sdk-go/backend.metricWrapper.func2({0x10650a978, 0x140008f62d0})\n\t/go/pkg/mod/github.com/grafana/grafana-plugin-sdk-go@v0.241.0/backend/adapter_utils.go:76 +0x8c\ngithub.com/grafana/grafana-plugin-sdk-go/backend.wrapHandler.tracingWrapper.func3({0x10650a978, 0x140008f62a0})\n\t/go/pkg/mod/github.com/grafana/grafana-plugin-sdk-go@v0.241.0/backend/adapter_utils.go:105 +0x66c\ngithub.com/grafana/grafana-plugin-sdk-go/backend.wrapHandler({0x10650a978?, 0x140008f6120?}, {0x1, {0x14000047d60, 0x1f}, {0x14000749322, 0x5}, 0x140007b35c0, 0x0, 0x14000a82dd0, ...}, ...)\n\t/go/pkg/mod/github.com/grafana/grafana-plugin-sdk-go@v0.241.0/backend/adapter_utils.go:33 +0x164\ngithub.com/grafana/grafana-plugin-sdk-go/backend.(*dataSDKAdapter).QueryData(0x1400055c320, {0x10650a978?, 0x140008f6090?}, 0x14000812af0)\n\t/go/pkg/mod/github.com/grafana/grafana-plugin-sdk-go@v0.241.0/backend/data_adapter.go:27 +0x124\ngithub.com/grafana/grafana-plugin-sdk-go/backend/grpcplugin.(*dataGRPCServer).QueryData(0x1070d6340?, {0x10650a978?, 0x140008f6090?}, 0x1400080d698?)\n\t/go/pkg/mod/github.com/grafana/grafana-plugin-sdk-go@v0.241.0/backend/grpcplugin/grpc_data.go:48 +0x30\ngithub.com/grafana/grafana-plugin-sdk-go/genproto/pluginv2._Data_QueryData_Handler.func1({0x10650a978?, 0x140008f6090?}, {0x106407b00?, 0x14000812af0?})\n\t/go/pkg/mod/github.com/grafana/grafana-plugin-sdk-go@v0.241.0/genproto/pluginv2/backend_grpc.pb.go:205 +0xd0\ngithub.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors/recovery.UnaryServerInterceptor.func1({0x10650a978?, 0x140008f6090?}, {0x106407b00?, 0x14000812af0?}, 0x105c06f7d?, 0x5?)\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.1.0/interceptors/recovery/interceptors.go:34 +0x88\ngoogle.golang.org/grpc.getChainUnaryHandler.func1({0x10650a978, 0x140008f6090}, {0x106407b00, 0x14000812af0})\n\t/go/pkg/mod/google.golang.org/grpc@v1.64.1/server.go:1196 +0xa0\ngithub.com/grafana/grafana-plugin-sdk-go/backend.defaultGRPCMiddlewares.(*ServerMetrics).UnaryServerInterceptor.UnaryServerInterceptor.func9({0x10650a978, 0x140008f6090}, {0x106407b00, 0x14000812af0}, 0x140005972c0?, 0x140007b3580)\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.1.0/interceptors/server.go:22 +0x1f0\ngoogle.golang.org/grpc.NewServer.chainUnaryServerInterceptors.chainUnaryInterceptors.func1({0x10650a978, 0x140008f6090}, {0x106407b00, 0x14000812af0}, 0x140005972c0, 0x1064836a0?)\n\t/go/pkg/mod/google.golang.org/grpc@v1.64.1/server.go:1187 +0x88\ngithub.com/grafana/grafana-plugin-sdk-go/genproto/pluginv2._Data_QueryData_Handler({0x1062df2e0, 0x1400055c7e0}, {0x10650a978, 0x140008f6090}, 0x140008ea400, 0x14000596700)\n\t/go/pkg/mod/github.com/grafana/grafana-plugin-sdk-go@v0.241.0/genproto/pluginv2/backend_grpc.pb.go:207 +0x148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC(0x140005c2000, {0x10650a978, 0x14000a95f20}, {0x106514a60, 0x1400044c300}, 0x140004fde60, 0x14000592f60, 0x1070d62e0, 0x0)\n\t/go/pkg/mod/google.golang.org/grpc@v1.64.1/server.go:1379 +0xb58\ngoogle.golang.org/grpc.(*Server).handleStream(0x140005c2000, {0x106514a60, 0x1400044c300}, 0x140004fde60)\n\t/go/pkg/mod/google.golang.org/grpc@v1.64.1/server.go:1790 +0xb20\ngoogle.golang.org/grpc.(*Server).serveStreams.func2.1()\n\t/go/pkg/mod/google.golang.org/grpc@v1.64.1/server.go:1029 +0x8c\ncreated by google.golang.org/grpc.(*Server).serveStreams.func2 in goroutine 48\n\t/go/pkg/mod/google.golang.org/grpc@v1.64.1/server.go:1040 +0x13c\n"
ERROR[08-21|13:43:32] InternalError                            logger=context userId=1 orgId=1 uname=admin error="[plugin.downstreamError] client: failed to query data: Failed to query data: rpc error: code = Unknown desc = panic triggered: interface conversion: interface {} is *string, not *float64" remote_addr=[::1] traceID=

Also note that the only error in the frontend in this case is "the plugin encountered an error" - which could be improved to help a user attempting to troubleshoot this, without access to the backend debug logs.

[ivanahuckova]

I have done investigation and found the root cause of the issue. The problem when querying https://aviationweather.gov/api/data/metar?format=json is with field visib, that has sometimes number values (e.g. 5) and sometimes string values (e.g. 10+). We have a function in backend getFieldTypeFromSlice that returns a field type based on value. But it only checks a first non-null value. And therefore if the first field is number, we'll set the data frame field type to number and incoming string later will cause the panic in sliceToFrame when adding values to frame. This is not an issue on frontend cause typescript is more flexible about incorrect types.

The workaround would be just using default (frontent) mode for now. If backend mode is required, I would suggest to just select columns that you are interested in - e.g.

Unfortunately, there is no way to get information on what field is causing an issue right now. We should improve that and return more user friendly error. To properly fix this, we should have a better logic to get field type from more values, but here the downside would be performance burden, as looping over a lot of values is going to take much more time than just 1 value. Maybe we could check first X non-null values?