Exploration failing immediately upon request

[sebastian]

I am trying to access the latest API endpoint. All requests fail with:

fail: Microsoft.AspNetCore.Server.Kestrel[13]
      Connection id "0HM1JN64UTNFD", Request id "0HM1JN64UTNFD:00000001": An unhandled exception was thrown by the application.
System.Text.Json.JsonException: The JSON value could not be converted to System.Nullable`1[System.Boolean]. Path: $[7].tables[0].columns[1].isolated | LineNumber: 0 | BytePositionInLine: 33767.
 ---> System.InvalidOperationException: Cannot get the value of a token type 'String' as a boolean.
   at System.Text.Json.Utf8JsonReader.GetBoolean()
   at System.Text.Json.JsonPropertyInfoNullable`2.OnRead(ReadStack& state, Utf8JsonReader& reader)
   at System.Text.Json.JsonSerializer.ReadCore(JsonSerializerOptions options, Utf8JsonReader& reader, ReadStack& readStack)
   --- End of inner exception stack trace ---
   at System.Text.Json.ThrowHelper.ReThrowWithPath(ReadStack& readStack, Utf8JsonReader& reader, Exception ex)
   at System.Text.Json.JsonSerializer.ReadCore(JsonSerializerOptions options, Utf8JsonReader& reader, ReadStack& readStack)
   at System.Text.Json.JsonSerializer.ReadCore(JsonReaderState& readerState, Boolean isFinalBlock, ReadOnlySpan`1 buffer, JsonSerializerOptions options, ReadStack& readStack)
   at System.Text.Json.JsonSerializer.ReadAsync[TValue](Stream utf8Json, Type returnType, JsonSerializerOptions options, CancellationToken cancellationToken)
   at Aircloak.JsonApi.JsonApiClient.ParseJson[T](HttpResponseMessage httpResponse, JsonSerializerOptions options, CancellationToken cancellationToken) in /build/src/aircloak/JsonApi/JsonApiClient.cs:line 364
   at Aircloak.JsonApi.JsonApiClient.GetDataSources(Uri apiUrl, CancellationToken cancellationToken) in /build/src/aircloak/JsonApi/JsonApiClient.cs:line 72
   at Explorer.Api.ContextBuilder.Build(Uri apiUrl, String dataSource, String table, IEnumerable`1 columns) in /build/src/explorer.api/Exploration/Setup/ContextBuilder.cs:line 45
   at Explorer.Api.Controllers.ExploreController.Explore(ExploreParams data, ExplorationLauncher launcher, IAircloakAuthenticationProvider authProvider, ContextBuilder contextBuilder, AircloakConnectionBuilder connectionBuilder) in /build/src/explorer.api/Controllers/ExploreController.cs:line 61
   at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.TaskOfIActionResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeActionMethodAsync>g__Awaited|12_0(ControllerActionInvoker invoker, ValueTask`1 actionResultValueTask)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeNextActionFilterAsync>g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeInnerFilterAsync>g__Awaited|13_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeFilterPipelineAsync>g__Awaited|19_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Logged|17_1(ResourceInvoker invoker)
   at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)
   at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
   at Sentry.AspNetCore.SentryMiddleware.InvokeAsync(HttpContext context)
   at Sentry.AspNetCore.SentryMiddleware.InvokeAsync(HttpContext context)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)

This happens for every single request, and results in a 500 exception. Why isn't this reported to sentry?

[dandanlen]

Looks to me like it might due to a difference in api version on the air you are using compared to what we test against?

There's an issue parsing the reply from the /datasources endpoint. Specifically, the isolating specifier: it expects a json boolean value but is getting a string instead.

As to why this doesn't show in Sentry... I have no idea, I'll look into it.

[sebastian]

The value as it is today is what you can expect going forward (at least for the current version). Seems like it's a multi-value field. It can take any of the following values (see here and here for more details):

• true
• false
• "failed"
• "pending"
• "unknown_column"

[dandanlen]

Ok, I'll fix this and will log a separate issue for the missing sentry event.

[dandanlen]

Should the explorer assume a field is Isolating until proven otherwise? Or the reverse?

[sebastian]

Isolating until proven otherwise

[AndreiBozantan]

It can take any of the following values (see here and here for more details):

• true
• false
• "failed"
• "pending"
• "unknown_column"

Would it make sense to modify this, so that all values will have the same type?

• "yes"
• "no"
• "failed"
• "pending"
• "unknown_column"

[sebastian]

This will require significant changes in cloak, so this will not happen.

Internally it's treated as:

• cache lookup for the column succeeded and the result was a boolean true or false
• cache lookup for the column failed and you got an error condition description which is one of "failed", "pending", "unknown_column".

I think that is correct as such. Arguably bunching all these different outcomes into a single value without indicating whether it is a success or error condition is awkward.