search

CUTR-at-USF / opentripplanner-client-library

A Kotlin Multiplatform Mobile client library for the OpenTripPlanner project (http://www.opentripplanner.org/) server (v2 and higher)
Apache License 2.0
2 stars 1 forks source link

Fail to parse message on unknown JSON keys #24

Closed barbeau closed 3 years ago

barbeau commented 3 years ago

Describe the bug When trying to parse an OTP response on a large number of requests, some of them fail with:

Exception in thread "main" kotlinx.serialization.json.internal.JsonDecodingException: Unexpected JSON token at offset 326: Encountered an unknown key
'msg'.
Use 'ignoreUnknownKeys = true' in 'Json {}' builder to ignore unknown keys.
JSON input: .....able. Please try again later."}}
        at kotlinx.serialization.json.internal.JsonExceptionsKt.JsonDecodingException(JsonExceptions.kt:24)
        at kotlinx.serialization.json.internal.JsonExceptionsKt.JsonDecodingException(JsonExceptions.kt:32)
        at kotlinx.serialization.json.internal.JsonReader.fail(JsonReader.kt:338)
        at kotlinx.serialization.json.internal.JsonReader.fail$default(JsonReader.kt:337)
        at kotlinx.serialization.json.internal.StreamingJsonDecoder.decodeObjectIndex(StreamingJsonDecoder.kt:144)
        at kotlinx.serialization.json.internal.StreamingJsonDecoder.decodeElementIndex(StreamingJsonDecoder.kt:87)
        at edu.usf.cutr.otp.plan.model.error.Error$$serializer.deserialize(Error.kt)
        at edu.usf.cutr.otp.plan.model.error.Error$$serializer.deserialize(Error.kt:23)
        at kotlinx.serialization.json.internal.PolymorphicKt.decodeSerializableValuePolymorphic(Polymorphic.kt:63)
        at kotlinx.serialization.json.internal.StreamingJsonDecoder.decodeSerializableValue(StreamingJsonDecoder.kt:32)
        at kotlinx.serialization.encoding.AbstractDecoder.decodeSerializableValue(AbstractDecoder.kt:43)
        at kotlinx.serialization.encoding.AbstractDecoder.decodeNullableSerializableElement(AbstractDecoder.kt:79)
        at edu.usf.cutr.otp.plan.model.Planner$$serializer.deserialize(Planner.kt)
        at edu.usf.cutr.otp.plan.model.Planner$$serializer.deserialize(Planner.kt:24)
        at kotlinx.serialization.json.internal.PolymorphicKt.decodeSerializableValuePolymorphic(Polymorphic.kt:63)
        at kotlinx.serialization.json.internal.StreamingJsonDecoder.decodeSerializableValue(StreamingJsonDecoder.kt:32)
        at kotlinx.serialization.json.Json.decodeFromString(Json.kt:85)
        at edu.usf.cutr.otp.plan.api.PlanApi$getPlan$1.invokeSuspend(PlanApi.kt:59)
        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
        at io.ktor.util.pipeline.SuspendFunctionGun.resumeRootWith(SuspendFunctionGun.kt:191)
        at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:147)
        at io.ktor.util.pipeline.SuspendFunctionGun.access$loop(SuspendFunctionGun.kt:15)
        at io.ktor.util.pipeline.SuspendFunctionGun$continuation$1.resumeWith(SuspendFunctionGun.kt:93)
        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:46)
        at io.ktor.util.pipeline.SuspendFunctionGun.resumeRootWith(SuspendFunctionGun.kt:191)
        at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:147)
        at io.ktor.util.pipeline.SuspendFunctionGun.access$loop(SuspendFunctionGun.kt:15)
        at io.ktor.util.pipeline.SuspendFunctionGun$continuation$1.resumeWith(SuspendFunctionGun.kt:93)
        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:46)
        at io.ktor.util.pipeline.SuspendFunctionGun.resumeRootWith(SuspendFunctionGun.kt:191)
        at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:147)
        at io.ktor.util.pipeline.SuspendFunctionGun.access$loop(SuspendFunctionGun.kt:15)
        at io.ktor.util.pipeline.SuspendFunctionGun$continuation$1.resumeWith(SuspendFunctionGun.kt:93)
        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:46)
        at io.ktor.util.pipeline.SuspendFunctionGun.resumeRootWith(SuspendFunctionGun.kt:191)
        at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:147)
        at io.ktor.util.pipeline.SuspendFunctionGun.access$loop(SuspendFunctionGun.kt:15)
        at io.ktor.util.pipeline.SuspendFunctionGun$continuation$1.resumeWith(SuspendFunctionGun.kt:93)
        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:46)
        at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
        at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571)
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:750)
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678)
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665)
        Suppressed: kotlinx.serialization.json.internal.JsonDecodingException: Unexpected JSON token at offset 329: Encountered an unknown key 'msg'.
Use 'ignoreUnknownKeys = true' in 'Json {}' builder to ignore unknown keys.
JSON input: .....able. Please try again later."}}
                ... 44 more
        Suppressed: kotlinx.serialization.json.internal.JsonDecodingException: Unexpected JSON token at offset 329: Encountered an unknown key 'msg'.
Use 'ignoreUnknownKeys = true' in 'Json {}' builder to ignore unknown keys.
JSON input: .....able. Please try again later."}}
                ... 44 more
        Suppressed: kotlinx.serialization.json.internal.JsonDecodingException: Unexpected JSON token at offset 328: Encountered an unknown key 'msg'.
Use 'ignoreUnknownKeys = true' in 'Json {}' builder to ignore unknown keys.
JSON input: .....able. Please try again later."}}
                ... 44 more

To Reproduce Steps to reproduce the behavior:

  1. Use https://github.com/CUTR-at-USF/gnss-ride-hailing-analyzer/blob/a40b93358f0b935bb56e514f56b8948ac9fefde2/README.md on a large number of records

Expected behavior Parse message without errors

Actual behavior One of the responses has JSON included that we aren't handling/expecting

barbeau commented 3 years ago

One solution for this is to ignore all unknown JSON properties, which is probably the best solution for production release where OTP will evolve with new fields that we won't know about. From https://github.com/Kotlin/kotlinx.serialization/issues/169, possible solutions

                val httpClient = HttpClient {
                    install(JsonFeature) {
                        serializer = KotlinxSerializer(KotlinJson { ignoreUnknownKeys = true })
                    }
                }

or:

                val httpClient = HttpClient {
                    Json {
                        ignoreUnknownKeys = true
                    }
                }
barbeau commented 3 years ago

For configuration with Ktor - https://ktor.io/docs/json.html#install_feature.

barbeau commented 3 years ago

Specifically, https://ktor.io/docs/json.html#kotlinx gives the example:

                    install(JsonFeature) {
                        serializer = KotlinxSerializer(kotlinx.serialization.json.Json {
                            isLenient = true
                            ignoreUnknownKeys = true
                        })
                    }

...which requires the implementation("io.ktor:ktor-client-serialization:$ktorVersion") dependency.

This compiles, but when I test it it doesn't have any effect and I get the same error.

barbeau commented 3 years ago

https://ktor.io/docs/kotlin-serialization.html#register_converter gives a different example:

                    install(ContentNegotiation) {
                        json(Json {
                            isLenient = true
                            ignoreUnknownKeys = true
                            // ...
                        })
                    }

However, this doesn't seem to compile for me even when adding implementation "io.ktor:ktor-serialization:$ktor_version" and import io.ktor.serialization.*.

EDIT Looks like this ContentNegotiation is deprecated - https://github.com/ktorio/ktor/blob/main/ktor-features/ktor-serialization/jvm/src/io/ktor/serialization/JsonSupport.kt#L48

barbeau commented 3 years ago

I'm actually getting a slightly different error after the solution in https://github.com/CUTR-at-USF/opentripplanner-client-library/issues/24#issuecomment-884926484 is added to the release, but it could just be because the trips are being executed in a different order and it's hitting this issue first. It's still doesn't seem like https://github.com/CUTR-at-USF/opentripplanner-client-library/issues/24#issuecomment-884926484 is having any effect.

Exception in thread "main" kotlinx.serialization.json.internal.JsonDecodingException: Unexpected JSON token at offset 8270: Encountered an unknown key
 'exit'.
Use 'ignoreUnknownKeys = true' in 'Json {}' builder to ignore unknown keys.
JSON input: .....ection":"NORTHEAST","exit":"1","stayOn":false,"area":false,".....
        at kotlinx.serialization.json.internal.JsonExceptionsKt.JsonDecodingException(JsonExceptions.kt:24)
        at kotlinx.serialization.json.internal.JsonExceptionsKt.JsonDecodingException(JsonExceptions.kt:32)
        at kotlinx.serialization.json.internal.JsonReader.fail(JsonReader.kt:338)
        at kotlinx.serialization.json.internal.JsonReader.fail$default(JsonReader.kt:337)
        at kotlinx.serialization.json.internal.StreamingJsonDecoder.decodeObjectIndex(StreamingJsonDecoder.kt:144)
        at kotlinx.serialization.json.internal.StreamingJsonDecoder.decodeElementIndex(StreamingJsonDecoder.kt:87)
        at edu.usf.cutr.otp.plan.model.Step$$serializer.deserialize(Step.kt)
        at edu.usf.cutr.otp.plan.model.Step$$serializer.deserialize(Step.kt:23)
        at kotlinx.serialization.json.internal.PolymorphicKt.decodeSerializableValuePolymorphic(Polymorphic.kt:63)
        at kotlinx.serialization.json.internal.StreamingJsonDecoder.decodeSerializableValue(StreamingJsonDecoder.kt:32)
        at kotlinx.serialization.encoding.AbstractDecoder.decodeSerializableValue(AbstractDecoder.kt:43)
        at kotlinx.serialization.encoding.AbstractDecoder.decodeSerializableElement(AbstractDecoder.kt:70)
        at kotlinx.serialization.encoding.CompositeDecoder$DefaultImpls.decodeSerializableElement$default(Decoding.kt:536)
        at kotlinx.serialization.internal.ListLikeSerializer.readElement(CollectionSerializers.kt:80)
        at kotlinx.serialization.internal.AbstractCollectionSerializer.readElement$default(CollectionSerializers.kt:51)
        at kotlinx.serialization.internal.AbstractCollectionSerializer.merge(CollectionSerializers.kt:36)
        at kotlinx.serialization.internal.AbstractCollectionSerializer.deserialize(CollectionSerializers.kt:43)
        at kotlinx.serialization.json.internal.PolymorphicKt.decodeSerializableValuePolymorphic(Polymorphic.kt:63)
        at kotlinx.serialization.json.internal.StreamingJsonDecoder.decodeSerializableValue(StreamingJsonDecoder.kt:32)
        at kotlinx.serialization.encoding.AbstractDecoder.decodeSerializableValue(AbstractDecoder.kt:43)
        at kotlinx.serialization.encoding.AbstractDecoder.decodeNullableSerializableElement(AbstractDecoder.kt:79)
        at edu.usf.cutr.otp.plan.model.Leg$$serializer.deserialize(Leg.kt)
        at edu.usf.cutr.otp.plan.model.Leg$$serializer.deserialize(Leg.kt:23)
        at kotlinx.serialization.json.internal.PolymorphicKt.decodeSerializableValuePolymorphic(Polymorphic.kt:63)
        at kotlinx.serialization.json.internal.StreamingJsonDecoder.decodeSerializableValue(StreamingJsonDecoder.kt:32)
        at kotlinx.serialization.encoding.AbstractDecoder.decodeSerializableValue(AbstractDecoder.kt:43)
        at kotlinx.serialization.encoding.AbstractDecoder.decodeSerializableElement(AbstractDecoder.kt:70)
        at kotlinx.serialization.encoding.CompositeDecoder$DefaultImpls.decodeSerializableElement$default(Decoding.kt:536)
        at kotlinx.serialization.internal.ListLikeSerializer.readElement(CollectionSerializers.kt:80)
        at kotlinx.serialization.internal.AbstractCollectionSerializer.readElement$default(CollectionSerializers.kt:51)
        at kotlinx.serialization.internal.AbstractCollectionSerializer.merge(CollectionSerializers.kt:36)
        at kotlinx.serialization.internal.AbstractCollectionSerializer.deserialize(CollectionSerializers.kt:43)
        at kotlinx.serialization.json.internal.PolymorphicKt.decodeSerializableValuePolymorphic(Polymorphic.kt:63)
        at kotlinx.serialization.json.internal.StreamingJsonDecoder.decodeSerializableValue(StreamingJsonDecoder.kt:32)
        at kotlinx.serialization.encoding.AbstractDecoder.decodeSerializableValue(AbstractDecoder.kt:43)
        at kotlinx.serialization.encoding.AbstractDecoder.decodeNullableSerializableElement(AbstractDecoder.kt:79)
        at edu.usf.cutr.otp.plan.model.Itinerary$$serializer.deserialize(Itinerary.kt)
        at edu.usf.cutr.otp.plan.model.Itinerary$$serializer.deserialize(Itinerary.kt:23)
        at kotlinx.serialization.json.internal.PolymorphicKt.decodeSerializableValuePolymorphic(Polymorphic.kt:63)
        at kotlinx.serialization.json.internal.StreamingJsonDecoder.decodeSerializableValue(StreamingJsonDecoder.kt:32)
        at kotlinx.serialization.encoding.AbstractDecoder.decodeSerializableValue(AbstractDecoder.kt:43)
        at kotlinx.serialization.encoding.AbstractDecoder.decodeSerializableElement(AbstractDecoder.kt:70)
        at kotlinx.serialization.encoding.CompositeDecoder$DefaultImpls.decodeSerializableElement$default(Decoding.kt:536)
        at kotlinx.serialization.internal.ListLikeSerializer.readElement(CollectionSerializers.kt:80)
        at kotlinx.serialization.internal.AbstractCollectionSerializer.readElement$default(CollectionSerializers.kt:51)
        at kotlinx.serialization.internal.AbstractCollectionSerializer.merge(CollectionSerializers.kt:36)
        at kotlinx.serialization.internal.AbstractCollectionSerializer.deserialize(CollectionSerializers.kt:43)
        at kotlinx.serialization.json.internal.PolymorphicKt.decodeSerializableValuePolymorphic(Polymorphic.kt:63)
        at kotlinx.serialization.json.internal.StreamingJsonDecoder.decodeSerializableValue(StreamingJsonDecoder.kt:32)
        at kotlinx.serialization.encoding.AbstractDecoder.decodeSerializableValue(AbstractDecoder.kt:43)
        at kotlinx.serialization.encoding.AbstractDecoder.decodeNullableSerializableElement(AbstractDecoder.kt:79)
        at edu.usf.cutr.otp.plan.model.Plan$$serializer.deserialize(Plan.kt)
        at edu.usf.cutr.otp.plan.model.Plan$$serializer.deserialize(Plan.kt:24)
        at kotlinx.serialization.json.internal.PolymorphicKt.decodeSerializableValuePolymorphic(Polymorphic.kt:63)
        at kotlinx.serialization.json.internal.StreamingJsonDecoder.decodeSerializableValue(StreamingJsonDecoder.kt:32)
        at kotlinx.serialization.encoding.AbstractDecoder.decodeSerializableValue(AbstractDecoder.kt:43)
        at kotlinx.serialization.encoding.AbstractDecoder.decodeNullableSerializableElement(AbstractDecoder.kt:79)
        at edu.usf.cutr.otp.plan.model.Planner$$serializer.deserialize(Planner.kt)
        at edu.usf.cutr.otp.plan.model.Planner$$serializer.deserialize(Planner.kt:24)
        at kotlinx.serialization.json.internal.PolymorphicKt.decodeSerializableValuePolymorphic(Polymorphic.kt:63)
        at kotlinx.serialization.json.internal.StreamingJsonDecoder.decodeSerializableValue(StreamingJsonDecoder.kt:32)
        at kotlinx.serialization.json.Json.decodeFromString(Json.kt:85)
        at edu.usf.cutr.otp.plan.api.PlanApi$getPlan$1.invokeSuspend(PlanApi.kt:79)
        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
        at io.ktor.util.pipeline.SuspendFunctionGun.resumeRootWith(SuspendFunctionGun.kt:191)
        at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:147)
        at io.ktor.util.pipeline.SuspendFunctionGun.access$loop(SuspendFunctionGun.kt:15)
        at io.ktor.util.pipeline.SuspendFunctionGun$continuation$1.resumeWith(SuspendFunctionGun.kt:93)
        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:46)
        at io.ktor.util.pipeline.SuspendFunctionGun.resumeRootWith(SuspendFunctionGun.kt:191)
        at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:147)
        at io.ktor.util.pipeline.SuspendFunctionGun.access$loop(SuspendFunctionGun.kt:15)
        at io.ktor.util.pipeline.SuspendFunctionGun$continuation$1.resumeWith(SuspendFunctionGun.kt:93)
        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:46)
        at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
        at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571)
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:750)
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678)
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665)

E:\Git Projects\gnss-ride-hailing-analyzer>
barbeau commented 3 years ago

Looks like this is the underlying cause of the first error message in the initial post - the message key name is different when the trip planner is unavailable: https://github.com/opentripplanner/OpenTripPlanner/issues/3581

barbeau commented 3 years ago

alerts is another unknown key, in Step and Leg. Looks like this is the OTP server model class - https://github.com/opentripplanner/OpenTripPlanner/blob/77d24b64973ff2873f14c0cf374333c23031ac76/src/main/java/org/opentripplanner/api/model/ApiAlert.java.

barbeau commented 3 years ago

For leg mode values see: https://github.com/opentripplanner/OpenTripPlanner/issues/3248#issuecomment-890547824

EDIT - Actually, I see that we just have these are strings for now