Closed barbeau closed 3 years ago
One solution for this is to ignore all unknown JSON properties, which is probably the best solution for production release where OTP will evolve with new fields that we won't know about. From https://github.com/Kotlin/kotlinx.serialization/issues/169, possible solutions
val httpClient = HttpClient {
install(JsonFeature) {
serializer = KotlinxSerializer(KotlinJson { ignoreUnknownKeys = true })
}
}
or:
val httpClient = HttpClient {
Json {
ignoreUnknownKeys = true
}
}
For configuration with Ktor - https://ktor.io/docs/json.html#install_feature.
Specifically, https://ktor.io/docs/json.html#kotlinx gives the example:
install(JsonFeature) {
serializer = KotlinxSerializer(kotlinx.serialization.json.Json {
isLenient = true
ignoreUnknownKeys = true
})
}
...which requires the implementation("io.ktor:ktor-client-serialization:$ktorVersion")
dependency.
This compiles, but when I test it it doesn't have any effect and I get the same error.
https://ktor.io/docs/kotlin-serialization.html#register_converter gives a different example:
install(ContentNegotiation) {
json(Json {
isLenient = true
ignoreUnknownKeys = true
// ...
})
}
However, this doesn't seem to compile for me even when adding implementation "io.ktor:ktor-serialization:$ktor_version"
and import io.ktor.serialization.*
.
EDIT Looks like this ContentNegotiation is deprecated - https://github.com/ktorio/ktor/blob/main/ktor-features/ktor-serialization/jvm/src/io/ktor/serialization/JsonSupport.kt#L48
I'm actually getting a slightly different error after the solution in https://github.com/CUTR-at-USF/opentripplanner-client-library/issues/24#issuecomment-884926484 is added to the release, but it could just be because the trips are being executed in a different order and it's hitting this issue first. It's still doesn't seem like https://github.com/CUTR-at-USF/opentripplanner-client-library/issues/24#issuecomment-884926484 is having any effect.
Exception in thread "main" kotlinx.serialization.json.internal.JsonDecodingException: Unexpected JSON token at offset 8270: Encountered an unknown key
'exit'.
Use 'ignoreUnknownKeys = true' in 'Json {}' builder to ignore unknown keys.
JSON input: .....ection":"NORTHEAST","exit":"1","stayOn":false,"area":false,".....
at kotlinx.serialization.json.internal.JsonExceptionsKt.JsonDecodingException(JsonExceptions.kt:24)
at kotlinx.serialization.json.internal.JsonExceptionsKt.JsonDecodingException(JsonExceptions.kt:32)
at kotlinx.serialization.json.internal.JsonReader.fail(JsonReader.kt:338)
at kotlinx.serialization.json.internal.JsonReader.fail$default(JsonReader.kt:337)
at kotlinx.serialization.json.internal.StreamingJsonDecoder.decodeObjectIndex(StreamingJsonDecoder.kt:144)
at kotlinx.serialization.json.internal.StreamingJsonDecoder.decodeElementIndex(StreamingJsonDecoder.kt:87)
at edu.usf.cutr.otp.plan.model.Step$$serializer.deserialize(Step.kt)
at edu.usf.cutr.otp.plan.model.Step$$serializer.deserialize(Step.kt:23)
at kotlinx.serialization.json.internal.PolymorphicKt.decodeSerializableValuePolymorphic(Polymorphic.kt:63)
at kotlinx.serialization.json.internal.StreamingJsonDecoder.decodeSerializableValue(StreamingJsonDecoder.kt:32)
at kotlinx.serialization.encoding.AbstractDecoder.decodeSerializableValue(AbstractDecoder.kt:43)
at kotlinx.serialization.encoding.AbstractDecoder.decodeSerializableElement(AbstractDecoder.kt:70)
at kotlinx.serialization.encoding.CompositeDecoder$DefaultImpls.decodeSerializableElement$default(Decoding.kt:536)
at kotlinx.serialization.internal.ListLikeSerializer.readElement(CollectionSerializers.kt:80)
at kotlinx.serialization.internal.AbstractCollectionSerializer.readElement$default(CollectionSerializers.kt:51)
at kotlinx.serialization.internal.AbstractCollectionSerializer.merge(CollectionSerializers.kt:36)
at kotlinx.serialization.internal.AbstractCollectionSerializer.deserialize(CollectionSerializers.kt:43)
at kotlinx.serialization.json.internal.PolymorphicKt.decodeSerializableValuePolymorphic(Polymorphic.kt:63)
at kotlinx.serialization.json.internal.StreamingJsonDecoder.decodeSerializableValue(StreamingJsonDecoder.kt:32)
at kotlinx.serialization.encoding.AbstractDecoder.decodeSerializableValue(AbstractDecoder.kt:43)
at kotlinx.serialization.encoding.AbstractDecoder.decodeNullableSerializableElement(AbstractDecoder.kt:79)
at edu.usf.cutr.otp.plan.model.Leg$$serializer.deserialize(Leg.kt)
at edu.usf.cutr.otp.plan.model.Leg$$serializer.deserialize(Leg.kt:23)
at kotlinx.serialization.json.internal.PolymorphicKt.decodeSerializableValuePolymorphic(Polymorphic.kt:63)
at kotlinx.serialization.json.internal.StreamingJsonDecoder.decodeSerializableValue(StreamingJsonDecoder.kt:32)
at kotlinx.serialization.encoding.AbstractDecoder.decodeSerializableValue(AbstractDecoder.kt:43)
at kotlinx.serialization.encoding.AbstractDecoder.decodeSerializableElement(AbstractDecoder.kt:70)
at kotlinx.serialization.encoding.CompositeDecoder$DefaultImpls.decodeSerializableElement$default(Decoding.kt:536)
at kotlinx.serialization.internal.ListLikeSerializer.readElement(CollectionSerializers.kt:80)
at kotlinx.serialization.internal.AbstractCollectionSerializer.readElement$default(CollectionSerializers.kt:51)
at kotlinx.serialization.internal.AbstractCollectionSerializer.merge(CollectionSerializers.kt:36)
at kotlinx.serialization.internal.AbstractCollectionSerializer.deserialize(CollectionSerializers.kt:43)
at kotlinx.serialization.json.internal.PolymorphicKt.decodeSerializableValuePolymorphic(Polymorphic.kt:63)
at kotlinx.serialization.json.internal.StreamingJsonDecoder.decodeSerializableValue(StreamingJsonDecoder.kt:32)
at kotlinx.serialization.encoding.AbstractDecoder.decodeSerializableValue(AbstractDecoder.kt:43)
at kotlinx.serialization.encoding.AbstractDecoder.decodeNullableSerializableElement(AbstractDecoder.kt:79)
at edu.usf.cutr.otp.plan.model.Itinerary$$serializer.deserialize(Itinerary.kt)
at edu.usf.cutr.otp.plan.model.Itinerary$$serializer.deserialize(Itinerary.kt:23)
at kotlinx.serialization.json.internal.PolymorphicKt.decodeSerializableValuePolymorphic(Polymorphic.kt:63)
at kotlinx.serialization.json.internal.StreamingJsonDecoder.decodeSerializableValue(StreamingJsonDecoder.kt:32)
at kotlinx.serialization.encoding.AbstractDecoder.decodeSerializableValue(AbstractDecoder.kt:43)
at kotlinx.serialization.encoding.AbstractDecoder.decodeSerializableElement(AbstractDecoder.kt:70)
at kotlinx.serialization.encoding.CompositeDecoder$DefaultImpls.decodeSerializableElement$default(Decoding.kt:536)
at kotlinx.serialization.internal.ListLikeSerializer.readElement(CollectionSerializers.kt:80)
at kotlinx.serialization.internal.AbstractCollectionSerializer.readElement$default(CollectionSerializers.kt:51)
at kotlinx.serialization.internal.AbstractCollectionSerializer.merge(CollectionSerializers.kt:36)
at kotlinx.serialization.internal.AbstractCollectionSerializer.deserialize(CollectionSerializers.kt:43)
at kotlinx.serialization.json.internal.PolymorphicKt.decodeSerializableValuePolymorphic(Polymorphic.kt:63)
at kotlinx.serialization.json.internal.StreamingJsonDecoder.decodeSerializableValue(StreamingJsonDecoder.kt:32)
at kotlinx.serialization.encoding.AbstractDecoder.decodeSerializableValue(AbstractDecoder.kt:43)
at kotlinx.serialization.encoding.AbstractDecoder.decodeNullableSerializableElement(AbstractDecoder.kt:79)
at edu.usf.cutr.otp.plan.model.Plan$$serializer.deserialize(Plan.kt)
at edu.usf.cutr.otp.plan.model.Plan$$serializer.deserialize(Plan.kt:24)
at kotlinx.serialization.json.internal.PolymorphicKt.decodeSerializableValuePolymorphic(Polymorphic.kt:63)
at kotlinx.serialization.json.internal.StreamingJsonDecoder.decodeSerializableValue(StreamingJsonDecoder.kt:32)
at kotlinx.serialization.encoding.AbstractDecoder.decodeSerializableValue(AbstractDecoder.kt:43)
at kotlinx.serialization.encoding.AbstractDecoder.decodeNullableSerializableElement(AbstractDecoder.kt:79)
at edu.usf.cutr.otp.plan.model.Planner$$serializer.deserialize(Planner.kt)
at edu.usf.cutr.otp.plan.model.Planner$$serializer.deserialize(Planner.kt:24)
at kotlinx.serialization.json.internal.PolymorphicKt.decodeSerializableValuePolymorphic(Polymorphic.kt:63)
at kotlinx.serialization.json.internal.StreamingJsonDecoder.decodeSerializableValue(StreamingJsonDecoder.kt:32)
at kotlinx.serialization.json.Json.decodeFromString(Json.kt:85)
at edu.usf.cutr.otp.plan.api.PlanApi$getPlan$1.invokeSuspend(PlanApi.kt:79)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at io.ktor.util.pipeline.SuspendFunctionGun.resumeRootWith(SuspendFunctionGun.kt:191)
at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:147)
at io.ktor.util.pipeline.SuspendFunctionGun.access$loop(SuspendFunctionGun.kt:15)
at io.ktor.util.pipeline.SuspendFunctionGun$continuation$1.resumeWith(SuspendFunctionGun.kt:93)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:46)
at io.ktor.util.pipeline.SuspendFunctionGun.resumeRootWith(SuspendFunctionGun.kt:191)
at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:147)
at io.ktor.util.pipeline.SuspendFunctionGun.access$loop(SuspendFunctionGun.kt:15)
at io.ktor.util.pipeline.SuspendFunctionGun$continuation$1.resumeWith(SuspendFunctionGun.kt:93)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:46)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:750)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665)
E:\Git Projects\gnss-ride-hailing-analyzer>
Looks like this is the underlying cause of the first error message in the initial post - the message key name is different when the trip planner is unavailable: https://github.com/opentripplanner/OpenTripPlanner/issues/3581
alerts
is another unknown key, in Step and Leg. Looks like this is the OTP server model class - https://github.com/opentripplanner/OpenTripPlanner/blob/77d24b64973ff2873f14c0cf374333c23031ac76/src/main/java/org/opentripplanner/api/model/ApiAlert.java.
For leg mode values see: https://github.com/opentripplanner/OpenTripPlanner/issues/3248#issuecomment-890547824
EDIT - Actually, I see that we just have these are strings for now
Describe the bug When trying to parse an OTP response on a large number of requests, some of them fail with:
To Reproduce Steps to reproduce the behavior:
Expected behavior Parse message without errors
Actual behavior One of the responses has JSON included that we aren't handling/expecting