Strict JSON encountered unknown key

[rodrigohsb]

I tried some basic tests and I came across an unexpected scenario

This scenario works perfectly:

@Serializable
data class SimpleApiResponse (val id: String, val label: String)

{
    "id": "123456789,
    "label": "abc"
}

But when added another key into the json above, like this:

{
    "id": "123456789,
    "label": "abc",
    "anotherKey": "def"
}

I got this error

kotlinx.serialization.SerializationException: Strict JSON encountered unknown key: anotherKey
                      at kotlinx.serialization.json.JSON$JsonInput.readElement(JSON.kt:294)
                      at co.stone.kyc.datasource.webservice.PendingQuestionsResponse$$serializer.load(Unknown Source:18)
                      at co.stone.kyc.datasource.webservice.PendingQuestionsResponse$$serializer.load(payload.kt:13)
                      at kotlinx.serialization.KInput.read(Serialization.kt:209)
                      at kotlinx.serialization.json.JSON.parse(JSON.kt:43)
                      at kotlinx.serialization.json.JSON$Companion.parse(JSON.kt:53)
                      at co.stone.kyc.datasource.KYCDefaultDataSource$loadData$1.onResponse(KYCDefaultDataSource.kt:132)
                      at okhttp3.RealCall$AsyncCall.execute(RealCall.java:153)
                      at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
                      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1162)
                      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:636)
                      at java.lang.Thread.run(Thread.java:764)

Is there any way to escape unmapped keys? Any annotation?

[JakeWharton]

JSON.nonstrict

[rodrigohsb]

Tks! Any reason for nonstrict not be used as default?

[pdvrieze]

Because you easily noticed the problem that it was stricter than desired but the opposite you would most likely not have noticed.

[qwwdfsad]

Any reason for nonstrict not be used as default

Yes, by default, we opt-in for maximal safety, including type checks and runtime checks. If frontend sends to backend slightly malformed JSON, it's better to know about it in advance, not when your system failed because field name has a typo and some value was ignored. In most cases, such check prevents errors on integration testing stage.

Though sometimes, you have to ignore missing fields, then you can use nonstrict mode

[raydenvoldeskine]

Could not find any documentation on how to use nonstrict mode if serializer is created with the @Serializable, not manually. Can someone help here?

[sandwwraith]

@fracturizer nonstrict is the property of Json format, not model. It should work with arbitrary serializer.

[raydenvoldeskine]

In other words, I should use Json.nonstrict.parse() instead of Json.parse(). Found it. Thanks.

[mochadwi]

Any reason for nonstrict not be used as default

Yes, by default, we opt-in for maximal safety, including type checks and runtime checks. If frontend sends to backend slightly malformed JSON, it's better to know about it in advance, not when your system failed because field name has a typo and some value was ignored. In most cases, such check prevents errors on integration testing stage.

Though sometimes, you have to ignore missing fields, then you can use nonstrict mode

How to use this for retrofit? Does this only apply for custom / manual JSON serialization?

@qwwdfsad @sandwwraith

EDITED For those doesn't using custom Json de/serialization, use JsonConfiguration(strictMode = false) on retrofit converter factory

E.g:

// your retrofit builder
.addConverterFactory(
    Json(
        JsonConfiguration(strictMode = false)
    ).asConverterFactory(MediaType.get("application/json"))
)

[sandwwraith]

@mochadwi I think your question is more related to the repository of retrofit converter (https://github.com/JakeWharton/retrofit2-kotlinx-serialization-converter). But yes, I think your solution is correct

[ngortheone]

Is it possible to save all encountered unknown keys to some Map<String, JsonObj> value? e.g.

data class MyData(val x: Int, val unknown: Map<String,JsonObj>)

[sandwwraith]

@ngortheone Currently this is not possible. You can deserialize json to JsonObject though which is merely a map.

[tellisnz-shift]

@sandwwraith - is there a recommended way of doing schema migration?

for example, say I add a field in a new version of a data class and send that to a client using the old version. I'd want clients to be able to deserialize, use the fields they know about, and then serialize the object to pass back to the server, without losing the new field if the server sent it. I guess sort of proxy the client data object to the JsonObject and serialize the JsonObject, if that makes sense?

because i guess if i just happily use non strict, on serialisation to send back to the server the new field would be dropped?

[sandwwraith]

@tellisnz-shift Well, if you try not to lose any fields client don't know about, then JsonObject is your way. However, I think it is pretty OK for a client to lose fields it doesn't know about and send back old JSON

[joaquini]

Latest versions of both Kotlin Serialization and JakeWharton's Converter should do:

Json {
    ignoreUnknownKeys = true
}.asConverterFactory(contentType)

[FerhatBahceci]

Using the Spring Webclient awaitBody() will trigger Kotlin serialization for the respective @Serializable receiver response class.

Unknown keys exception is thrown on the receiver side since some of the fields are missing.

Is it possible to set ignoreUnknownKeys = true somehow for these receiver classes other than implementing custom serializers?

SenderA(id,name,age) --> GET awaitBody<(id,name)>() throws unknown keys.

[FerhatBahceci]

val json = Json { ignoreUnknownKeys = true isLenient = true }

val strategies = ExchangeStrategies
    .builder()
    .codecs { clientDefaultCodecsConfigurer ->
        run {
            clientDefaultCodecsConfigurer.defaultCodecs()
                .kotlinSerializationJsonDecoder(KotlinSerializationJsonDecoder(json))
            clientDefaultCodecsConfigurer.defaultCodecs()
                .kotlinSerializationJsonEncoder(KotlinSerializationJsonEncoder(json))

        }
    }.build()

return WebClient
    .builder()
    .exchangeStrategies(strategies)
    .baseUrl(baseUrl!!)
    .build()

[jramalv]

I set ignoreUnknownKeys on my retrofit configuration

` val contentType = "application/json".toMediaType() val json = Json { ignoreUnknownKeys = true}

val webservice by lazy {
    Retrofit.Builder()
        .client(
            OkHttpClient.Builder()
                .connectTimeout(1000, TimeUnit.SECONDS)
                .readTimeout(1000, TimeUnit.SECONDS)
                .addInterceptor(BenefitsInteceptor(context))
                .build())
        .baseUrl(BuildConfig.BASE_URL)
        .addConverterFactory(json.asConverterFactory(contentType))
        .addCallAdapterFactory(RxJava2CallAdapterFactory.create())
        .build().create(BenefitsApiService::class.java)
}`

When i get the errorbody on the response i get the same message "Strict JSON encountered unknown key: anotherKey". I display the error to the user with a toast

`class ToastUtils {

companion object{
    var json = Json { ignoreUnknownKeys = true }

    fun showErrorToastFromJson(context: Context, response:String){
        Toast.makeText(context,json.decodeFromString<ErrorResponse>(
            JSONObject(response).toString())
            .message, Toast.LENGTH_SHORT).show()

    }
}

}`

But, i need to indicate the same line of code " var json = Json { ignoreUnknownKeys = true }" which is already set on my retrofit configuration. How can i avoid to declare the same configuration two times?

[choutman]

For future reference, as I turned up here finding a solution to this problem. Just setting the ignoreUnknownKeys property did not suffice. Additionally, I also needed to set explicitNulls to false. Thus, the configuration that worked for me was:

Retrofit.Builder()
...
.addConverterFactory(Json {
    ignoreUnknownKeys = true
    explicitNulls = false
}.asConverterFactory("application/json".toMediaType()))