How to handle optional/default values with a more advanced serializer?

[JoaaoVerona]

To the point: I need to take a JSON input like { "number": { "innerNumber": 4 } }, and parse it into a top-level number property (like the input were { "number": 4 }.

Therefore, I implemented the serializer like this:

object TestSerializer : KSerializer<Int?> {
    override val descriptor = PrimitiveDescriptor("Test", PrimitiveKind.INT)

    override fun deserialize(decoder: Decoder) =
        (decoder as JsonInput).decodeJson().jsonObject.getPrimitiveOrNull("innerNumber")?.intOrNull

    override fun serialize(encoder: Encoder, value: Int?) =
        (encoder as JsonOutput).encodeJson(json { "innerNumber" to value })
}

... along with my data holder:

@Serializable
data class TestData(
    @Serializable(with = TestSerializer::class)
    val number: Int? = -1
)

Notice the default value of -1.

If I feed it with a valid JSON input, it works flawlessly:

fun main() {
    val input = """{ "number": { "innerNumber": 324 } } """
    val output = Json(JsonConfiguration.Stable).parse(TestData.serializer(), input)

    println(output) // Prints TestData(number = 324)
}

However, let's suppose that I get a malformed JSON as input, like { "number": { } }. In this case, I get TestData(number = null) -- because it identified that the number property exists, and, therefore, called the deserialize method of my custom serializer, which returns null if it couldn't find innerNumber inside the JSON.

How can I indicate/notify the decoder, that, even though a property with the same name (number) exists, its decoding/parsing failed, and it should fallback to the default value?

[Elvis10ten]

+1

[pdvrieze]

Fairly simple, the custom deserializer should handle all cases, and either use local variables or run expressions to inject defaults in the right places. Something like:

override fun deserialize(decoder: Decoder) =
    (decoder as JsonInput).decodeJson().jsonObject.run { getPrimitiveOrNull("innerNumber")?.run{ intOrNull ?: -1 } ?: intOrNull ?: -1 }

[Elvis10ten]

@pdvrieze what I want (and I assume OP wants) is to define the default in the model only. I think your example duplicates the default in the serializer.

Essentially, we want something like coerceInput, but where the default value is used when the serializer throws an exception (that is the serializer deemed the data to be invalid).

[pdvrieze]

@Elvis10ten The problem is that the format/nor descriptor is able to know whether the value is default or what the default value is. Normally this is encoded in the generated serializer, but if you make a custom serializer you will have to do that by hand.

If you want more complex behaviour, what you do is create a separate (nested private) class for (de)serialization, delegate the de(serialization) to the serializer for that class, and then generate the actual model from that (with more complex rules/behaviour).

[Elvis10ten]

@pdvrieze thanks. It seems the library allows annotation lookup when they have SerialInfo. I considering adding an annotation that contains the default value. This seems easier to maintain than having two different classes. What do you think? Why doesn't the library do this by default?

Second question: Why is my custom serializer skipped when deserializing a value that is null, with coerceInputValues enabled. It somehow sets the default value correctly while skipping my serializer. Only when set the JSON value to a non-null value do I receive a call in my serializer.

[Elvis10ten]

Also, this issue has nothing to do with the fact that I defined a custom serializer.

It doesn't work with a generated serializer also. This is because I think the library doesn't have this capability.

Concrete example:

data class UiModel(
  val alpha: Float = 1
)

Alpha should range from 0 to 1.

Example JSONs:

1. "alpha": null => When coerceInputValues is true, the default value is used.
2. "alpha": 0.5 => Valid input, used.
3. "alpha": 20 => Valid float, but invalid alpha value. Default value should be used.

While I could define a custom Alpha class, it doesn't work well:

Imagine I have a custom Color class. It could be used in two different models, where one wants the default to be black and the other wants it to be white. In other words, default value is based on the situation not type.

---

Your suggestion to use a separate class looks high maintenance and error prone (especially when you have lots of classes to handle).

---

My suggestion is that the library either provides the serializer with access to the default value

Or

Allows the serializer throw a custom exception that forces the default value to be used. That is extending coerceInputValues to not only work with null and enums, but to be used when an error occurs parsing a property.

[pdvrieze]

@Elvis10ten On your first question. You can use SerialInfo (but the default cannot be a runtime value - a string that could be deserialized is a way to go). The library is intended to be a generic serialization framework. Handling invalid (or even out of range) values is not really something that the library can do (what is valid, and what do you do if it isn't). If you want to do this, the way I would do it is to have a "dumb" private nested class that you make serializable, and then in the custom serializer you serialize/serialize through that. At that point you can validate/coerce values in various ways (you then still the the advantage of getting the actual serialization generated).

The second question, the reason that your serializer doesn't show up is because nullable value serialization uses the NullableSerializer first, and only the type serializer if the value is not null. Again, if you want more extensive behaviour, the serialdelegate route is the way to go:

@Serializable(TestData.Serializer::class)
data class TestData(
    val number: Int? = -1,
    val attr2: String
) {

    @Serializable
    private class SerialDelegate(
        val number: JsonElement,
        val attr2: String
    )

    class Serializer: KSerializer<TestData> {
        private val delegateSerializer = SerialDelegate.serializer()
        override val descriptor = SerialDescriptor("my.package.TestData", delegateSerializer.descriptor())

        override fun serialize(encoder: Encoder, value: TestData) {
            delegateSerializer.serialize(encoder, SerialDelegate(JsonPrimitive(number), attr2))
        }

        override fun deserialize(decoder: Decoder): TestData {
            val delegate = delegateSerializer.deserialize(decoder)
            val number = when (val e = delegate.number) {
                is JsonNullSerializer -> DEFAULT
                is JsonLiteral -> e.content.toIntOrNull() ?: DEFAULT
                is JsonObject -> (e.content["innerNumber"] as? JsonPrimitive)?.content?.toIntOrNull() ?: DEFAULT
                else DEFAULT
            }
            return TestData(number, delegate.attr2)
        }
    }

    companion object {
        const val DEFAULT=-1
    }
}

[sandwwraith]

Yeah, we do not currently have a way to communicate from custom serialzier to the outer serializer things like 'this value should be coerced', neither we can pass serial info in it. Unfortunately, writing a custom serialzier for outer class seems to be the only approach.

Or, it is possible to serialize the private value and make a public val like this: val number: Int get() = _number ?: -1

[rodrigogs]

@Elvis10ten On your first question. You can use SerialInfo (but the default cannot be a runtime value - a string that could be deserialized is a way to go). The library is intended to be a generic serialization framework. Handling invalid (or even out of range) values is not really something that the library can do (what is valid, and what do you do if it isn't). If you want to do this, the way I would do it is to have a "dumb" private nested class that you make serializable, and then in the custom serializer you serialize/serialize through that. At that point you can validate/coerce values in various ways (you then still the the advantage of getting the actual serialization generated).

The second question, the reason that your serializer doesn't show up is because nullable value serialization uses the NullableSerializer first, and only the type serializer if the value is not null. Again, if you want more extensive behaviour, the serialdelegate route is the way to go:

@Serializable(TestData.Serializer::class)
data class TestData(
    val number: Int? = -1,
    val attr2: String
) {

    @Serializable
    private class SerialDelegate(
        val number: JsonElement,
        val attr2: String
    )

    class Serializer: KSerializer<TestData> {
        private val delegateSerializer = SerialDelegate.serializer()
        override val descriptor = SerialDescriptor("my.package.TestData", delegateSerializer.descriptor())

        override fun serialize(encoder: Encoder, value: TestData) {
            delegateSerializer.serialize(encoder, SerialDelegate(JsonPrimitive(number), attr2))
        }

        override fun deserialize(decoder: Decoder): TestData {
            val delegate = delegateSerializer.deserialize(decoder)
            val number = when (val e = delegate.number) {
                is JsonNullSerializer -> DEFAULT
                is JsonLiteral -> e.content.toIntOrNull() ?: DEFAULT
                is JsonObject -> (e.content["innerNumber"] as? JsonPrimitive)?.content?.toIntOrNull() ?: DEFAULT
                else DEFAULT
            }
            return TestData(number, delegate.attr2)
        }
    }

    companion object {
        const val DEFAULT=-1
    }
}

Thanks, it works!

[Elvis10ten]

@pdvrieze thanks once again.

My issue with the extra class is just the maintenance overhead.

I have tens of these classes I now need to create this classes for, and ensure they are in sync.

---

The library is intended to be a generic serialization framework. Handling invalid...

1. I would argue that falling back to the default value when a type serializer throws an exception makes sense and is still generic.

The library doesn't need to care about validation directly.

2. Alternatively/In addition, I think extending the library to support adding metadata to any properties.

I would prefer to have a single source of truth (one data class) with all the info needed. Than to have two or more classes just for serialization.

[pdvrieze]

@Elvis10ten The Json format in the library has some configurability but not to the extend that it can support what you want. The XML format that I have has some more options for extensibility (it delegates some elements to a policy where you can plug in behaviour), but even that has limits as to what can be customized, and some things cannot be supported at that level (it supports unexpected elements/attributes but not invalid values). If you really need something like that the option you have is to fork your own format (reusing the parser etc) that behaves differently.

However, even that has limits, one of points is that the library does not support references (or pointers) as such, so only tree-structures can be serialized (cycles are not valid, nor are latices (2 different containers referring to the same object)) - note that neither Json nor XML support that either. If you have a data structure with those properties you will have to do some limited transformation work anyway. Having played around with quite complex structures myself I have to say that using a delegate (private to the type - or internal and behind an optin when a container delegate needs a containee delegate directly) is overall much less complex than the workarounds needed to keep all logic in a custom serializer.

[mogud]

Use case: Protobuf schema generator cannot generate message schema correctly with fields that have non-type-specific default values in kotlin codes. But protobuf itself has had support default value for fields. The only limitation is that we cannot get the default value from SerialDescriptor interface.

[mgroth0]

Is this an appropriate place to request a feature that we be able to access or construct the default serializer for class even if we have provided a replacement? I see in the example above that a new class had to be created:

@Serializable
    private class SerialDelegate(
        val number: JsonElement,
        val attr2: String
    )

I'm requesting that instead of having to create a new class, that there be a way to just access the original generated serializer. If the original generalted serializer is never generated because a replacement was provided, then it would be perfectly sufficient if we could just regenerate the original serializer ourselves.

I thought of creating a new issue, but I figured I can't be the first person who has requested this.

[pdvrieze]

@mgroth0 I assume that the context you are asking is that you want to specify a custom serializer in the @Serializable annotation, but still have access to a default serializer. This should be (was) possible with a fully autogenerated custom serializer (with the @Serializer annotation), but note that there are various ways that overriding in the presence of this annotation is broken.

[mgroth0]

Thanks @pdvrieze . is there an issue here you could link that I can subscribe to for updates on the ability to access the default serializer, or shall I make new one?

[sandwwraith]

@mgroth0 Are you talking about #1169 ?

[mgroth0]

@sandwwraith yes. Thanks.