Allow `ListSerializer` to parse partial good results and ignore bad ones

[baruchn]

What is your use-case and why do you need this feature? I'm receiving a list of items from the server. Some of the items might miss some required fields or be otherwise malformed. In this case, I would like to ignore the items that failed to parse but still parse the ones that I can. Unfortunately, it looks like ListSerializer is failing as soon as it encounters a single list item which it fails to parse.

Describe the solution you'd like Some sort of ignoreFailedListItems or allowPartialListResults configuration.

*This probably applies to all collections.

[pdvrieze]

This is a problem with the format, not the ListSerializer. The common formats don't work with prepended list lengths so don't provide ListSerializer with a predefined amount of elements. While it is not easily possible to handle syntactically incorrect inputs, there is some legitimate reason to support (optional) lenience in skipping invalid entries for kinds where this could be valid (lists and optional parameters) - depending on the semantics of the format. Note that ignoring missing fields might mean the field should be optional, not mandatory.

[qwwdfsad]

FTR: https://stackoverflow.com/questions/54145519/moshi-adapter-to-skip-bad-objects-in-the-listt

[fabianhjr]

To add some context and use case:

I am using inline classes in the spirit of parse, don't validate. In particular I am doing something like:

@Serializable
data class Cooperative(
    val name: CooperativeName,
    val city: CityName,
    val country: CountryCode,
    val latitude: Latitude,
    val longitude: Longitude
) {
    companion object {
        fun setFromJSONString(string: String): Set<Cooperative> =
            lenientJson
                .decodeFromString<Set<Cooperative>>(string)

        fun toJSONString(cooperatives: Set<Cooperative>) = lenientJson.encodeToString(cooperatives)
    }
}

where every type has specific init requirements, examples:

class KUppercaseSerializer : KSerializer<String> {
    override val descriptor: SerialDescriptor = PrimitiveSerialDescriptor("UppercaseString", PrimitiveKind.STRING)
    override fun serialize(encoder: Encoder, value: String) = encoder.encodeString(value.uppercase())
    override fun deserialize(decoder: Decoder): String = decoder.decodeString().uppercase()
}

@JvmInline
@Serializable
value class CooperativeName(val name: String) {
    init {
        require(name.isNotBlank())
        require(name.trim() == name)
    }
}

@JvmInline
@Serializable
value class CountryCode(
    @Serializable(with = KUppercaseSerializer::class)
    val code: String
) {
    init {
        require(code.length == 2)
        require(code.all { it.isLetter() && it.isUpperCase() })
    }
}

@JvmInline
@Serializable
value class Latitude(
    @Serializable(with = KBigDecimalStringSerializer::class) val latitude: BigDecimal
) {
    init {
        require(latitude >= BigDecimal(-90))
        require(latitude <= BigDecimal(90))
    }
}

Currently, I have to do some boilerplate code around this very strict parsers. Something like:

@Serializable
private data class LaxCooperative(
    val name: String,
    val city: String,
    val country: String,
    val latitude: String? = null,
    val longitude: String? = null
)

private fun validate(cooperative: LaxCooperative): Cooperative? =
    try {
        if (cooperative.mail !== null && cooperative.latitude !== null && cooperative.longitude !== null)
            Cooperative(
                CooperativeName(cooperative.name),
                CityName(cooperative.city),
                CountryCode(cooperative.country),
                Latitude(BigDecimal(cooperative.latitude)),
                Longitude(BigDecimal(cooperative.longitude)),
            ) else null
    } catch (t: Throwable) { null }

and change setFromJSONString to

        fun setFromJSONString(string: String): Set<Cooperative> =
            lenientJson
                .decodeFromString<Set<LaxCooperative>>(string)
                .mapNotNull { validate(it) }
                .toSet()

This way I do a "lax" parse of my data class and then attempt to build the strict version.

[illuzor]

I made my own implementation

Easy to apply it to list:

@Serializable
private data class Data(val name: String, val value: Int)

val jsonString ="""[{"name":"Name1","value":11},{"name":"Name2","value":22}]"""
val serializer = kindListSerializer(Data.serializer())
Json.decodeFromString(serializer, jsonString)

But what to do if list is property of serializable class?

@Serializable
private data class DataContainer(
    val name: String,
    val data: List<Data>, // TODO ???
)

[illuzor]

I found solution: But faced a new issue: It happen even if I use builtin list serializer: Is it possible to fix it? I have no idea what to do

[sandwwraith]

It looks like the error is imprecise. If your CustomDataSerializer is a serializer for List, it should be applied to the List, not to its generic type arg: val data: @Serializable(CustomDataSerializer::class) List<Data>

[illuzor]

@sandwwraith thank you! It works

[illuzor]

@sandwwraith but not always... Here the case where compilation error occurs. I can`t understand why. I can provide sample if needed

[pdvrieze]

A serializer for a type with generic parameter will need a constructor that takes a serializer for that generic parameter. In this case you want to use @SerializeWith(KindListSerializer::class), you don't need to create a subclass for the parameter type (that will actually not work as you found).

[illuzor]

@pdvrieze Thank you! It works, but I can't figure out how. But why does it works here?

[werner77]

The solution I chose uses a wrapped value which becomes null if it cannot be decoded:

/**
 * Serializable class which wraps a value which is optionally decoded (in case the client does not have support for
 * the corresponding value implementation).
 */
@Serializable(with = WrappedSerializer::class)
data class Wrapped<T : Any>(val value: T?) {
    fun unwrapped(default: T): T {
        return value ?: default
    }

    override fun toString(): String {
        return value?.toString() ?: "null"
    }
}

/**
 * Serializer for [Wrapped] values.
 */
class WrappedSerializer<T : Any>(private val valueSerializer: KSerializer<T?>) : KSerializer<Wrapped<T>> {
    override val descriptor: SerialDescriptor = valueSerializer.descriptor

    private val objectSerializer = JsonObject.serializer()

    override fun serialize(encoder: Encoder, value: Wrapped<T>) {
        valueSerializer.serialize(encoder, value.value)
    }

    override fun deserialize(decoder: Decoder): Wrapped<T> {
        val decoderProxy = DecoderProxy(decoder)
        return try {
            Wrapped(valueSerializer.deserialize(decoderProxy))
        } catch (ex: Exception) {
            println("Failed deserializing of ${valueSerializer.descriptor}: ${ex.message}")
            // Consume the rest of the input if we are inside a structure
            decoderProxy.compositeDecoder?.let {
                decoderProxy.decodeSerializableValue(objectSerializer)
                it.endStructure(valueSerializer.descriptor)
            }
            Wrapped(null)
        }
    }
}

private class DecoderProxy(private val decoder: Decoder) : Decoder by decoder {

    var compositeDecoder: CompositeDecoder? = null

    override fun beginStructure(descriptor: SerialDescriptor): CompositeDecoder {
        val compositeDecoder = decoder.beginStructure(descriptor)
        this.compositeDecoder = compositeDecoder
        return compositeDecoder
    }
}

with this class you can make Safe collections, for example a safe list which behaves just like a list using delegation:

@Suppress("DataClassPrivateConstructor")
@Serializable(with = SafeList.Serializer::class)
data class SafeList<E : Any> private constructor(
    private val _values: List<Wrapped<E>>
) : List<E> by _values.unwrappedList() {
    constructor(values: Iterable<E>) : this(values.wrappedList())
    constructor() : this(emptyList())

    class Serializer<E: Any>(valueSerializer: KSerializer<E?>) : KSerializer<SafeList<E>> {

        private val _serializer = ListSerializer(WrappedSerializer(valueSerializer))

        override val descriptor: SerialDescriptor = _serializer.descriptor

        override fun deserialize(decoder: Decoder): SafeList<E> {
            return SafeList(_values = _serializer.deserialize(decoder))
        }

        override fun serialize(encoder: Encoder, value: SafeList<E>) {
            _serializer.serialize(encoder, value._values)
        }
    }

    override fun toString(): String {
        return "[" + _values.unwrappedList().joinToString(", ") + "]"
    }

    companion object {
        @JvmStatic
        fun <E: Any>of(vararg elements: E) : SafeList<E> {
            return safeListOf(*elements)
        }
    }
}

fun <E: Any>safeListOf(vararg elements: E) : SafeList<E> {
    return SafeList(elements.asIterable())
}

/**
 * Convenience function to wrap a list of [Any] values
 */
fun <T : Any> Iterable<T>.wrappedList(): List<Wrapped<T>> {
    return this.map { Wrapped(it) }
}

/**
 * Convenience function to unwrap a collection of [Wrapped] values to a list
 */
fun <T : Any> Iterable<Wrapped<T>>.unwrappedList(): List<T> {
    return this.mapNotNull { it.value }
}

Now instead of an ordinary List you would use a SafeList in your data objects:

@Serializable
data class Foo(val bars: SafeList<Bar>)

@Serializable
data class Bar(val name: String)

[valeriyo]

Has this still not landed yet?