Generic Data Structure Serialize / Deserialize

[mattinger]

When parsing JSON, it is not always desirable, or required to map the parse result onto a custom data structure. There are many libraries (gson comes to mind), that allow the user to parse JSON data both to a custom data structure and a generic one. For instance:

    val input = "{}"
    val parser = JsonParser()
    val root = parser.parse(input)
    val prop = when (root) {
        is JsonObject -> root.get("prop")?.asString
        else -> null
    }

(yes i'm aware this is less than optimal kotlin code, but it illustrates the point, especially if you needed to do something different in the array case)

The current classes in kotlinx.serialization are focused on mapping to custom data structures.

A nice bonus of having this as part of the kotlinx.serialization package would be that it would be cross platform. Currently, as far as i can tell, there is no cross platform way of parsing JSON data. As a result, i'm using expect and actual for something that really could be platform agnostic.

[elizarov]

+1. We need design a class hierarchy for Json with JsonObject and others and support it in our JSON and CBOR parsers.

[czeidler]

Another related use case is to parse json dynamically. For example, consider the following json data structure: { type: "type", data: {...}} In this example the data field depends on the type field, i.e. depending on the type I like to deserialize different data classes.

Parsing json data dynamically could work as follows:

@Serializable
class Container(val type: String, val data: JsonObject)

val container = JSON.parse(jsonString)

when (container.type) {
    "data1" -> JSON.parse<Data1>(container.data)
    "data2" -> JSON.parse<Data2>(container.data)
}

[czeidler]

A JsonObject would also nice for serialization. Using a JsonObject the KOutput interface can be simplified to:

interface KOutput {
    fun <T> save(object: T): JsonObject
}

As far as I can see it is currently not possible to manually serialize a class that cannot be stored into a single variable using the current KOutput interface. For example if I want to serialize:

class Example(val value1: String = "v1",
              val value2: String = "v2",
              val value3: String = "v3")

to

{ value1 = "v1", merged = "v2/v3" }

This would be trivial to do with the proposed KOutput interface.

[JakeWharton]

Moshi (Gson's successor) uses Map<String, Any?>, List<Any?>, String, Long, and Double types to represent arbitrary JSON using the term "JSON values". This eliminates the need for wasting classes on modeling JSON values as a type hierarchy when collections are already sufficient.

[mattinger]

Agreed, it's nice the way Moshi does it, and I would be fine with that. However, from what i've been able to test so far, It doesn't seem like one can parse that way currently.

[sandwwraith]

I'd rather avoid including concept such as JsonObject to any interfaces or APIs, because this framework tries to be format-agnostic, and some formats don't have concept of hierarchical or key/value structures at all. However, deserialization to JsonObject/Array/etc should be possible, and implemented in such way that it will give more type-safety than usual Map<String, Any?>

[konmik]

@sandwwraith I afraid that deserialization into JsonObject gives more boilerplate code than type-safety. I mean, we still have to check what kind of data is inside JsonObject absolutely the same way we do it with Map<String, Any?>, so no type safety for us. Moshi approach is great.

I've just tried to use kotlinx.serialization for the first time today and immediately stuck with this issue when object type is being passed inside json object itself, see @czeidler's first example. It is a very typical case with json when we have a field that can be of several different types depending on it's content. The lack of this feature is a major no-go for us.

[czeidler]

@sandwwraith yes JsonObject is a bad name and SerializationObject is probably what we are talking about here. I agree that Map + List is probably sufficient for such an serialization object.

I don't understand your comment about format-agnostic and hierarchical key/value structures. You have the same problem with classes don't you? However, other than classes some kind of serialization object would make more complex, custom serialization/deserialization possible.

For a bit of type safety you can wrap the Map/List into a JsonObject/CborObject/ProtoBufferObject. For example, a ProtoBufferObject would not allow to add null values but would allow to add raw ByteArrays.

[ariefannur]

can i serialize data class with generic type like that?

@Serializable
 data class BaseResult<T>(
    val page: Int,
    val total_results: Int,
    val total_pages: Int,
    val results: List<T>
)

hope next version can do this

[sandwwraith]

@ariefannur This is already supported. And this issue is not about that, it's about tree-like structure for abstract data manipulating

[sandwwraith]

I'm closing this as 'won't fix' for now because providing AST for every possible format is a huge design task with unclear benefits — it is a rare case when you need to convert schemaless data from one format to another. For JSON, there's JsonElement hierarchy