kotlinx-serialization - Store/fetch data classes with JSON fields

[radoye]

Hello! What is the recommended way to work with data classes containing JsonObject or JsonElement fields, i.e., an untyped, open-ended key-value element?

Here's a simple example

// Metadata.kt
import kotlinx.serialization.Serializable
import kotlinx.serialization.json.JsonElement

typealias JSON = JsonElement

@Serializable
data class Metadata<T>(val _id: String, val tags: T)

// Main.kt

        val cli = org.litote.kmongo.reactivestreams.KMongo.createClient("mongodb://localhost:27017").coroutine
        runBlocking {
            val db = cli.getDatabase("db-0")
            val coll = db.getCollection<Metadata<JSON>>("coll-0")

            val json : JSON = buildJsonObject {
                put("count", 42)
                put("status", "success")
            }

            val original = Metadata("sample000", json)

            println("ORIGINAL $original")
            coll.insertOne(original)

            val readback = coll.findOne(Metadata<JSON>::_id eq "sample000")!!
            println("READBACK $readback")

            val copycat = Metadata("sample000-copy", readback.tags)
            println("COPYCAT $copycat")
            coll.insertOne(copycat)

            val reread = coll.findOne(Metadata<JSON>::_id eq "sample000-copy")!!
            println("REREAD $reread")

        }

This results in the following log trace

ORIGINAL Metadata(_id=sample000, tags={"count":42,"status":"success"})
22:03:36.403 [Thread-3] INFO org.mongodb.driver.connection - Opened connection [connectionId{localValue:6, serverValue:131}] to localhost:27017
22:03:36.404 [Thread-3] DEBUG org.mongodb.driver.operation - retryWrites set to true but the server is a standalone server.
22:03:36.409 [Thread-3] DEBUG org.mongodb.driver.protocol.command - Sending command '{"insert": "coll-0", "ordered": true, "$db": "db-0", "lsid": {"id": {"$binary": {"base64": "mi33y32hTD6Py/braGLJzg==", "subType": "04"}}}, "documents": [{"_id": "sample000", "tags": {"count": {"isString": false, "content": "42"}, "status": {"isString": true, "content": "success"}}}]}' with request id 15 to database db-0 on connection [connectionId{localValue:6, serverValue:131}] to server localhost:27017
22:03:36.426 [Thread-7] DEBUG org.mongodb.driver.protocol.command - Execution of command with request id 15 completed successfully in 17.34 ms on connection [connectionId{localValue:6, serverValue:131}] to server localhost:27017
22:03:36.428 [main] DEBUG org.mongodb.driver.protocol.command - Sending command '{"find": "coll-0", "filter": {"_id": "sample000"}, "limit": 1, "singleBatch": true, "$db": "db-0", "lsid": {"id": {"$binary": {"base64": "mi33y32hTD6Py/braGLJzg==", "subType": "04"}}}}' with request id 16 to database db-0 on connection [connectionId{localValue:6, serverValue:131}] to server localhost:27017
22:03:36.433 [Thread-16] DEBUG org.mongodb.driver.protocol.command - Execution of command with request id 16 completed successfully in 4.80 ms on connection [connectionId{localValue:6, serverValue:131}] to server localhost:27017
READBACK Metadata(_id=sample000, tags={count={isString=false, content=42}, status={isString=true, content=success}})
COPYCAT Metadata(_id=sample000-copy, tags={count={isString=false, content=42}, status={isString=true, content=success}})
22:03:36.435 [main] DEBUG org.mongodb.driver.operation - retryWrites set to true but the server is a standalone server.
22:03:36.437 [main] DEBUG org.mongodb.driver.protocol.command - Sending command '{"insert": "coll-0", "ordered": true, "$db": "db-0", "lsid": {"id": {"$binary": {"base64": "mi33y32hTD6Py/braGLJzg==", "subType": "04"}}}, "documents": [{"_id": "sample000-copy", "tags": {"count": {"isString": false, "content": "42"}, "status": {"isString": true, "content": "success"}}}]}' with request id 17 to database db-0 on connection [connectionId{localValue:6, serverValue:131}] to server localhost:27017
22:03:36.438 [Thread-8] DEBUG org.mongodb.driver.protocol.command - Execution of command with request id 17 completed successfully in 1.08 ms on connection [connectionId{localValue:6, serverValue:131}] to server localhost:27017
22:03:36.439 [main] DEBUG org.mongodb.driver.protocol.command - Sending command '{"find": "coll-0", "filter": {"_id": "sample000-copy"}, "limit": 1, "singleBatch": true, "$db": "db-0", "lsid": {"id": {"$binary": {"base64": "mi33y32hTD6Py/braGLJzg==", "subType": "04"}}}}' with request id 18 to database db-0 on connection [connectionId{localValue:6, serverValue:131}] to server localhost:27017
22:03:36.440 [Thread-9] DEBUG org.mongodb.driver.protocol.command - Execution of command with request id 18 completed successfully in 0.89 ms on connection [connectionId{localValue:6, serverValue:131}] to server localhost:27017
REREAD Metadata(_id=sample000-copy, tags={count={isString=false, content=42}, status={isString=true, content=success}})

Note how some additional metadata, e.g., isString=false, gets inserted. How to avoid this?

Interestingly, if we remove generic parameter in the definition of Metadata it completely breaks.

// Metadata.kt
import kotlinx.serialization.Serializable
import kotlinx.serialization.json.JsonElement

typealias JSON = JsonElement

@Serializable
data class Metadata(val _id: String, val tags: JsonElement) // <-- simply removed generic parameter & inlined JsonElement

This breaks with the following exception (com.fasterxml.jackson.databind.exc.InvalidDefinitionException: Cannot construct instance of `kotlinx.serialization.json.JsonElement)

ORIGINAL Metadata(_id=sample000, tags={"count":42,"status":"success"})
22:13:29.001 [Thread-3] INFO org.mongodb.driver.connection - Opened connection [connectionId{localValue:6, serverValue:137}] to localhost:27017
22:13:29.001 [Thread-3] DEBUG org.mongodb.driver.operation - retryWrites set to true but the server is a standalone server.
22:13:29.005 [Thread-3] DEBUG org.mongodb.driver.protocol.command - Sending command '{"insert": "coll-0", "ordered": true, "$db": "db-0", "lsid": {"id": {"$binary": {"base64": "6529JjXTTHS/tx5h9lfNKg==", "subType": "04"}}}, "documents": [{"_id": "sample000", "tags": {"count": {"isString": false, "content": "42"}, "status": {"isString": true, "content": "success"}}}]}' with request id 15 to database db-0 on connection [connectionId{localValue:6, serverValue:137}] to server localhost:27017
22:13:29.016 [cluster-rtt-ClusterId{value='602a11084fc3fb06cd9311d5', description='null'}-localhost:27017] INFO org.mongodb.driver.connection - Opened connection [connectionId{localValue:5, serverValue:135}] to localhost:27017
22:13:29.022 [Thread-4] DEBUG org.mongodb.driver.protocol.command - Execution of command with request id 15 completed successfully in 16.84 ms on connection [connectionId{localValue:6, serverValue:137}] to server localhost:27017
22:13:29.023 [main] DEBUG org.mongodb.driver.protocol.command - Sending command '{"find": "coll-0", "filter": {"_id": "sample000"}, "limit": 1, "singleBatch": true, "$db": "db-0", "lsid": {"id": {"$binary": {"base64": "6529JjXTTHS/tx5h9lfNKg==", "subType": "04"}}}}' with request id 16 to database db-0 on connection [connectionId{localValue:6, serverValue:137}] to server localhost:27017
22:13:29.024 [Thread-8] DEBUG org.mongodb.driver.protocol.command - Execution of command with request id 16 completed successfully in 0.92 ms on connection [connectionId{localValue:6, serverValue:137}] to server localhost:27017
22:13:29.029 [Thread-8] DEBUG org.mongodb.driver.operation - Unable to retry operation find due to error "com.fasterxml.jackson.databind.exc.InvalidDefinitionException: Cannot construct instance of `kotlinx.serialization.json.JsonElement` (no Creators, like default constructor, exist): abstract types either need to be mapped to concrete types, have custom deserializer, or contain additional type information
 at [Source: de.undercouch.bson4jackson.io.LittleEndianInputStream@55530c38; pos: 25] (through reference chain: datakube.Metadata["tags"])"

[zigzago]

Hello,

Here you use the kmongo-coroutine dependency (jackson mapping is used)

You may prefer to use the kmongo-coroutine-serialization and then you get a more meaningful error:

java.lang.IllegalStateException: This serializer can be used only with Json format.Expected Encoder to be JsonEncoder, got class com.github.jershell.kbson.BsonEncoder.

At this point I do not know any way to achieve what you want with kmongo-coroutine-serialization.

Could be a nice feature to add to https://github.com/jershell/kbson

[NathanPB]

Does anyone have solutions on how to store non-typesafe values like a Document or JsonElement described in the issue? Also looking for this :eyes:

[zigzago]

Due to the type safe nature of kotlinx.serialization, this is not easy (but doable). An other option is to use kmongo jackson mapping that already supports its.

This comment: https://github.com/Kotlin/kotlinx.serialization/issues/296#issuecomment-758136040 could be be a good starting point if someone would like to implement it in kbson.

[NathanPB]

I don't think I get the point where kBson could help on this. Doesn't it enforces type safe data as well?

[zigzago]

kbson is the lib that kmongo uses under the hood to support bson kotlinx.serialization. If you need to support generic Map (or Document) serialization, it would be better (and easier) to add the feature directly in kbson.

[NathanPB]

Got it. Never worked with that before, but I may be trying something later

[ArthurSav]

@zigzago Not sure i understand the issue.

This is decodable by kotlin serialization

@Serializable
data class MyData(val id: String, val stuff: kotlinx.serialization.json.JsonObject)

Why does kmongo throw?

 java.lang.IllegalStateException: This serializer can be used only with Json format.Expected Encoder to be JsonEncoder, got class com.github.jershell.kbson.BsonEncoder

[zigzago]

kmongo uses kbson to encode/decode bson (mongodb data format). Kbson does not support JsonObject encoding or decoding for now.

HTH

[ArthurSav]

Is it possible to use a serializer to transform a kotlinx JsonObject to a bson JsonObject and vice versa? As far as i know, bson JsonObject can be sent with Documents.

[zigzago]

What you can do is to write a KSerializer that encodes and decodes a JsonObject and register this serializer. It should work. It would be a nice PR for kmongo & kbson.

[copyandexecute]

bump