OutOfMemoryError happens when try to decode the wrong ByteArray

sang-eun commented 4 months ago

Describe the bug Sometimes our server get wrong ByteArray. We hoped the server throw wrong format error immediately when we try to decode the message. However, protobuf decoder blocks until java heap space out of memory error happens.

From our debugging, message is keep pushBacked by reader.pushBackTag() and index never moves forward from this method.

private fun decodeTaggedListIndex(): Int {
        val protoId = if (index == -1) {
            // For the very first element tag is already read by the parent
            reader.currentId
        } else {
            reader.readTag()
        }

        return if (protoId == tagOrSize.protoId) {
            ++index
        } else {
            // If we read tag of a different message, push it back to the reader and bail out
            reader.pushBackTag()
            CompositeDecoder.DECODE_DONE
        }
    }

To Reproduce

// class 
@Serializable
@SerialName("FrontRequest")
data class FrontRequest (
    val requests: List<String>,
) {
    companion object {
        @OptIn(ExperimentalSerializationApi::class)
        fun fromProtobuf(bytes: ByteArray): FrontRequest
            = ProtoBuf.decodeFromByteArray(bytes)
    }
}

//test 
val payloadBase64 = "+kCbAII+8m6eL8J/nzj6GfwQwzzAa6gzw2uhO8FJ/AGbOv9vr2y/P8l1o1SWPfcylGOnaMFroz7BOak6xGykapQ6qG6Sa/M5lD+jb5RupzvFbvBvxGmnbZVt9WiSaqY6yWL1bpQ/8z3EbvdpwG6pacRj9GrEbfI8xmr1OpM+o2nHPvBqwT7wb8Fup2g="
val payload = Base64.getDecoder().decode(payloadBase64)
//keep working until java heap space error
val frontRequestReceived = FrontRequest.fromProtobuf(payload)

// result

Java heap space
java.lang.OutOfMemoryError: Java heap space
    at java.base/java.util.Arrays.copyOf(Arrays.java:3609)
    at kotlinx.serialization.protobuf.internal.ProtobufTaggedBase.expand(ProtobufTaggedBase.kt:40)
    at kotlinx.serialization.protobuf.internal.ProtobufTaggedBase.pushTag(ProtobufTaggedBase.kt:34)
    at kotlinx.serialization.protobuf.internal.ProtobufTaggedBase.access$pushTag(ProtobufTaggedBase.kt:17)
    at kotlinx.serialization.protobuf.internal.ProtobufTaggedDecoder.decodeSerializableElement(ProtobufTaggedDecoder.kt:110)
    at com.dunamu.quot.realtime.front.request.FrontRequest$$serializer.deserialize(FrontRequest.kt:13)

Expected behavior throw wrong format error immediately.

Environment

Kotlin version: 1.8.21
Library version: 1.5.1
Kotlin platforms: JVM
Gradle version: 7.5.1

sandwwraith commented 4 months ago

Thanks, reproduced on 1.9.21 + 1.6.3, too

Kotlin / kotlinx.serialization

OutOfMemoryError happens when try to decode the wrong ByteArray #2649