Performance Comparison: Kotlin Serialization vs Moshi

[ngehad]

Issue Description This issue compares the performance of KotlinX Serialization with Moshi for JSON parsing and serialization operations. The benchmark focuses on time taken (execution time) and the number of objects allocated during the process. Benchmarking was done with Jetpack Microbenchmark, and we tested with small json response and a large one which counts approximately x21 the size of the small one.

TLDR: KotlinX serialization allocates way more objects in the memory than Moshi specially with large json response, and its execution time during serializing json is longer than Moshi.

Looking at the below table, in which we recorded our experiment results, we can see that: 1- With Small Payload (first 2 result rows): Moshi executed in slightly less time when converting from json, while kotlinx serialization executed in half the time of Moshi when converting to json. Also, the number of objects allocated in the memory was slightly higher for converting from json and equal for converting to json. 2- With Larger Response (last 2 rows): KotlinX serialization has jumped to double the execution time of Moshi when converting from json, but it performed better (time-wise) when it converted to json. On the other hand, KotlinX serialization looks to have allocated way too many objects in the memory compared to moshi for converting to json, which seems off to me. (PS. I'm aware that the number of objects allocated doesn't exactly reflect amount of memory used but my concern is about the overhead created for the garbage collector).

Detailed results in json format: moshi results, kotlinx serialization results

Please share your thoughts if you think we're doing something wrong.

To Reproduce Json Test data: small json, larger json

1. Setup micro-benchmark library
2. Add model classes for both responses here with ktx serialization annotations and here with moshi Annotations

3. Use measureRepeated method from BenchmarkRule object in a test examples: benchmarkRule.measureRepeated { Json.decodeFromString<StResponse>(data) }

   benchmarkRule.measureRepeated { Json.encodeToString(data) }

   moshi = Moshi.Builder().add(KotlinJsonAdapterFactory()).build() val data = prepareStResponseJson() benchmarkRule.measureRepeated { moshi.fromJson<StResponse>(data) }

   moshi = Moshi.Builder().add(KotlinJsonAdapterFactory()).build() val data = prepareStoreResponseObject() benchmarkRule.measureRepeated { moshi.toJson(data) }

   Util Functions inline fun <reified T> Moshi.toJson(data: T): String = adapter(T::class.java).toJson(data) inline fun <reified T> Moshi.fromJson(json: String): T? = adapter(T::class.java).fromJson(json)

Environment

• Kotlin version: 1.9.22
• Library version: 1.6.0
• Kotlin platforms: Android (JVM)
• Gradle version: 8.0
• Moshi version: 1.15.1
  • Android Device: Huawei Mate 20 lite

[sandwwraith]

Hi! Thanks for the detailed report, I appreciate it. I have a couple of questions to clarify:

1. You said that 'Moshi performed executed in slightly less time when converting serializing json,' while corresponding benchmark name is FromJson. Did you actually mean DEserializing from json to data class, i.e. Moshi.fromJson and Json.decodeFromString functions? And ToJson benchmark corresponds to serializing data class to json — Moshi.toJson and Json.encodeToString.
2. 'generate model classes from json for both responses' — what generator did you use? Or better, you can just insert StResponse definition here.
3. It would be really helpful if you provide self-contained project (in ZIP or on GitHub) with all the benchmarks which I can just open in Android Studio or IDEA and start investigating — to see if results are reproducible and what I can do to lower the number of allocations. I have some ideas where the problem may be, but having benchmarks project directly at my disposal will tremendously help me. Thank you!

[ngehad]

Hi! Thanks for the detailed report, I appreciate it. I have a couple of questions to clarify:

1. You said that 'Moshi performed executed in slightly less time when converting serializing json,' while corresponding benchmark name is `FromJson`. Did you actually mean DEserializing from json to data class, i.e. `Moshi.fromJson` and `Json.decodeFromString` functions? And `ToJson` benchmark corresponds to serializing data class to json — `Moshi.toJson` and `Json.encodeToString`.

2. 'generate model classes from json for both responses' — what generator did you use? Or better, you can just insert `StResponse` definition here.

3. It would be really helpful if you provide self-contained project (in ZIP or on GitHub) with all the benchmarks which I can just open in Android Studio or IDEA and start investigating — to see if results are reproducible and what I can do to lower the number of allocations. I have some ideas where the problem may be, but having benchmarks project directly at my disposal will tremendously help me. Thank you!

Hi! Thanks for the prompt response.

1. for this point, sorry if I my phrasing confused you. So for serialization I refer to from JSON to a model object and by Parsing I refer to from model object to JSON.
2. No problem. Check the models here: here with ktx serialization annotations and here with moshi Annotations
3. It's a little hard to provide the project right now because I was doing the work in a project in my company and it's not open source and I had to change the structure of the json and model classes and add dummy data to share in this issue. So I will need to allocate time to copy it to an outside project and change the names and copy the dummy data files, etc.. . I can do that by end of day tomorrow or on Monday.

[sandwwraith]

Okay, it's just that usually everyone uses those terms in an opposite way, i.e. parsing is a process of consuming json and producing data object, that's why I got confused :) Thanks for the model classes, I can try to replicate results now without waiting for the whole project, so don't rush yourself.

[JakeWharton]

Definitely interested in how you are invoking the libraries.

Moshi, for example, is optimized for reading to and from UTF-8 bytes in a streaming fashion with Okio. If you are using java.io, or using Strings as your source or destination, the benchmark now includes UTF-8 encoding and decoding as well as forced byte array copies.

This library also has many modes in which it can be invoked, and the most performant is usually when targeting a string. Streaming is generally slower, but if you want to do apples-to-apples maybe that's what you want.

It may be beneficial having separate benchmark modes where you change the destination from bytes to strings in both libraries and compare, and then ultimately compare the best mode of both. Libraries like Retrofit can actually choose the most performant mode of each library, for example, but others might be interested in the same-mode comparisons where something like streaming is required.

[ngehad]

Okay, it's just that usually everyone uses those terms in an opposite way, i.e. parsing is a process of consuming json and producing data object, that's why I got confused :) Thanks for the model classes, I can try to replicate results now without waiting for the whole project, so don't rush yourself.

Oh, my bad! I updated the description to avoid the confusion.

[ngehad]

Definitely interested in how you are invoking the libraries.

Moshi, for example, is optimized for reading to and from UTF-8 bytes in a streaming fashion with Okio. If you are using java.io, or using Strings as your source or destination, the benchmark now includes UTF-8 encoding and decoding as well as forced byte array copies.

This library also has many modes in which it can be invoked, and the most performant is usually when targeting a string. Streaming is generally slower, but if you want to do apples-to-apples maybe that's what you want.

It may be beneficial having separate benchmark modes where you change the destination from bytes to strings in both libraries and compare, and then ultimately compare the best mode of both. Libraries like Retrofit can actually choose the most performant mode of each library, for example, but others might be interested in the same-mode comparisons where something like streaming is required.

Thanks for the comment! You're absolutely right, the current benchmark might be influenced by the data format used, since I'm using JSON data in string format for testing. I'm open to setting up a separate benchmark mode for byte data. But I'm really still curious where this difference came from since I used the string format in both tests.

[ngehad]

@sandwwraith Hello, hope you've been doin' well. I was wondering if we have an update on this issue.

[sandwwraith]

@ngehad I don't have any updates for now, but I still have this issue on my table

[Shady-Selim]

@sandwwraith no need to mention the importance of this issue, as it affects our company teams decision when starting a new project or revamping old one, for which library to adopt

[qwwdfsad]

Please take into account that performance is a delicate matter. Benchmarking is an analysis tool to supplement profiling, and without it, there is no interpretation of what library is "faster." We will avoid any generic comparisons, especially if they affect anyone's decision-making based on our claims.

We can provide a commentary on the benchmarking methodology, see if we have any systematic performance issues (e.g. like #2743), and figure out whether there are some improvements for specific scenarios, use-cases or modes.

Right now, we would like to ask you for a self-contained reproducer (the one that we can checkout/copy-paste & run) -- I've tested the data provided, but the data model and JSONs provided are mismatched -- there are seemingly incorrect property names, mismatched data types, missing non-optional fields etc.