search

fsanti68 / json-transformation-evaluation

JSON to JSON transformation alternatives evaluation
1 stars 0 forks source link

Source for JOLT & JSLT missing #1

Open larsga opened 5 years ago

larsga commented 5 years ago

Without the source code for the Jolt and JSLT evaluations it's hard to evaluate this comparison. I realize this may be only for your own purposes, in which case it doesn't matter, but for others to judge the results it would be very useful to see the source code for the other two.

fsanti68 commented 5 years ago

Hi Lars, I recognize haven't dedicated further attention to Jolt and JSLT after the first test: the class "TransformatioBenchmarkTest" includes them all (Jolt, Jslt, FM).

There is a bias to FreeMarker since it was easier for me to reach my business goal, but let me know if you have any suggestion to improve the comparison and let it more fair.

larsga commented 5 years ago

I don't know exactly what your business goals are, so there might be reaons why this test is the way it is that are not clear to an outsider. However, there are some things that are worth at least remarking on.

One thing is that omitting only the first iteration is not enough to exclude the JIT compilation. You need to run the test a few hundred times to make sure the JIT is done. Whether this has any systematic effects on the benchmark I don't know, but it will add noise.

Secondly, you time two thing as part of the comparison: (a) JSON loading, and (b) JSON transform. Ostensibly, this is about comparing the JSON transform, but the benchmark covers both phases. That may or may not be right for you. It depends on how the transforms are supposed to be used.

Thirdly, for the JOLT and Freemarker transforms the timed loop consists of (a) JSON loading, (b) JSON transform, but for JSLT it's (a) JSLT compile, (b) JSON loading, (c) JSON transform. That's not an equal comparison. The other two have the compilation/loading of the transform itself outside the timed loop.

Fourthly, the Freemarker benchmark not only produces the serialized JSON result inside the loop, but also parses it. That may or may not be right for your purposes. Maybe you want to measure JsonNode -> JsonNode (in which case it's right) or you may want to measure serialized JSON to serialized JSON (in which case it's wrong).