search

confluentinc / openmessaging-benchmark

Apache License 2.0
5 stars 59 forks source link

ERROR HistogramDeserializer - Failed to decode publish delay latency:... #21

Open jscott0918 opened 1 year ago

jscott0918 commented 1 year ago

At the end of high load workloads with high latency, runs fail to complete cleanly and no output summary is generated.

The following error(s) are printed + a very large table of hex values, presumably the raw content of the buffer that was the wrong size:

17:18:27.763 [AsyncHttpClient-3-1] ERROR HistogramDeserializer - Failed to decode publish delay latency: +-------------------------------------------------+ | 0 1 2 3 4 5 6 7 8 9 a b c d e f | +--------+-------------------------------------------------+----------------+ |00000000| 78 9c 2c b4 0b 68 9a f9 17 2d fa 37 31 6a 8c af |x.,..h...-.71j..| |00000010| f8 8a 51 93 a8 79 9b 97 c9 f4 94 de 12 7a 43 09 |..Q..y.......zC.| |00000020| a5 84 52 7a 4b 08 25 94 52 7a 4a 29 a5 94 52 ca |..RzK.%.RzJ)..R.| |00000030| 30 0c c3 20 22 5e 11 11 f1 88 88 88 88 57 44 44 |0.. "^.......WDD| |00000040| 44 44 44 44 82 88 88 57 44 82 88 88 04 09 22 22 |DDDD...WD.....""| |00000050| 22 41 82 88 88 84 b3 3e ff 67 be f9 3e 7f 8f bd |"A.....>.g..>...| ... |000fffd0| 12 a8 f5 77 83 f7 4f d5 ce 82 02 f7 be 86 2a 72 |...w..O.......*r| |000fffe0| cf 77 b5 49 a9 d5 d0 ea 93 44 f7 20 d2 4c b8 fa |.w.I.....D. .L..| |000ffff0| a1 dd d5 11 50 75 9b 79 |....Pu.y | +--------+-------------------------------------------------+----------------+ 17:18:27.844 [main] ERROR Benchmark - Failed to run the workload '500k rate 4 producers and 4 consumers on 1 topic / 100 partition' for driver 'driver-kafka/kafka-throughput.yaml' java.util.concurrent.CompletionException: java.util.concurrent.CompletionException: java.lang.RuntimeException: com.fasterxml.jackson.databind.JsonMappingException: java.lang.IllegalArgumentException: The buffer does not contain the indicated payload amount (through reference chain: io.openmessaging.benchmark.worker.commands.CumulativeLatencies["publishLatency"]) at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?] at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?] at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?] at java.lang.reflect.Constructor.newInstance(Constructor.java:490) ~[?:?] at java.util.concurrent.ForkJoinTask.getThrowableException(ForkJoinTask.java:600) ~[?:?] at java.util.concurrent.ForkJoinTask.reportException(ForkJoinTask.java:678) ~[?:?] at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:737) ~[?:?] at java.util.stream.ReduceOps$ReduceOp.evaluateParallel(ReduceOps.java:919) ~[?:?] at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233) ~[?:?] at java.util.stream.ReferencePipeline.reduce(ReferencePipeline.java:553) ~[?:?] at io.openmessaging.benchmark.worker.DistributedWorkersEnsemble.getCumulativeLatencies(DistributedWorkersEnsemble.java:250) ~[io.openmessaging.benchmark-benchmark-framework-0.0.1-SNAPSHOT.jar:?] at io.openmessaging.benchmark.WorkloadGenerator.printAndCollectStats(WorkloadGenerator.java:525) ~[io.openmessaging.benchmark-benchmark-framework-0.0.1-SNAPSHOT.jar:?] at io.openmessaging.benchmark.WorkloadGenerator.run(WorkloadGenerator.java:143) ~[io.openmessaging.benchmark-benchmark-framework-0.0.1-SNAPSHOT.jar:?] at io.openmessaging.benchmark.Benchmark.lambda$main$0(Benchmark.java:173) ~[io.openmessaging.benchmark-benchmark-framework-0.0.1-SNAPSHOT.jar:?] at java.util.ArrayList.forEach(ArrayList.java:1541) ~[?:?] at io.openmessaging.benchmark.Benchmark.lambda$main$1(Benchmark.java:153) ~[io.openmessaging.benchmark-benchmark-framework-0.0.1-SNAPSHOT.jar:?] at java.util.TreeMap.forEach(TreeMap.java:1002) [?:?] at io.openmessaging.benchmark.Benchmark.main(Benchmark.java:151) [io.openmessaging.benchmark-benchmark-framework-0.0.1-SNAPSHOT.jar:?] Caused by: java.util.concurrent.CompletionException: java.lang.RuntimeException: com.fasterxml.jackson.databind.JsonMappingException: java.lang.IllegalArgumentException: The buffer does not contain the indicated payload amount (through reference chain: io.openmessaging.benchmark.worker.commands.CumulativeLatencies["publishLatency"]) ......

travisdowns commented 1 year ago

This occurs when the fixed 1MB histogram serialization buffer is not large enough after compression: the histogram encoding is silently truncated in that case:

https://github.com/HdrHistogram/HdrHistogram/issues/201

Easy fix is to grow the buffer dynamically when we notice the encoded histogram filled the entire buffer.