Bookkeeper-driver doesn't support distributed mode

Environment:

docker-compose:1.23.2
zookeeper:3.5.5
bookkeeper:4.8.2
distributedlog-core-shaded:4.8.2

When i use 1 master and 4 worker nodes with producerRate=1000, i see only 500 , i.e. only 1st producer and 1st consumer is worked.

Start openmessaging: docker exec -i builds_bk-omb_1 /opt/openmessaging-benchmark/bin/benchmark -d driver-bookkeeper/bookkeeper.yaml -w 'http://builds_bk-omb_2:8383,http://builds_bk-omb_3:8383,http://builds_bk-omb_4:8383,http://builds_bk-omb_5:8383' workloads/1-topic-1-partition-100b.yaml

Test output:

SLF4J: Found binding in [jar:file:/root/.m2/repository/org/apache/logging/log4j/log4j-slf4j-impl/2.9.1/log4j-slf4j-impl-2.9.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/root/.m2/repository/ch/qos/logback/logback-classic/1.0.13/logback-classic-1.0.13.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
14:53:14.018 [main] INFO - Starting benchmark with config: {
  "drivers" : [ "driver-bookkeeper/bookkeeper.yaml" ],
  "workers" : [ "http://builds_bk-omb_2:8383", "http://builds_bk-omb_3:8383", "http://builds_bk-omb_4:8383", "http://builds_bk-omb_5:8383" ],
  "workersFile" : null,
  "workloads" : [ "workloads/1-topic-1-partition-100b.yaml" ]
}
14:53:14.064 [main] INFO - Workloads: {
  "1-topic-1-partition-100b" : {
    "name" : "1 topic / 1 partition / 100b",
    "topics" : 1,
    "partitionsPerTopic" : 1,
    "keyDistributor" : "NO_KEY",
    "messageSize" : 100,
    "payloadFile" : "payload/payload-100b.data",
    "subscriptionsPerTopic" : 1,
    "producersPerTopic" : 1,
    "consumerPerSubscription" : 1,
    "producerRate" : 1000,
    "consumerBacklogSizeGB" : 0,
    "testDurationMinutes" : 1
  }
}
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by io.netty.util.internal.ReflectionUtil (file:/root/.m2/repository/io/netty/netty-all/4.1.12.Final/netty-all-4.1.12.Final.jar) to constructor java.nio.DirectByteBuffer(long,
int)
WARNING: Please consider reporting this to the maintainers of io.netty.util.internal.ReflectionUtil
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
14:53:14.399 [main] INFO - Workers list - producers: [http://builds_bk-omb_2:8383, http://builds_bk-omb_3:8383]
14:53:14.399 [main] INFO - Workers list - consumers: [http://builds_bk-omb_4:8383, http://builds_bk-omb_5:8383]
14:53:14.405 [main] INFO - --------------- WORKLOAD : 1 topic / 1 partition / 100b --- DRIVER : BookKeeper---------------
14:53:14.965 [main] INFO - Created 1 topics in 78.950927 ms
14:53:15.022 [main] INFO - Created 1 consumers in 51.625864 ms
4:53:15.129 [main] INFO - Created 1 producers in 106.395072 ms
14:53:15.129 [main] INFO - Waiting for consumers to be ready
14:53:15.931 [main] INFO - All consumers are ready
14:53:15.932 [main] INFO - ----- Starting warm-up traffic ------
14:53:26.208 [main] INFO - Pub rate   538.7 msg/s /  0.1 Mb/s | Cons rate   500.1 msg/s /  0.0 Mb/s | Backlog:  0.4 K | Pub Latency (ms) avg:  6.3 - 50%:  3.3 - 99%: 48.7 - 99.9%: 50.1 - Max: 50.3
14:53:36.387 [main] INFO - Pub rate   502.2 msg/s /  0.0 Mb/s | Cons rate   541.7 msg/s /  0.1 Mb/s | Backlog: -0.0 K | Pub Latency (ms) avg:  3.1 - 50%:  3.0 - 99%:  8.6 - 99.9%: 15.1 - Max: 22.1
14:53:46.589 [main] INFO - Pub rate   498.6 msg/s /  0.0 Mb/s | Cons rate   461.3 msg/s /  0.0 Mb/s | Backlog:  0.4 K | Pub Latency (ms) avg:  2.9 - 50%:  3.0 - 99%:  6.3 - 99.9%:  8.1 - Max:  9.0
14:53:56.783 [main] INFO - Pub rate   500.7 msg/s /  0.0 Mb/s | Cons rate   494.8 msg/s /  0.0 Mb/s | Backlog:  0.4 K | Pub Latency (ms) avg:  3.0 - 50%:  3.0 - 99%:  7.0 - 99.9%: 14.2 - Max: 18.3
14:54:06.992 [main] INFO - Pub rate   499.7 msg/s /  0.0 Mb/s | Cons rate   541.3 msg/s /  0.1 Mb/s | Backlog:  0.0 K | Pub Latency (ms) avg:  2.8 - 50%:  3.0 - 99%:  5.4 - 99.9%:  7.8 - Max: 10.0
14:54:17.216 [main] INFO - Pub rate   498.2 msg/s /  0.0 Mb/s | Cons rate   500.1 msg/s /  0.0 Mb/s | Backlog: -0.0 K | Pub Latency (ms) avg:  2.9 - 50%:  3.0 - 99%:  5.8 - 99.9%: 11.0 - Max: 19.2
14:54:17.393 [main] INFO - ----- Aggregated Pub Latency (ms) avg:  3.5 - 50%:  3.0 - 95%:  5.0 - 99%: 30.9 - 99.9%: 49.4 - 99.99%: 50.1 - Max: 50.3
14:54:17.531 [main] INFO - ----- Starting benchmark traffic ------
14:54:27.659 [main] INFO - Pub rate   520.8 msg/s /  0.0 Mb/s | Cons rate   506.3 msg/s /  0.0 Mb/s | Backlog:  0.1 K | Pub Latency (ms) avg:  2.8 - 50%:  3.0 - 99%:  5.7 - 99.9%:  8.5 - Max: 14.1
14:54:37.785 [main] INFO - Pub rate   500.4 msg/s /  0.0 Mb/s | Cons rate   474.5 msg/s /  0.0 Mb/s | Backlog:  0.4 K | Pub Latency (ms) avg:  3.0 - 50%:  3.0 - 99%:  7.3 - 99.9%: 10.4 - Max: 11.2
14:54:47.944 [main] INFO - Pub rate   500.3 msg/s /  0.0 Mb/s | Cons rate   491.9 msg/s /  0.0 Mb/s | Backlog:  0.5 K | Pub Latency (ms) avg:  2.8 - 50%:  3.0 - 99%:  5.1 - 99.9%:  8.3 - Max: 11.3
14:54:58.081 [main] INFO - Pub rate   498.3 msg/s /  0.0 Mb/s | Cons rate   509.5 msg/s /  0.0 Mb/s | Backlog:  0.4 K | Pub Latency (ms) avg:  2.9 - 50%:  3.0 - 99%:  6.3 - 99.9%: 12.6 - Max: 16.6
14:55:08.225 [main] INFO - Pub rate   499.9 msg/s /  0.0 Mb/s | Cons rate   535.8 msg/s /  0.1 Mb/s | Backlog:  0.0 K | Pub Latency (ms) avg:  2.9 - 50%:  3.0 - 99%:  6.8 - 99.9%: 21.7 - Max: 31.3
14:55:18.347 [main] INFO - Pub rate   501.1 msg/s /  0.0 Mb/s | Cons rate   491.8 msg/s /  0.0 Mb/s | Backlog:  0.1 K | Pub Latency (ms) avg:  2.9 - 50%:  3.0 - 99%:  6.5 - 99.9%:  9.3 - Max: 10.9
14:55:18.480 [main] INFO - ----- Aggregated Pub Latency (ms) avg:  2.9 - 50%:  3.0 - 95%:  4.3 - 99%:  6.3 - 99.9%: 10.5 - 99.99%: 25.0 - Max: 31.3
14:55:18.702 [main] INFO - Writing test result into 1-topic-1-partition-100b-BookKeeper-2020-03-06-14-55-18.json

Docker stats during the test (only 2nd and 4th omb instances are processed)

And another example with 1 master and 6 workers (333 msg/s instead of 1000 msg/s):

openmessaging / benchmark

Bookkeeper-driver doesn't support distributed mode #171