search

apache / bookkeeper

Apache BookKeeper - a scalable, fault tolerant and low latency storage service optimized for append-only workloads
https://bookkeeper.apache.org/
Apache License 2.0
1.91k stars 904 forks source link

When the journal directory is even, the journal data is unevenly distributed #2974

Open 1559924775 opened 2 years ago

1559924775 commented 2 years ago

BUG REPORT

Describe the bug

When the journal directory is configured to be even, the ledger allocated to each journal will be uneven. The root cause is that almost all the ledger IDs generated by the LedgerIdGenerator class are even. The number of ZK temporary order nodes is determined by the cversion of the parent node. Whenever a child node is added or deleted, it will increase by 1. The LedgerIdGenerator class uses ZK's temporary order node to generate the ledgerid, and deletes the node immediately after generation. This way of use causes almost all cversions to be even, resulting in all ledgerids to be even.

To Reproduce

Steps to reproduce the behavior:

  1. Configure in bk_server.conf: jouranlDirectorie=/tmp/bk-txn1,/tmp/bk-txn2
  2. production data to bookie

Expected behavior

Almost all ledgerids are assigned to directory 1

Screenshots

ledgerIdGenerator class : ZKledgerIdGenerator

journal mathutil

Changing the routing of ledgers to journal directories can fix this bug

mathutilnew
Vanlightly commented 2 years ago

Changing the routing without some kind of migration will result in effective data loss. Plus an md5 hash is relatively expensive.

The issue here is that ledger ids are all even. I am seeing this myself which results in only even numbered read threads from being routed to. This problem of even ledger ids affects the selection of journals, ledger storage, read threads and write threads. It is a real problem.

@eolivelli This is a pretty serious problem as we're throwing away a bunch of scaling mechanisms.

dlg99 commented 2 years ago

@Vanlightly @eolivelli I updated OrderedExecutor to better distribute data across the threads, added tests reproducing the issue. please take a look:

https://github.com/apache/bookkeeper/pull/3023

This does not address the fact that generated ledger ids are skewed towards even numbers. This does not fix routing of the journals/data dirs.

shoothzj commented 6 months ago

@dlg99 Can we close this issue or need other works? :)

shoothzj commented 6 months ago

From Andrey: I think it makes sense to split this into bugs for "fix routing of the journals/data dirs" - at least to conform that these aren't skewed towards a subset of directories