search

apache / bookkeeper

Apache BookKeeper - a scalable, fault tolerant and low latency storage service optimized for append-only workloads
https://bookkeeper.apache.org/
Apache License 2.0
1.89k stars 899 forks source link

Data deletion logic doesn't work #2394

Open holmes07 opened 4 years ago

holmes07 commented 4 years ago

BUG REPORT

Describe the bug 1.I got log like this :Space left on device /data11/bookkeeper/ledgers/current : 159412490240, Used space fraction: 0.95951813 > threshold 0.95.

  1. then i set the namespace ttl from 86400 to 900 seconds conf after updated : "message_ttl_in_seconds" : 900, "subscription_expiration_time_minutes" : 0, "retention_policies" : { "retentionTimeInMinutes" : 10, "retentionSizeInMB" : 100 }
  2. but still no log below 20% can be compact. Compaction: entry log usage buckets[10% 20% 30% 40% 50% 60% 70% 80% 90% 100%] = [0, 0, 0, 2, 6, 91, 124, 102, 115, 2888]

4.It seems that the deleted ledger is not passed to the entrylog or ttl does not delete the ledger。iguess, but i don't know why??? Screenshots 企业微信截图_15961867477860 企业微信截图_15961868209711 企业微信截图_15961868977771

Additional context bookeeper version 4.11.0 pulsar version 2.6.0

sijie commented 4 years ago

@holmes07 The minor compaction threshold is 20% (https://github.com/apache/pulsar/blob/master/conf/bookkeeper.conf#L477). It means that minor compaction only kicks in when the active data usage is below 20%. From your log, it seems that the data usage of active ledgers are still above 20%. You might need to check why those ledgers are still active.