Closed geekity closed 7 years ago
Thanks for the kind words @geekity!
Now, concerning your issue, I'm not entirely sure what the reason is (since it's sorta hard to reproduce your exact environment) but here are two things OTTOMH that you could check:
test-kafka10-test-eu-west-1/brokers/topics/test
is the data there? I'm not entirely sure but to me the two screen shots look like from the same cluster?Can you pls first check if the backup process actually created the data on disk correctly and then we take it from there?
Thanks for the quick response! :)
With regards to your suggestion:
1) I checked the dump of data from my Exhibitor cluster 1 and under test-kafka10-test-eu-west-1/brokers/topics/test
there is only a partitions folder and no content file (that's where the znode data is saved, yea?). I can see how the 2 screen shots look very similar (since the second is a replica of the first) but they're definitely 2 different clusters.
2) The second zookeeper cluster is empty when I try to copy into it. I just spun all the infrastructure up for testing. I don't think the overwrite safety feature can apply to this case so. Good thing to know about it though, I hadn't noticed it earlier.
With regards to getting the dump of data from Exhibitor cluster 1 and storing it to local, this was the output from the command:
➜ burry -e 172.31.128.160:2181 -t local
INFO[0000] Selected operation: BACKUP func=main
INFO[0000] My config: {InfraService:zk Endpoint:172.31.128.160:2181 StorageTarget:local Creds:{StorageTargetEndpoint: Params:[]}} func=main
2017/05/17 17:16:06 Connected to 172.31.128.160:2181
2017/05/17 17:16:06 Authenticated: id=3917030430742675462, timeout=4000
2017/05/17 17:16:06 Re-submitting `0` credentials after reconnect
INFO[0001] Operation successfully completed. The snapshot ID is: 1495037766 func=main
➜ ls
1495037766.zip
If you'd like I can upload the full dump for you, it was a test cluster with nothing else in it.
I'd say for reproducing this problem you probably don't need to run all the extra replicas, just having a Kafka (I used Kafka 0.10.2.1) and Zookeeper and making a dummy test topic should be sufficient. The data missing between the origin and copy is the Kafka partition and replica assignment to different Kafka brokers for the given topic.
Thanks for the detailed report here. Hmmm, interesting … now, I'm still not entirely sure what the cause might be. Can you please try to run it with debug-level logging enabled, that is, DEBUG=true burry -e 172.31.128.160:2181 -t local
so that we can have a closer look at what's going on?
For the sake of completeness, the hunch I have is that it's an ephemeral znode that doesn't get reaped properly (which I thought would work correctly, so if that's true then I need to check if it's an issue with the ZK package I'm using or if it's me using it incorrectly).
Here's the command output.
➜ DEBUG=true burry -e 172.31.170.190:2181 -t local
INFO[0000] Selected operation: BACKUP func=main
INFO[0000] My config: {InfraService:zk Endpoint:172.31.170.190:2181 StorageTarget:local Creds:{StorageTargetEndpoint: Params:[]}} func=main
DEBU[0000] On node / func=visitZK
2017/05/18 18:03:55 Connected to 172.31.170.190:2181
2017/05/18 18:03:55 Authenticated: id=4205267394443673606, timeout=4000
2017/05/18 18:03:55 Re-submitting `0` credentials after reconnect
DEBU[0000] / has 2 children func=visitZK
DEBU[0000] Next visiting child /zookeeper func=visitZK
DEBU[0000] On node /zookeeper func=visitZK
DEBU[0000] /zookeeper has 1 children func=visitZK
DEBU[0000] Next visiting child /zookeeper/quota func=visitZK
DEBU[0000] On node /zookeeper/quota func=visitZK
DEBU[0000] /zookeeper/quota has 0 children func=visitZK
DEBU[0000] Stored /zookeeper/quota in /Users/nhanzlikova/training/burry/1495127035/zookeeper/quota with 0 bytes func=store
DEBU[0000] Next visiting child /test-kafka10-test-eu-west-1 func=visitZK
DEBU[0000] On node /test-kafka10-test-eu-west-1 func=visitZK
DEBU[0000] /test-kafka10-test-eu-west-1 has 8 children func=visitZK
DEBU[0000] Next visiting child /test-kafka10-test-eu-west-1/consumers func=visitZK
DEBU[0000] On node /test-kafka10-test-eu-west-1/consumers func=visitZK
DEBU[0000] /test-kafka10-test-eu-west-1/consumers has 0 children func=visitZK
DEBU[0000] Stored /test-kafka10-test-eu-west-1/consumers in /Users/nhanzlikova/training/burry/1495127035/test-kafka10-test-eu-west-1/consumers with 0 bytes func=store
DEBU[0000] Next visiting child /test-kafka10-test-eu-west-1/cluster func=visitZK
DEBU[0000] On node /test-kafka10-test-eu-west-1/cluster func=visitZK
DEBU[0000] /test-kafka10-test-eu-west-1/cluster has 1 children func=visitZK
DEBU[0000] Next visiting child /test-kafka10-test-eu-west-1/cluster/id func=visitZK
DEBU[0000] On node /test-kafka10-test-eu-west-1/cluster/id func=visitZK
DEBU[0000] /test-kafka10-test-eu-west-1/cluster/id has 0 children func=visitZK
DEBU[0000] Stored /test-kafka10-test-eu-west-1/cluster/id in /Users/nhanzlikova/training/burry/1495127035/test-kafka10-test-eu-west-1/cluster/id with 45 bytes func=store
DEBU[0000] Next visiting child /test-kafka10-test-eu-west-1/config func=visitZK
DEBU[0000] On node /test-kafka10-test-eu-west-1/config func=visitZK
DEBU[0000] /test-kafka10-test-eu-west-1/config has 3 children func=visitZK
DEBU[0000] Next visiting child /test-kafka10-test-eu-west-1/config/topics func=visitZK
DEBU[0000] On node /test-kafka10-test-eu-west-1/config/topics func=visitZK
DEBU[0000] /test-kafka10-test-eu-west-1/config/topics has 1 children func=visitZK
DEBU[0000] Next visiting child /test-kafka10-test-eu-west-1/config/topics/test func=visitZK
DEBU[0000] On node /test-kafka10-test-eu-west-1/config/topics/test func=visitZK
DEBU[0000] /test-kafka10-test-eu-west-1/config/topics/test has 0 children func=visitZK
DEBU[0000] Stored /test-kafka10-test-eu-west-1/config/topics/test in /Users/nhanzlikova/training/burry/1495127035/test-kafka10-test-eu-west-1/config/topics/test with 25 bytes func=store
DEBU[0000] Next visiting child /test-kafka10-test-eu-west-1/config/clients func=visitZK
DEBU[0000] On node /test-kafka10-test-eu-west-1/config/clients func=visitZK
DEBU[0000] /test-kafka10-test-eu-west-1/config/clients has 0 children func=visitZK
DEBU[0000] Stored /test-kafka10-test-eu-west-1/config/clients in /Users/nhanzlikova/training/burry/1495127035/test-kafka10-test-eu-west-1/config/clients with 0 bytes func=store
DEBU[0000] Next visiting child /test-kafka10-test-eu-west-1/config/changes func=visitZK
DEBU[0000] On node /test-kafka10-test-eu-west-1/config/changes func=visitZK
DEBU[0000] /test-kafka10-test-eu-west-1/config/changes has 0 children func=visitZK
DEBU[0000] Stored /test-kafka10-test-eu-west-1/config/changes in /Users/nhanzlikova/training/burry/1495127035/test-kafka10-test-eu-west-1/config/changes with 0 bytes func=store
DEBU[0000] Next visiting child /test-kafka10-test-eu-west-1/controller func=visitZK
DEBU[0000] On node /test-kafka10-test-eu-west-1/controller func=visitZK
DEBU[0000] /test-kafka10-test-eu-west-1/controller has 0 children func=visitZK
DEBU[0000] Stored /test-kafka10-test-eu-west-1/controller in /Users/nhanzlikova/training/burry/1495127035/test-kafka10-test-eu-west-1/controller with 54 bytes func=store
DEBU[0000] Next visiting child /test-kafka10-test-eu-west-1/isr_change_notification func=visitZK
DEBU[0000] On node /test-kafka10-test-eu-west-1/isr_change_notification func=visitZK
DEBU[0000] /test-kafka10-test-eu-west-1/isr_change_notification has 0 children func=visitZK
DEBU[0000] Stored /test-kafka10-test-eu-west-1/isr_change_notification in /Users/nhanzlikova/training/burry/1495127035/test-kafka10-test-eu-west-1/isr_change_notification with 0 bytes func=store
DEBU[0000] Next visiting child /test-kafka10-test-eu-west-1/admin func=visitZK
DEBU[0000] On node /test-kafka10-test-eu-west-1/admin func=visitZK
DEBU[0000] /test-kafka10-test-eu-west-1/admin has 1 children func=visitZK
DEBU[0000] Next visiting child /test-kafka10-test-eu-west-1/admin/delete_topics func=visitZK
DEBU[0000] On node /test-kafka10-test-eu-west-1/admin/delete_topics func=visitZK
DEBU[0000] /test-kafka10-test-eu-west-1/admin/delete_topics has 0 children func=visitZK
DEBU[0000] Stored /test-kafka10-test-eu-west-1/admin/delete_topics in /Users/nhanzlikova/training/burry/1495127035/test-kafka10-test-eu-west-1/admin/delete_topics with 0 bytes func=store
DEBU[0000] Next visiting child /test-kafka10-test-eu-west-1/brokers func=visitZK
DEBU[0000] On node /test-kafka10-test-eu-west-1/brokers func=visitZK
DEBU[0000] /test-kafka10-test-eu-west-1/brokers has 3 children func=visitZK
DEBU[0000] Next visiting child /test-kafka10-test-eu-west-1/brokers/seqid func=visitZK
DEBU[0000] On node /test-kafka10-test-eu-west-1/brokers/seqid func=visitZK
DEBU[0000] /test-kafka10-test-eu-west-1/brokers/seqid has 0 children func=visitZK
DEBU[0000] Stored /test-kafka10-test-eu-west-1/brokers/seqid in /Users/nhanzlikova/training/burry/1495127035/test-kafka10-test-eu-west-1/brokers/seqid with 0 bytes func=store
DEBU[0000] Next visiting child /test-kafka10-test-eu-west-1/brokers/topics func=visitZK
DEBU[0000] On node /test-kafka10-test-eu-west-1/brokers/topics func=visitZK
DEBU[0000] /test-kafka10-test-eu-west-1/brokers/topics has 1 children func=visitZK
DEBU[0000] Next visiting child /test-kafka10-test-eu-west-1/brokers/topics/test func=visitZK
DEBU[0000] On node /test-kafka10-test-eu-west-1/brokers/topics/test func=visitZK
DEBU[0000] /test-kafka10-test-eu-west-1/brokers/topics/test has 1 children func=visitZK
DEBU[0000] Next visiting child /test-kafka10-test-eu-west-1/brokers/topics/test/partitions func=visitZK
DEBU[0000] On node /test-kafka10-test-eu-west-1/brokers/topics/test/partitions func=visitZK
DEBU[0000] /test-kafka10-test-eu-west-1/brokers/topics/test/partitions has 6 children func=visitZK
DEBU[0000] Next visiting child /test-kafka10-test-eu-west-1/brokers/topics/test/partitions/3 func=visitZK
DEBU[0000] On node /test-kafka10-test-eu-west-1/brokers/topics/test/partitions/3 func=visitZK
DEBU[0000] /test-kafka10-test-eu-west-1/brokers/topics/test/partitions/3 has 1 children func=visitZK
DEBU[0000] Next visiting child /test-kafka10-test-eu-west-1/brokers/topics/test/partitions/3/state func=visitZK
DEBU[0000] On node /test-kafka10-test-eu-west-1/brokers/topics/test/partitions/3/state func=visitZK
DEBU[0000] /test-kafka10-test-eu-west-1/brokers/topics/test/partitions/3/state has 0 children func=visitZK
DEBU[0000] Stored /test-kafka10-test-eu-west-1/brokers/topics/test/partitions/3/state in /Users/nhanzlikova/training/burry/1495127035/test-kafka10-test-eu-west-1/brokers/topics/test/partitions/3/state with 76 bytes func=store
DEBU[0000] Next visiting child /test-kafka10-test-eu-west-1/brokers/topics/test/partitions/2 func=visitZK
DEBU[0000] On node /test-kafka10-test-eu-west-1/brokers/topics/test/partitions/2 func=visitZK
DEBU[0000] /test-kafka10-test-eu-west-1/brokers/topics/test/partitions/2 has 1 children func=visitZK
DEBU[0000] Next visiting child /test-kafka10-test-eu-west-1/brokers/topics/test/partitions/2/state func=visitZK
DEBU[0000] On node /test-kafka10-test-eu-west-1/brokers/topics/test/partitions/2/state func=visitZK
DEBU[0000] /test-kafka10-test-eu-west-1/brokers/topics/test/partitions/2/state has 0 children func=visitZK
DEBU[0000] Stored /test-kafka10-test-eu-west-1/brokers/topics/test/partitions/2/state in /Users/nhanzlikova/training/burry/1495127035/test-kafka10-test-eu-west-1/brokers/topics/test/partitions/2/state with 76 bytes func=store
DEBU[0000] Next visiting child /test-kafka10-test-eu-west-1/brokers/topics/test/partitions/1 func=visitZK
DEBU[0000] On node /test-kafka10-test-eu-west-1/brokers/topics/test/partitions/1 func=visitZK
DEBU[0000] /test-kafka10-test-eu-west-1/brokers/topics/test/partitions/1 has 1 children func=visitZK
DEBU[0000] Next visiting child /test-kafka10-test-eu-west-1/brokers/topics/test/partitions/1/state func=visitZK
DEBU[0000] On node /test-kafka10-test-eu-west-1/brokers/topics/test/partitions/1/state func=visitZK
DEBU[0000] /test-kafka10-test-eu-west-1/brokers/topics/test/partitions/1/state has 0 children func=visitZK
DEBU[0000] Stored /test-kafka10-test-eu-west-1/brokers/topics/test/partitions/1/state in /Users/nhanzlikova/training/burry/1495127035/test-kafka10-test-eu-west-1/brokers/topics/test/partitions/1/state with 76 bytes func=store
DEBU[0000] Next visiting child /test-kafka10-test-eu-west-1/brokers/topics/test/partitions/0 func=visitZK
DEBU[0000] On node /test-kafka10-test-eu-west-1/brokers/topics/test/partitions/0 func=visitZK
DEBU[0000] /test-kafka10-test-eu-west-1/brokers/topics/test/partitions/0 has 1 children func=visitZK
DEBU[0000] Next visiting child /test-kafka10-test-eu-west-1/brokers/topics/test/partitions/0/state func=visitZK
DEBU[0000] On node /test-kafka10-test-eu-west-1/brokers/topics/test/partitions/0/state func=visitZK
DEBU[0000] /test-kafka10-test-eu-west-1/brokers/topics/test/partitions/0/state has 0 children func=visitZK
DEBU[0000] Stored /test-kafka10-test-eu-west-1/brokers/topics/test/partitions/0/state in /Users/nhanzlikova/training/burry/1495127035/test-kafka10-test-eu-west-1/brokers/topics/test/partitions/0/state with 76 bytes func=store
DEBU[0000] Next visiting child /test-kafka10-test-eu-west-1/brokers/topics/test/partitions/5 func=visitZK
DEBU[0000] On node /test-kafka10-test-eu-west-1/brokers/topics/test/partitions/5 func=visitZK
DEBU[0000] /test-kafka10-test-eu-west-1/brokers/topics/test/partitions/5 has 1 children func=visitZK
DEBU[0000] Next visiting child /test-kafka10-test-eu-west-1/brokers/topics/test/partitions/5/state func=visitZK
DEBU[0000] On node /test-kafka10-test-eu-west-1/brokers/topics/test/partitions/5/state func=visitZK
DEBU[0000] /test-kafka10-test-eu-west-1/brokers/topics/test/partitions/5/state has 0 children func=visitZK
DEBU[0000] Stored /test-kafka10-test-eu-west-1/brokers/topics/test/partitions/5/state in /Users/nhanzlikova/training/burry/1495127035/test-kafka10-test-eu-west-1/brokers/topics/test/partitions/5/state with 76 bytes func=store
DEBU[0000] Next visiting child /test-kafka10-test-eu-west-1/brokers/topics/test/partitions/4 func=visitZK
DEBU[0000] On node /test-kafka10-test-eu-west-1/brokers/topics/test/partitions/4 func=visitZK
DEBU[0000] /test-kafka10-test-eu-west-1/brokers/topics/test/partitions/4 has 1 children func=visitZK
DEBU[0000] Next visiting child /test-kafka10-test-eu-west-1/brokers/topics/test/partitions/4/state func=visitZK
DEBU[0000] On node /test-kafka10-test-eu-west-1/brokers/topics/test/partitions/4/state func=visitZK
DEBU[0000] /test-kafka10-test-eu-west-1/brokers/topics/test/partitions/4/state has 0 children func=visitZK
DEBU[0000] Stored /test-kafka10-test-eu-west-1/brokers/topics/test/partitions/4/state in /Users/nhanzlikova/training/burry/1495127035/test-kafka10-test-eu-west-1/brokers/topics/test/partitions/4/state with 76 bytes func=store
DEBU[0000] Next visiting child /test-kafka10-test-eu-west-1/brokers/ids func=visitZK
DEBU[0000] On node /test-kafka10-test-eu-west-1/brokers/ids func=visitZK
DEBU[0000] /test-kafka10-test-eu-west-1/brokers/ids has 6 children func=visitZK
DEBU[0000] Next visiting child /test-kafka10-test-eu-west-1/brokers/ids/3 func=visitZK
DEBU[0000] On node /test-kafka10-test-eu-west-1/brokers/ids/3 func=visitZK
DEBU[0000] /test-kafka10-test-eu-west-1/brokers/ids/3 has 0 children func=visitZK
DEBU[0000] Stored /test-kafka10-test-eu-west-1/brokers/ids/3 in /Users/nhanzlikova/training/burry/1495127035/test-kafka10-test-eu-west-1/brokers/ids/3 with 278 bytes func=store
DEBU[0000] Next visiting child /test-kafka10-test-eu-west-1/brokers/ids/2 func=visitZK
DEBU[0000] On node /test-kafka10-test-eu-west-1/brokers/ids/2 func=visitZK
DEBU[0000] /test-kafka10-test-eu-west-1/brokers/ids/2 has 0 children func=visitZK
DEBU[0000] Stored /test-kafka10-test-eu-west-1/brokers/ids/2 in /Users/nhanzlikova/training/burry/1495127035/test-kafka10-test-eu-west-1/brokers/ids/2 with 280 bytes func=store
DEBU[0000] Next visiting child /test-kafka10-test-eu-west-1/brokers/ids/1 func=visitZK
DEBU[0000] On node /test-kafka10-test-eu-west-1/brokers/ids/1 func=visitZK
DEBU[0000] /test-kafka10-test-eu-west-1/brokers/ids/1 has 0 children func=visitZK
DEBU[0000] Stored /test-kafka10-test-eu-west-1/brokers/ids/1 in /Users/nhanzlikova/training/burry/1495127035/test-kafka10-test-eu-west-1/brokers/ids/1 with 280 bytes func=store
DEBU[0000] Next visiting child /test-kafka10-test-eu-west-1/brokers/ids/6 func=visitZK
DEBU[0000] On node /test-kafka10-test-eu-west-1/brokers/ids/6 func=visitZK
DEBU[0000] /test-kafka10-test-eu-west-1/brokers/ids/6 has 0 children func=visitZK
DEBU[0000] Stored /test-kafka10-test-eu-west-1/brokers/ids/6 in /Users/nhanzlikova/training/burry/1495127035/test-kafka10-test-eu-west-1/brokers/ids/6 with 280 bytes func=store
DEBU[0000] Next visiting child /test-kafka10-test-eu-west-1/brokers/ids/5 func=visitZK
DEBU[0000] On node /test-kafka10-test-eu-west-1/brokers/ids/5 func=visitZK
DEBU[0000] /test-kafka10-test-eu-west-1/brokers/ids/5 has 0 children func=visitZK
DEBU[0000] Stored /test-kafka10-test-eu-west-1/brokers/ids/5 in /Users/nhanzlikova/training/burry/1495127035/test-kafka10-test-eu-west-1/brokers/ids/5 with 280 bytes func=store
DEBU[0000] Next visiting child /test-kafka10-test-eu-west-1/brokers/ids/4 func=visitZK
DEBU[0000] On node /test-kafka10-test-eu-west-1/brokers/ids/4 func=visitZK
DEBU[0000] /test-kafka10-test-eu-west-1/brokers/ids/4 has 0 children func=visitZK
DEBU[0000] Stored /test-kafka10-test-eu-west-1/brokers/ids/4 in /Users/nhanzlikova/training/burry/1495127035/test-kafka10-test-eu-west-1/brokers/ids/4 with 280 bytes func=store
DEBU[0000] Next visiting child /test-kafka10-test-eu-west-1/controller_epoch func=visitZK
DEBU[0000] On node /test-kafka10-test-eu-west-1/controller_epoch func=visitZK
DEBU[0000] /test-kafka10-test-eu-west-1/controller_epoch has 0 children func=visitZK
DEBU[0000] Stored /test-kafka10-test-eu-west-1/controller_epoch in /Users/nhanzlikova/training/burry/1495127035/test-kafka10-test-eu-west-1/controller_epoch with 1 bytes func=store
DEBU[0000] Added metadata to /Users/nhanzlikova/training/burry/1495127035 func=addmeta
DEBU[0000] 1495127035/.burrymeta func=arch
DEBU[0000] 1495127035/test-kafka10-test-eu-west-1/admin/delete_topics/content func=arch
DEBU[0000] 1495127035/test-kafka10-test-eu-west-1/brokers/ids/1/content func=arch
DEBU[0000] 1495127035/test-kafka10-test-eu-west-1/brokers/ids/2/content func=arch
DEBU[0000] 1495127035/test-kafka10-test-eu-west-1/brokers/ids/3/content func=arch
DEBU[0000] 1495127035/test-kafka10-test-eu-west-1/brokers/ids/4/content func=arch
DEBU[0000] 1495127035/test-kafka10-test-eu-west-1/brokers/ids/5/content func=arch
DEBU[0000] 1495127035/test-kafka10-test-eu-west-1/brokers/ids/6/content func=arch
DEBU[0000] 1495127035/test-kafka10-test-eu-west-1/brokers/seqid/content func=arch
DEBU[0000] 1495127035/test-kafka10-test-eu-west-1/brokers/topics/test/partitions/0/state/content func=arch
DEBU[0000] 1495127035/test-kafka10-test-eu-west-1/brokers/topics/test/partitions/1/state/content func=arch
DEBU[0000] 1495127035/test-kafka10-test-eu-west-1/brokers/topics/test/partitions/2/state/content func=arch
DEBU[0000] 1495127035/test-kafka10-test-eu-west-1/brokers/topics/test/partitions/3/state/content func=arch
DEBU[0000] 1495127035/test-kafka10-test-eu-west-1/brokers/topics/test/partitions/4/state/content func=arch
DEBU[0000] 1495127035/test-kafka10-test-eu-west-1/brokers/topics/test/partitions/5/state/content func=arch
DEBU[0000] 1495127035/test-kafka10-test-eu-west-1/cluster/id/content func=arch
DEBU[0000] 1495127035/test-kafka10-test-eu-west-1/config/changes/content func=arch
DEBU[0000] 1495127035/test-kafka10-test-eu-west-1/config/clients/content func=arch
DEBU[0000] 1495127035/test-kafka10-test-eu-west-1/config/topics/test/content func=arch
DEBU[0000] 1495127035/test-kafka10-test-eu-west-1/consumers/content func=arch
DEBU[0000] 1495127035/test-kafka10-test-eu-west-1/controller/content func=arch
DEBU[0000] 1495127035/test-kafka10-test-eu-west-1/controller_epoch/content func=arch
DEBU[0000] 1495127035/test-kafka10-test-eu-west-1/isr_change_notification/content func=arch
DEBU[0000] 1495127035/zookeeper/quota/content func=arch
DEBU[0000] Backup available in /Users/nhanzlikova/training/burry/1495127035.zip func=arch
INFO[0000] Operation successfully completed. The snapshot ID is: 1495127035 func=main
This was actually run on a replicated set-up described above, since I had deleted my original clusters due to running costs. I had set up the configuration and topic the same as last time, although the exact data expected in the test
znode is slightly different. To make sure everything is still the same from dump perspective, I checked the dump path test-kafka10-test-eu-west-1/brokers/topics/test
, which still only contains the partitions
folder and no data.
Thank you! This is really strange since the detailed output tells us that apparently there was data, so properly reaped, for example here:
Stored /test-kafka10-test-eu-west-1/brokers/topics/test/partitions/3/state in /Users/nhanzlikova/training/burry/1495127035/test-kafka10-test-eu-west-1/brokers/topics/test/partitions/3/state
Just to make 100% sure and to confirm: when you look into the file 1495127035.zip
then there is no sub-dir /test-kafka10-test-eu-west-1/brokers/topics/test/partitions/3/state
, yes?
There is a sub-dir /test-kafka10-test-eu-west-1/brokers/topics/test/partitions/3/state
, which has content
. I think the problem is with /test-kafka10-test-eu-west-1/brokers/topics/test
, which should have a content
for the data in the znode but only has a sub-dir partitions
. Could the problem be that the znode normally stores data as well as acts as a parent to more znodes?
OMG, you're absolutely correct! Only leaf nodes are reaped (== stored) , see here but that can certainly be changed. I'm not 100% sure why I had that policy in the first place but maybe it's time to review that decision?
Well, Kafka uses the top level topic node for storing the partition-to-broker mappings, and the leaf nodes for the current ISR data. Without the partition-to-broker mapping the data on the brokers is pretty inaccessible (unless you manually restore this mapping, which would probably be pretty tricky) afaik. In my case, Kafka coordination is my main use case for Zookeeper, so if you do decide to revisit the policy, I'd be all for it :D
Yeah, I can't think of any reason now that this should not be valid use case (or cause some data corruption) so let's do it. Now, I'm on travel from tomorrow on, heading to Sydney for a week and hence it would be helpful if you could give me a bit of an indication how urgent and how important this is for you. I'm more than happy to put a few cycles into it, essentially working on a branch that you can try out and when we're happy make this the master branch.
It's not particularly urgent, so please don't worry on my account. I was looking for a tool to use at work for backing up my Zookeeper and this seems like a really nice one, but it was more preventative rather than urgent. If you're busy and you'd like I can have a crack at making a PR with it, although it would be my first time with Go, but if you're not taking PRs I'm also perfectly happy leaving it in your hands.
Got it. It would be awesome to get an PR on this! IMO it theoretically should be as simple as adding fn(…)
somewhere in this fragment. And: no worries, Go is simple to read and I suppose rather simple to write as well ;)
Hi,
Great tool, by the way. I've been experimenting with it a little to replicate Kafka config stored in Zookeeper (with Exhibitor), to basically allow a Kafka cluster to move onto new Zookeeper without losing information about topic partition assignment.
At the moment I'm running into an issue where the structure of the Kafka data being stored is being replicated correctly but the data in the topic znode is not copied.
My set up is:
Procedure:
test
with 6 partitions and replication factor 3. This results in topic znode shown in the screen shot below, with the partition to Kafka broker mapping as data on the node.burry -e xx.xx.xx.xx:2181 -t local
burry -o restore -e yy.yy.yy.yy:2181 -t local -s <snapID>
Is there anything obviously wrong with my usage here, or something else I'm missing? Thanks! :)