Unable to restore Zookeeper in Kubernetes while running Kafka

[ZZZanoY]

Hey Great tool. I just have one issue with it. I am running a Kubernetes cluster with a Zookeeper and Kafka. Both of these are running single instance clusters. Creating the backup from zookeeper and pushing it to s3 works flawlessly, but the problem I have is that I am unable to restore a fresh zookeeper from that backup when kafka is running. This is what I ran:

burry --endpoint=localhost:2181 --operation=restore --target=s3 --snapshot=1534148038 --credentials=s3.amazonaws.com,ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID,SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY,BUCKET=Example_bucket

DEBU[0005] Visited /brokers/topics                       func=visitZKReverse
DEBU[0005] Attempting to insert /brokers/topics/ExampleTopic-Source as leaf znode  func=visitZKReverse
INFO[0010] Restored /brokers/topics/ExampleTopic-Source  func=visitZKReverse
DEBU[0010] Value: {"version":1,"partitions":{"12":[0],"8":[0],"19":[0],"23":[0],"4":[0],"15":[0],"11":[0],"9":[0],"22":[0],"26":[0],"13":[0],"24":[0],"16":[0],"5":[0],"10":[0],"21":[0],"6":[0],"1":[0],"17":[0],"25":[0],"14":[0],"31":[0],"0":[0],"20":[0],"27":[0],"2":[0],"18":[0],"30":[0],"7":[0],"29":[0],"3":[0],"28":[0]}}  func=visitZKReverse
DEBU[0010] Visited /brokers/topics/ExampleTopic-Source  func=visitZKReverse
DEBU[0010] Visited /brokers/topics/ExampleTopic-Source/content  func=visitZKReverse
DEBU[0010] Attempting to insert /brokers/topics/ExampleTopic-Source/partitions as leaf znode  func=visitZKReverse
ERRO[0011] zk: node already exists:/brokers/topics/ExampleTopic-Source/partitions  func=visitZKReverse
ERRO[0011] zk: node already exists                       func=restoreZK
ERRO[0012] Operation completed with error(s).            func=main

From what I can tell it seems like burry first successfully restores /brokers/topics/ExampleTopic-Source but before it manages to restore brokers/topics/ExampleTopic-Source/partitions, kafka has already created that node.

Is this a known limitation or am I doing something wrong? Thanks!

[mhausenblas]

Sorry for the delay @ZZZanoY I'm on vacation ATM, will have a look next week.

[JoseAntonioRodriguez]

Hi @mhausenblas, I'm running into the same problem. Any update on this?

[mhausenblas]

Oops, that one fell between the cracks. Hmmm, no idea TBH, not sure how I can re-create it.

[JoseAntonioRodriguez]

In my understanding Burry doesn't restore already existing znodes (it doesn't overwrite). But in this case it seems like it fails because it thinks a znode doesn't exist, but when it is going to create it, it does exist (in our case because Kafka has eventually created it, I guess). What about ignoring these kind of errors or providing a flag (like --ignore-existing or something similar) that simply ignore already existing znodes?

[mhausenblas]

Ah! SGTM yes. I'm currently on a tight schedule with little cycles left, anyone wants to send in a patch for it?