'dump' errors when dumping acl or query backups.

[michaelgrosstarget]

Hello,

Got another one (this one might be a little more complicated) - I've built the binary using the latest code (although this happens with 1.6.2 release as well). 'dump' seems to only work for KVs and not acls or queries. Consul is on version 0.9.3.

I perform a backup using the following:

# consul-backinator backup -file kvs.bak -acls acls.bak -queries query.bak -key [SECRET] -token [CONSUL_TOKEN]
2017/11/03 03:42:14 [Success] Backed up 1 keys from / to kvs.bak
2017/11/03 03:42:14 [Success] Backed up 3 ACL tokens from  to acls.bak
2017/11/03 03:42:14 [Success] Backed up 1 query definitions from  to query.bak
Keep your backup and signature files in a safe place.
You will need both to restore your data.

Then dump the KVs (which works as expected)

# consul-backinator dump -file kvs.bak -key [SECRET]
[
  {
    "Key": "consul/backup/lock",
    "CreateIndex": 23,
    "ModifyIndex": 3123,
    "LockIndex": 7,
    "Flags": 0,
    "Value": "",
    "Session": ""
  }
]

Dumping acls complains that consul.bak does not exist (as I specifed a different filename during backup) and -nokv isn't implemented for dump

# consul-backinator dump -acls acls.bak -key [SECRET]
2017/11/03 04:00:24 [Error] Failed to dump data: open consul.bak: no such file or directory

# consul-backinator dump -nokv -acls acls.bak -key [SECRET]
flag provided but not defined: -nokv
Usage: consul-backinator dump [options]

    Dump the contents of a backup file to stdout.

Options:

    -file         Source filename (default: "consul.bak")
    -key          Passphrase for data encryption and signature validation (default: "password")
    -plain        Dump a reduced set of information
    -acls         Specified file is an ACL token backup file
    -queries      Specified file is a prepared query backup file (consider using plain for query files)

Please see documentation on GitHub for a detailed explanation of all options.
https://github.com/myENA/consul-backinator

If I backup kvs to consul.bak and then try to dump acls.bak the result seems to be dependent upon where I place the flags (placing the -acls flag at the end just causes it to dump consul.bak as KVs, seemingly ignoring that the flag is even there):

# consul-backinator backup -file consul.bak -key [SECRET] -token [CONSUL_TOKEN]
2017/11/03 04:04:52 [Success] Backed up 1 keys from / to consul.bak
Keep your backup and signature files in a safe place.
You will need both to restore your data.

# consul-backinator dump -acls acls.bak -key [SECRET]
2017/11/03 04:05:56 [Error] Failed to dump data: gzip: invalid header

# consul-backinator dump -key [SECRET] -acls acls.bak
[
  {
    "Key": "consul/backup/lock",
    "CreateIndex": 23,
    "ModifyIndex": 3564,
    "LockIndex": 1,
    "Flags": 0,
    "Value": "",
    "Session": ""
  }
]

The result is the same when I try dumping queries.

Not sure if I'm doing something fundamentally wrong but any help would be appreciated.

Thank you, Mike

[aaronhurt]

I’m sure that’s correct. I added the dump option as a feature request for KVs before I supported ACLs or prepared queries. I’ll take a look today and try to fix that.

On Nov 2, 2017, at 11:50 PM, michaelgrosstarget notifications@github.com wrote:

Hello,

Got another one (this one might be a little more complicated) - I've built the binary using the latest code (although this happens with 1.6.2 release as well). 'dump' seems to only work for KVs and not acls or queries. Consul is on version 0.9.3.

I perform a backup using the following:

consul-backinator backup -file kvs.bak -acls acls.bak -queries query.bak -key [SECRET] -token [CONSUL_TOKEN]

2017/11/03 03:42:14 [Success] Backed up 1 keys from / to kvs.bak 2017/11/03 03:42:14 [Success] Backed up 3 ACL tokens from to acls.bak 2017/11/03 03:42:14 [Success] Backed up 1 query definitions from to query.bak Keep your backup and signature files in a safe place. You will need both to restore your data. Then dump the KVs (which works as expected)

consul-backinator dump -file kvs.bak -key [SECRET]

[ { "Key": "consul/backup/lock", "CreateIndex": 23, "ModifyIndex": 3123, "LockIndex": 7, "Flags": 0, "Value": "", "Session": "" } ] Dumping acls complains that consul.bak does not exist (as I specifed a different filename during backup) and -nokv isn't implemented for dump

consul-backinator dump -acls acls.bak -key [SECRET]

2017/11/03 04:00:24 [Error] Failed to dump data: open consul.bak: no such file or directory

consul-backinator dump -nokv -acls acls.bak -key [SECRET]

flag provided but not defined: -nokv Usage: consul-backinator dump [options]

Dump the contents of a backup file to stdout.

Options:

-file Source filename (default: "consul.bak") -key Passphrase for data encryption and signature validation (default: "password") -plain Dump a reduced set of information -acls Specified file is an ACL token backup file -queries Specified file is a prepared query backup file (consider using plain for query files)

Please see documentation on GitHub for a detailed explanation of all options. https://github.com/myENA/consul-backinator If I backup kvs to consul.bak and then try to dump acls.bak the result seems to be dependent upon where I place the flags (placing the -acls flag at the end just causes it to dump consul.bak as KVs, seemingly ignoring that the flag is even there):

consul-backinator backup -file consul.bak -key [SECRET] -token [CONSUL_TOKEN]

2017/11/03 04:04:52 [Success] Backed up 1 keys from / to consul.bak Keep your backup and signature files in a safe place. You will need both to restore your data.

consul-backinator dump -acls acls.bak -key [SECRET]

2017/11/03 04:05:56 [Error] Failed to dump data: gzip: invalid header

consul-backinator dump -key [SECRET] -acls acls.bak

[ { "Key": "consul/backup/lock", "CreateIndex": 23, "ModifyIndex": 3564, "LockIndex": 1, "Flags": 0, "Value": "", "Session": "" } ] The result is the same when I try dumping queries.

Not sure if I'm doing something fundamentally wrong but any help would be appreciated.

Thank you, Mike

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

[michaelgrosstarget]

Thanks! It's less critical for us but a nice to have. Btw - awesome product, we continue to choose it over consul snapshot due to the built-in encryption and ability to restore partial kv trees. Great work!

[aaronhurt]

Thank you. Yes, we take regular snapshots for total disaster recovery but still use this tool for moving KVs between hosts and doing partial restores when someone fat fingers something :) ...

So, I think it works as designed it's just not every clear and I probably need to fix the way it works. The -acls and -queries options are just bool flags and don't actually take input. To dump a file you still need to pass the -file option. When dumping out a KV backup you should only need the -file option if you saved your backup to something other than consul.bak. When dumping out an ACL or prepared query backup you will need to pass both -file and -acls or -queries. You can also only dump one file at a time.

[michaelgrosstarget]

Yep, that's exactly it - syntax error :-) Carefully reading the documentation I get it now, I had previously skimmed it assuming it was the same as backup/restore (also explains why the -nokv option isn't relevant in the dump context). The below worked for me. Thanks again for all your help!

# consul-backinator dump -file kvs.bak -key [SECRET]
# consul-backinator dump -file acls.bak -acls -key [SECRET]
# consul-backinator dump -file queries.bak -queries -key [SECRET]

[aaronhurt]

Not a perfect fix but at least it's presenting an error if it doesn't recognize the syntax now.

Errors on unhandled arguments

$ consul-backinator dump -acls foo
2017/11/03 09:50:08 [Error] Setup failed: Unknown non-flag argument(s) present after command

Still works when passed:

$ consul-backinator dump -acls -file foo
[
  {
    "CreateIndex": 11,
    "ModifyIndex": 11,
    "ID": "345665ec-3bb4-ffe1-a2f7-a776bfce6d1b",
    "Name": "dummy-acl02",
    "Type": "client",
    "Rules": ""
  },
  {
    "CreateIndex": 10,
    "ModifyIndex": 10,
    "ID": "7f60b00f-8727-6cb3-e3a9-ff74a2d9a9b2",
    "Name": "dummy-acl01",
    "Type": "client",
    "Rules": ""
  },
  {
    "CreateIndex": 4,
    "ModifyIndex": 4,
    "ID": "anonymous",
    "Name": "Anonymous Token",
    "Type": "client",
    "Rules": ""
  },
  {
    "CreateIndex": 5,
    "ModifyIndex": 5,
    "ID": "eb431e74-e29b-9633-e285-9cdccda985e3",
    "Name": "Master Token",
    "Type": "management",
    "Rules": ""
  }
]