problems restoring backup containing None

[jantman]

I'm running consulate from master (as of 3c6ce3c4f7a675d68046eb22ed244306a033111a) with current Consul (0.5.2). I originally tried using the version of consulate on PyPi and ran into the same issue as #61. Installing from master, that issue (when backing up) is fixed, but I get the corresponding issue on the "other side" when trying to restore the backup I just made:

(consulate)jantman@phoenix:pts/14:~/tmp/consulate$ consulate --api-host myhost.mydomain.com --api-port 8500 kv backup --base64 -f consul_backup.json
(consulate)jantman@phoenix:pts/14:~/tmp/consulate$ echo $?
0
(consulate)jantman@phoenix:pts/14:~/tmp/consulate$ ls -l consul_backup.json 
-rw-r--r-- 1 jantman jantman 851677 Oct 14 14:35 consul_backup.json
(consulate)jantman@phoenix:pts/14:~/tmp/consulate$ consulate --api-host 127.0.0.1 --api-port 8500 kv restore -b -n -f consul_backup.json
Traceback (most recent call last):
  File "/tmp/jantman/consulate/bin/consulate", line 9, in <module>
    load_entry_point('consulate==0.6.0', 'console_scripts', 'consulate')()
  File "/tmp/jantman/consulate/src/consulate/consulate/cli.py", line 469, in main
    KV_ACTIONS[args.action](consul, args)
  File "/tmp/jantman/consulate/src/consulate/consulate/cli.py", line 310, in kv_restore
    row[2] = base64.b64decode(row[2])
  File "/usr/lib64/python2.7/base64.py", line 73, in b64decode
    return binascii.a2b_base64(s)
TypeError: must be string or buffer, not None

If I apply the following patch:

diff --git a/consulate/cli.py b/consulate/cli.py
index 09a838f..d08153b 100644
--- a/consulate/cli.py
+++ b/consulate/cli.py
@@ -307,7 +307,11 @@ def kv_restore(consul, args):
             row = [row['Key'], row['Flags'], row['Value']]

         if args.base64:
-            row[2] = base64.b64decode(row[2])
+            try:
+                row[2] = base64.b64decode(row[2])
+            except Exception as ex:
+                print(row)
+                raise ex

         # Here's an awesome thing to make things work
         if not utils.PYTHON3 and isinstance(row[2], unicode):

Then it becomes a bit more clear:

(consulate)jantman@phoenix:pts/14:~/tmp/consulate$ consulate --api-host 127.0.0.1 --api-port 8500 kv restore -b -n -f consul_backup.json
[u'configs/test/', 0, None]
Traceback (most recent call last):
  File "/tmp/jantman/consulate/bin/consulate", line 9, in <module>
    load_entry_point('consulate==0.6.0', 'console_scripts', 'consulate')()
  File "/tmp/jantman/consulate/src/consulate/consulate/cli.py", line 473, in main
    KV_ACTIONS[args.action](consul, args)
  File "/tmp/jantman/consulate/src/consulate/consulate/cli.py", line 314, in kv_restore
    raise ex
TypeError: must be string or buffer, not None

[jantman]

I was able to get a successful restore by simply ignoring any None/null values, though I'm not sure what the actual impact of that is...

diff --git a/consulate/cli.py b/consulate/cli.py
index 09a838f..3238ff2 100644
--- a/consulate/cli.py
+++ b/consulate/cli.py
@@ -307,6 +307,10 @@ def kv_restore(consul, args):
             row = [row['Key'], row['Flags'], row['Value']]

         if args.base64:
+            if row[2] is None:
+                print("skipping None/null value: item=%s flags=%s", row[0],
+                      row[1])
+                continue
             row[2] = base64.b64decode(row[2])

         # Here's an awesome thing to make things work

[jantman]

I can confirm that the issue I described in my last comment is fixed in https://github.com/bdclark/consulate/tree/develop as of 886864c16d6c6c0b1822feb9668c0f0510939444

[jantman]

@gmr Is there any hope of getting @bdclark's fix merged?

[gmr]

This has been merged and will be released along with 0.7

[devlinmr]

+1 on a release please.