consul snapshot restore - not restoring KV pairs

[camAtGitHub]

When performing consul snapshot restore backup.xx KV pairs are NOT restored.

consul version for both Client and Server

Client: Consul v0.8.1 Server: Consul v0.8.1

consul info for both Client and Server

Client: agent:
    check_monitors = 0
    check_ttls = 0
    checks = 0
    services = 0
build:
    prerelease = 
    revision = 'e9ca44d
    version = 0.8.1
consul:
    known_servers = 1
    server = false
runtime:
    arch = amd64
    cpu_count = 2
    goroutines = 37
    max_procs = 2
    os = linux
    version = go1.8.1
serf_lan:
    encrypted = false
    event_queue = 0
    event_time = 4
    failed = 0
    health_score = 0
    intent_queue = 0
    left = 0
    member_time = 8
    members = 2
    query_queue = 0
    query_time = 1

Server:

Client: agent:
    check_monitors = 0
    check_ttls = 0
    checks = 0
    services = 0
build:
    prerelease = 
    revision = 'e9ca44d
    version = 0.8.1
consul:
    known_servers = 1
    server = false
runtime:
    arch = amd64
    cpu_count = 2
    goroutines = 37
    max_procs = 2
    os = linux
    version = go1.8.1
serf_lan:
    encrypted = false
    event_queue = 0
    event_time = 4
    failed = 0
    health_score = 0
    intent_queue = 0
    left = 0
    member_time = 8
    members = 2
    query_queue = 0
    query_time = 1

Operating system and Environment details

Redhat 7 x64

Description of the Issue (and unexpected/desired result)

When performing consul snapshot restore backup.xx KV pairs are NOT restored.

Reproduction steps

./vault write secret/test value=DATA-BEFORE-SNAPSHOT
./consul snapshot save backup.zz
./vault write secret/test value=DATA-AFTER-SNAPSHOT
./consul snapshot restore backup.zz
./vault read secret/test
Key                 Value
---                 -----
refresh_interval    768h0m0s
value               DATA-AFTER-SNAPSHOT

Log Fragments

[cm@vault1 ~]$ ./vault write secret/test value=DATA-BEFORE-SNAPSHOT
    2017/05/04 01:04:17 [INFO] agent: Synced node info
Success! Data written to: secret/test

[cm@vault1 ~]$ ./vault read secret/test
Key                 Value
---                 -----
refresh_interval    768h0m0s
value               DATA-BEFORE-SNAPSHOT

[cm@vault1 ~]$ ./consul snapshot save backup.zz
    2017/05/04 01:04:42 [INFO] consul.fsm: snapshot created in 35.642µs
    2017/05/04 01:04:42 [INFO] raft: Starting snapshot up to 531
    2017/05/04 01:04:42 [INFO] snapshot: Creating new snapshot at /tmp/consul/raft/snapshots/4-531-1493859882830.tmp
    2017/05/04 01:04:42 [INFO] snapshot: reaping snapshot /tmp/consul/raft/snapshots/4-430-1493858918976
    2017/05/04 01:04:42 [INFO] raft: Snapshot to 531 complete
Saved and verified snapshot to index 531

[cm@vault1 ~]$ ./vault write secret/test value=DATA-AFTER-SNAPSHOT
Success! Data written to: secret/test

[cm@vault1 ~]$ ./vault read secret/test
Key                 Value
---                 -----
refresh_interval    768h0m0s
value               DATA-AFTER-SNAPSHOT

[cm@vault2 ~]$  ./consul snapshot restore backup.zz
    2017/05/04 01:11:00 [DEBUG] http: Request PUT /v1/snapshot (1.6681683s) from=127.0.0.1:43556
Restored snapshot
# Note: restoring from vault1 results in same behaviour

[cm@vault1 ~]$ ./vault read secret/test
Key                 Value
---                 -----
refresh_interval    768h0m0s
value               DATA-AFTER-SNAPSHOT

[slackpad]

Hi @camAtGitHub I think you are getting a cached result from Vault. If you stop Vault, do the restore, and start Vault again you should see the pre-snapshot value.

[camAtGitHub]

Ahhhhh, you are correct. I did look for caching parameters but in the wrong place obviously. Thanks for the help and sorry for the time waste.

[slackpad]

NP - glad you figured it out! I DM-ed one of the Vault devs and said "Vault caches, right" and they said "yep" so not a big deal :-)