Cannot download raft storage snapshot from UI

[denmaddog]

Describe the bug When I login to Vault UI and go to Status/Raft storage --> Snapshots --> Download It hangs for some time and then it shows message in browser: {"errors":["permission denied"]}

URL is: https://vault-cluster.example.com:8200/ui/vault/storage/raft

To Reproduce Steps to reproduce the behavior:

1. Login to Vaults UI and go to Status/Raft storage --> Snapshots --> Download

Expected behavior It should offer snapshot to download.

Environment:

• Vault Server Version (retrieve with vault status): vault status Key Value

  ---

  Seal Type shamir Initialized true Sealed false Total Shares 5 Threshold 3 Version 1.12.0 Build Date 2022-10-10T18:14:33Z Storage Type raft Cluster Name vault-cluster-xxxxxx Cluster ID xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx HA Enabled true HA Cluster https://vault-node1.example.com:8201 HA Mode active Active Since 2022-10-19T16:55:47.797228758Z Raft Committed Index 2003050 Raft Applied Index 2003050

• Vault CLI Version (retrieve with vault version): Vault v1.12.0 (558abfa75702b5dab4c98e86b802fb9aef43b0eb), built 2022-10-10T18:14:33Z
• Server Operating System/Architecture: PRETTY_NAME="Debian GNU/Linux 10 (buster)" NAME="Debian GNU/Linux" VERSION_ID="10" VERSION="10 (buster)" VERSION_CODENAME=buster ID=debian HOME_URL="https://www.debian.org/" SUPPORT_URL="https://www.debian.org/support" BUG_REPORT_URL="https://bugs.debian.org/" Vault server configuration file(s):

root@vault-node1:~# cat /etc/vault.d/vault.hcl
storage "raft" {
  path    = "/raft-data/"
  node_id = "node1"

#  retry_join {
#    leader_api_addr = "https://vault-node1.example.com:8200"
#  }

#  retry_join {
#    leader_api_addr = "https://vault-node2.example.com:8200"
#  }

#  retry_join {
#    leader_api_addr = "https://vault-node3.example.com:8200"
#  }

#  retry_join {
#    leader_api_addr = "https://vault-node4.example.com:8200"
#  }
}

listener "tcp" {
  address = "0.0.0.0:8200"
  cluster_address = "0.0.0.0:8201"
#  tls_disable = true
  tls_cert_file    = "/etc/vault.d/example.com.crt"
  tls_key_file     = "/etc/vault.d/example.com.key"
}

disable_mlock = true
api_addr = "https://vault-node1.example.com:8200"
cluster_addr = "https://vault-node1.example.com:8201"
ui = true
####################

root@vault-node2:~# cat /etc/vault.d/vault.hcl
storage "raft" {
  path    = "/raft-data/"
  node_id = "node2"

  retry_join {
    leader_api_addr = "https://vault-node1.example.com:8200"
  }

  retry_join {
    leader_api_addr = "https://vault-node3.example.com:8200"
  }

  retry_join {
    leader_api_addr = "https://vault-node4.example.com:8200"
  }

  retry_join {
    leader_api_addr = "https://vault-node5.example.com:8200"
  }
}

listener "tcp" {
  address = "0.0.0.0:8200"
  cluster_address = "0.0.0.0:8201"
#  tls_disable = true
  tls_cert_file    = "/etc/vault.d/example.com.crt"
  tls_key_file     = "/etc/vault.d/example.com.key"
}

disable_mlock = true
api_addr = "https://vault-node2.example.com:8200"
cluster_addr = "https://vault-node2.example.com:8201"
ui = true

vault operator raft autopilot state Healthy: true Failure Tolerance: 2 Leader: node1 Voters: node1 node2 node3 node4 node5 Servers: node1 Name: node1 Address: vault-node1.example.com:8201 Status: leader Node Status: alive Healthy: true Last Contact: 0s Last Term: 48 Last Index: 2003100 Version: 1.12.0 Node Type: voter node2 Name: node2 Address: vault-node2.example.com:8201 Status: voter Node Status: alive Healthy: true Last Contact: 139.461806ms Last Term: 48 Last Index: 2003100 Version: 1.12.0 Node Type: voter node3 Name: node3 Address: vault-node3.example.com:8201 Status: voter Node Status: alive Healthy: true Last Contact: 434.560895ms Last Term: 48 Last Index: 2003100 Version: 1.12.0 Node Type: voter node4 Name: node4 Address: vault-node4.example.com:8201 Status: voter Node Status: alive Healthy: true Last Contact: 1.706376111s Last Term: 48 Last Index: 2003100 Version: 1.12.0 Node Type: voter node5 Name: node5 Address: vault-node5.example.com:8201 Status: voter Node Status: alive Healthy: true Last Contact: 156.711955ms Last Term: 48 Last Index: 2003100 Version: 1.12.0 Node Type: voter

Additional context Manual creation of snapshot works: vault operator raft snapshot save 19102022.snapshot

ls -lah total 9.2M drwxr-xr-x 2 root root 4.0K Oct 19 17:01 . drwx------ 8 root root 4.0K Oct 19 17:01 .. -rw------- 1 root root 9.2M Oct 19 17:01 19102022.snapshot

[hsimon-hashicorp]

Just to clarify, the manual creation of the snapshot with the vault operator command is with the same credentials, correct?

[denmaddog]

Yes, using root token for authentication with both vault cli and UI.

[hsimon-hashicorp]

Thank you! I'll have our engineering teams take a look. :)

[denmaddog]

np, thank you! :)

[atabacar]

I am having the same issue. I had Vault 1.8.8 and the download snapshot from UI was working fine. Then I upgraded to 1.11.3 and that functionality broke. In version 1.11.3 I tried both being logged as my usr and as root user. Same behaviour. I can take the snapshot using the cli but from UI it hangs and eventually I get permission denied message in the browser.

[denmaddog]

Same with v1.12.1.

[hashishaw]

This was fixed in PR #17769 -- will be released with 1.11.6 and 1.12.2. Thanks all!