search

hashicorp / vault

A tool for secrets management, encryption as a service, and privileged access management
https://www.vaultproject.io/
Other
30.86k stars 4.17k forks source link

vault key-status failing with 500 error #319

Closed kenbreeman closed 9 years ago

kenbreeman commented 9 years ago

Ran 'vault key-status' a couple of times using the 'file' backend and got a 500 error. I had run various cli commands prior to this including:

vault unseal
vault auth
vault status
vault audit-enable file path=/var/log/vault_audit.log

Command:

> vault key-status
Error reading audits: Get https://REDACTED_FQDN:443/v1/sys/key-status: EOF

Version:

> vault version
Vault v0.1.3-dev (798892a47020e72e9de53423aa1d6e260a07e259)

Config:

> cat /etc/vault.cfg
backend "file" {
  path = "/var/vault"
}

listener "tcp" {
  address = "0.0.0.0:443"
  tls_cert_file = "/etc/..."
  tls_key_file = "/etc/..."
}

Log:

2015/06/04 20:38:44 http: panic serving 172.16.2.25:33461: reflect.Value.Interface: cannot return value obtained from unexported field or method
goroutine 125 [running]:
net/http.func·011()
        /usr/lib/golang/src/net/http/server.go:1130 +0xbb
reflect.valueInterface(0x9274c0, 0xec25c0, 0xf8, 0x40b801, 0x0, 0x0)
        /usr/lib/golang/src/reflect/value.go:883 +0xf4
reflect.Value.Interface(0x9274c0, 0xec25c0, 0xf8, 0x0, 0x0)
        /usr/lib/golang/src/reflect/value.go:872 +0x53
github.com/hashicorp/vault/audit.(*hashWalker).Primitive(0xc2086c7360, 0x9274c0, 0xec25c0, 0xf8, 0x0, 0x0)
        /home/kbreeman/go/src/github.com/hashicorp/vault/audit/hashstructure.go:191 +0x1be
github.com/mitchellh/reflectwalk.walkPrimitive(0x9274c0, 0xec25c0, 0xf8, 0xad4820, 0xc2086c7360, 0x0, 0x0)
        /home/kbreeman/go/src/github.com/mitchellh/reflectwalk/reflectwalk.go:186 +0x96
github.com/mitchellh/reflectwalk.walk(0x9274c0, 0xec25c0, 0xf8, 0xad4820, 0xc2086c7360, 0x0, 0x0)
        /home/kbreeman/go/src/github.com/mitchellh/reflectwalk/reflectwalk.go:117 +0x36b
github.com/mitchellh/reflectwalk.walkStruct(0xab9b20, 0xec25c0, 0xf9, 0xad4820, 0xc2086c7360, 0x0, 0x0)
        /home/kbreeman/go/src/github.com/mitchellh/reflectwalk/reflectwalk.go:264 +0x44a
github.com/mitchellh/reflectwalk.walk(0xab9b20, 0xec25c0, 0xf9, 0xad4820, 0xc2086c7360, 0x0, 0x0)
        /home/kbreeman/go/src/github.com/mitchellh/reflectwalk/reflectwalk.go:126 +0x516
github.com/mitchellh/reflectwalk.walkStruct(0xb0f040, 0xc20844d560, 0x59, 0xad4820, 0xc2086c7360, 0x0, 0x0)
        /home/kbreeman/go/src/github.com/mitchellh/reflectwalk/reflectwalk.go:264 +0x44a
github.com/mitchellh/reflectwalk.walk(0xb0f040, 0xc20844d560, 0x59, 0xad4820, 0xc2086c7360, 0x0, 0x0)
        /home/kbreeman/go/src/github.com/mitchellh/reflectwalk/reflectwalk.go:126 +0x516
github.com/mitchellh/reflectwalk.walkMap(0x924b40, 0xc2086e52f0, 0x15, 0xad4820, 0xc2086c7360, 0x0, 0x0)
        /home/kbreeman/go/src/github.com/mitchellh/reflectwalk/reflectwalk.go:168 +0x508
github.com/mitchellh/reflectwalk.walk(0x924b40, 0xc2086e52f0, 0x15, 0xad4820, 0xc2086c7360, 0x0, 0x0)
        /home/kbreeman/go/src/github.com/mitchellh/reflectwalk/reflectwalk.go:120 +0x472
github.com/mitchellh/reflectwalk.Walk(0x924b40, 0xc2086e52f0, 0xad4820, 0xc2086c7360, 0x0, 0x0)
        /home/kbreeman/go/src/github.com/mitchellh/reflectwalk/reflectwalk.go:70 +0x137
github.com/hashicorp/vault/audit.HashStructure(0x924b40, 0xc2086e52f0, 0xc2086d90c0, 0x0, 0x0, 0x0, 0x0)
        /home/kbreeman/go/src/github.com/hashicorp/vault/audit/hashstructure.go:84 +0x13b
github.com/hashicorp/vault/audit.Hash(0xa00c60, 0xc2086e5110, 0x0, 0x0)
        /home/kbreeman/go/src/github.com/hashicorp/vault/audit/hashstructure.go:62 +0x2ea
github.com/hashicorp/vault/builtin/audit/file.(*Backend).LogResponse(0xc2086e40f0, 0xc208391900, 0xc208391980, 0xc2086e5110, 0x0, 0x0, 0x0, 0x0)
        /home/kbreeman/go/src/github.com/hashicorp/vault/builtin/audit/file/backend.go:117 +0x363
github.com/hashicorp/vault/vault.(*AuditBroker).LogResponse(0xc208333e30, 0xc208391700, 0xc208391680, 0xc2086e4f30, 0x0, 0x0, 0x0, 0x0)
        /home/kbreeman/go/src/github.com/hashicorp/vault/vault/audit.go:299 +0x299
github.com/hashicorp/vault/vault.(*Core).handleRequest(0xc208098000, 0xc208391680, 0x0, 0x0, 0x0)
        /home/kbreeman/go/src/github.com/hashicorp/vault/vault/core.go:444 +0xea2
github.com/hashicorp/vault/vault.(*Core).HandleRequest(0xc208098000, 0xc208391680, 0x0, 0x0, 0x0)
        /home/kbreeman/go/src/github.com/hashicorp/vault/vault/core.go:345 +0x1f3
github.com/hashicorp/vault/http.func·020(0x7f6e963498e8, 0xc2086c7040, 0xc208386270)
        /home/kbreeman/go/src/github.com/hashicorp/vault/http/sys_rotate.go:20 +0xff
net/http.HandlerFunc.ServeHTTP(0xc2080a0160, 0x7f6e963498e8, 0xc2086c7040, 0xc208386270)
        /usr/lib/golang/src/net/http/server.go:1265 +0x41
net/http.(*ServeMux).ServeHTTP(0xc20826c570, 0x7f6e963498e8, 0xc2086c7040, 0xc208386270)
        /usr/lib/golang/src/net/http/server.go:1541 +0x17d
github.com/hashicorp/vault/http.func·001(0x7f6e963498e8, 0xc2086c7040, 0xc208386270)
        /home/kbreeman/go/src/github.com/hashicorp/vault/http/help.go:18 +0x128
net/http.HandlerFunc.ServeHTTP(0xc20826bc00, 0x7f6e963498e8, 0xc2086c7040, 0xc208386270)
        /usr/lib/golang/src/net/http/server.go:1265 +0x41
net/http.serverHandler.ServeHTTP(0xc208054540, 0x7f6e963498e8, 0xc2086c7040, 0xc208386270)
        /usr/lib/golang/src/net/http/server.go:1703 +0x19a
net/http.(*conn).serve(0xc208354500)
        /usr/lib/golang/src/net/http/server.go:1204 +0xb57
created by net/http.(*Server).Serve
        /usr/lib/golang/src/net/http/server.go:1751 +0x35e

Audit log:

{"type":"request","auth":{"display_name":"root","policies":["root"],"metadata":null},"request":{"operation":"read","path":"sys/key-status","data":null}}

Filesystem:

> find /var/vault/
/var/vault/
/var/vault/sys
/var/vault/sys/expire
/var/vault/sys/expire/id
/var/vault/sys/expire/id/secret
/var/vault/sys/expire/id/secret/hello
/var/vault/sys/expire/id/secret/hello/_4170f899-41a5-755d-30fa-db05f06fb4e9
/var/vault/sys/expire/id/auth
/var/vault/sys/expire/id/auth/token
/var/vault/sys/expire/id/auth/token/create
/var/vault/sys/expire/id/auth/token/create/_a28a09dab7ce922e9129a6794dbaff6bd66c92f2
/var/vault/sys/expire/token
/var/vault/sys/expire/token/44b186587b6fc70083edde902b6f6c73386e3e9d
/var/vault/sys/expire/token/44b186587b6fc70083edde902b6f6c73386e3e9d/_e0b91688b1c13f98501c8fac8357999a6f2e3c80
/var/vault/sys/token
/var/vault/sys/token/id
/var/vault/sys/token/id/_44b186587b6fc70083edde902b6f6c73386e3e9d
/var/vault/sys/token/id/_a28a09dab7ce922e9129a6794dbaff6bd66c92f2
/var/vault/sys/token/_salt
/var/vault/sys/token/parent
/var/vault/sys/token/parent/44b186587b6fc70083edde902b6f6c73386e3e9d
/var/vault/sys/token/parent/44b186587b6fc70083edde902b6f6c73386e3e9d/_a28a09dab7ce922e9129a6794dbaff6bd66c92f2
/var/vault/logical
/var/vault/logical/c6b299d1-6dcf-9d6d-2592-3c0a5a7b9de7
/var/vault/logical/c6b299d1-6dcf-9d6d-2592-3c0a5a7b9de7/_hello
/var/vault/core
/var/vault/core/_master
/var/vault/core/_keyring
/var/vault/core/_seal-config
/var/vault/core/_auth
/var/vault/core/_mounts
/var/vault/core/_audit
/var/vault/auth
/var/vault/auth/65804ced-89a1-f069-7611-663540783ed1
kenbreeman commented 9 years ago

Everything else appears to be working, I can read and write secrets

kenbreeman commented 9 years ago

Vault is running as the root user (this is a test host) with the following command line:

vault server -config /etc/vault.cfg -log-level=debug

Audit log permissions:

> ls -la /var/log/vault_audit.log 
-rw------- 1 root root 10927 Jun  4 20:58 /var/log/vault_audit.log
mitchellh commented 9 years ago

@armon So the Data field of a response is map[string]interface{} to allow for rich structures, but not for Go types (since it has to turn into an HTTP field anyways). The audit hashing doesn't expect this, since it expects to hash only primitives, in the case of a response (things that can go over JSON).

I think we need to change the sys/key-status endpoint in vault/logical_system.go to properly return the non-rich Time type, probably just a UTC timestamp.

armon commented 9 years ago

Fixed by #373

amenaafreen commented 4 years ago

I have vault deployed on kubernetes cluster with vault dev server enabled, helm install --name=vault \

--set 'ui.enabled=true' --set 'server.affinity=' \ --set 'authDelegator.enabled=true' vault-helm But then, when I deploy the application to read the secrets I am getting an Internal server error

Screen Shot 2020-02-06 at 2 48 28 PM