hashicorp / vault

A tool for secrets management, encryption as a service, and privileged access management
https://www.vaultproject.io/
Other
31.22k stars 4.22k forks source link

NPE/panic when seal config is missing #28547

Open taitelman opened 1 month ago

taitelman commented 1 month ago

scenario : vault cluster with HA enabled , leader lost leadership for some reason the standby see this:

https://github.com/hashicorp/vault/blob/2d46c8d1fdd3f71253b707fb0219dbbc16da218b/vault/core.go#L3202

c.PhysicalBarrierSealConfig(ctx) at top of https://github.com/hashicorp/vault/blob/2d46c8d1fdd3f71253b707fb0219dbbc16da218b/vault/core.go#L3196 can return nil,nil under some edge cases. and then the code will fail in the next pointer usage since barrierSealConfig == nil

and the result is :

vault debug not reloading seals config since there is no seal generation info in storage

vault error panic: runtime error: invalid memory address or nil pointer dereference
vault [signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x33db0ce]
vault goroutine 232 [running]:
vault github.com/hashicorp/vault/vault.(*Core).migrateMultiSealConfig(0xc0001b1000, {0xc656e30, 0xc0038ecdc0})
    /opt/app-root/src/vault/core.go:3203 +0xae
vault github.com/hashicorp/vault/vault.(*Core).migrateSeal(0xc656e30?, {0xc656e30?, 0xc0038ecdc0?})
    /opt/app-root/src/vault/core.go:1906 +0xc17
vault github.com/hashicorp/vault/vault.(*Core).waitForLeadership(0xc0001b1000, 0x0?, 0xc003a9f740, 0xc003a9f920)
    /opt/app-root/src/vault/ha.go:604 +0x77f
vault github.com/hashicorp/vault/vault.(*Core).runStandby.func9()
    /opt/app-root/src/vault/ha.go:475 +0x25

solution: switch case should handle nil values for barrierSealConfig more gracefully.

taitelman commented 1 month ago

debug line from : https://github.com/hashicorp/vault/blob/2d46c8d1fdd3f71253b707fb0219dbbc16da218b/command/server.go#L3163

aphorise commented 2 weeks ago

@taitelman out of interest - how did you achieve the seal change? - was it through a config change to an existing seal stanza & then a reload?