search

canonical / vault-k8s-operator

Vault secure, store and tightly control access to tokens, passwords, certificates, encryption keys for protecting secrets and other sensitive data.
https://charmhub.io/vault-k8s
Apache License 2.0
8 stars 4 forks source link

VaultDown error when removing sealed unit #307

Closed gruyaume closed 2 months ago

gruyaume commented 2 months ago

Describe the bug

We are getting a VaultDown error when trying to remove a sealed unit

To Reproduce

  1. Deploy Vault with 1 unit
  2. Initialize, unseal, and authorize vault
  3. Add 1 unit
  4. Wait for the unit to go to blocked status ("Please unseal vault")
  5. Remove the unit

Expected behavior

No error

Logs

unit-vault-1: 13:20:49 INFO unit.vault/1.juju-log Removed Vault's main database
unit-vault-1: 13:20:49 INFO unit.vault/1.juju-log Removed Vault's Raft database
unit-vault-1: 13:20:49 ERROR unit.vault/1.juju-log Uncaught exception while in charm code:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-vault-1/charm/./src/charm.py", line 1345, in <module>
    main(VaultCharm)
  File "/var/lib/juju/agents/unit-vault-1/charm/venv/ops/main.py", line 544, in main
    manager.run()
  File "/var/lib/juju/agents/unit-vault-1/charm/venv/ops/main.py", line 520, in run
    self._emit()
  File "/var/lib/juju/agents/unit-vault-1/charm/venv/ops/main.py", line 509, in _emit
    _emit_charm_event(self.charm, self.dispatcher.event_name)
  File "/var/lib/juju/agents/unit-vault-1/charm/venv/ops/main.py", line 143, in _emit_charm_event
    event_to_emit.emit(*args, **kwargs)
  File "/var/lib/juju/agents/unit-vault-1/charm/venv/ops/framework.py", line 352, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-vault-1/charm/venv/ops/framework.py", line 851, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-vault-1/charm/venv/ops/framework.py", line 941, in _reemit
    custom_handler(event)
  File "/var/lib/juju/agents/unit-vault-1/charm/./src/charm.py", line 298, in _on_remove
    and vault.is_node_in_raft_peers(node_id=self._node_id)
  File "/var/lib/juju/agents/unit-vault-1/charm/lib/charms/vault_k8s/v0/vault_client.py", line 397, in is_node_in_raft_peers
    raft_config = self._client.sys.read_raft_config()
  File "/var/lib/juju/agents/unit-vault-1/charm/venv/hvac/api/system_backend/raft.py", line 76, in read_raft_config
    return self._adapter.get(
  File "/var/lib/juju/agents/unit-vault-1/charm/venv/hvac/adapters.py", line 146, in get
    return self.request("get", url, **kwargs)
  File "/var/lib/juju/agents/unit-vault-1/charm/venv/hvac/adapters.py", line 408, in request
    response = super().request(*args, **kwargs)
  File "/var/lib/juju/agents/unit-vault-1/charm/venv/hvac/adapters.py", line 376, in request
    self._raise_for_error(method, url, response)
  File "/var/lib/juju/agents/unit-vault-1/charm/venv/hvac/adapters.py", line 294, in _raise_for_error
    utils.raise_for_error(
  File "/var/lib/juju/agents/unit-vault-1/charm/venv/hvac/utils.py", line 41, in raise_for_error
    raise exceptions.VaultError.from_status(
hvac.exceptions.VaultDown: Vault is sealed, on get https://vault-1.vault-endpoints.demo.svc.cluster.local:8200/v1/sys/storage/raft/configuration
unit-vault-1: 13:20:49 ERROR juju.worker.uniter.operation hook "remove" (via hook dispatching script: dispatch) failed: exit status 1
unit-vault-1: 13:20:49 INFO juju.worker.uniter awaiting error resolution for "remove" hook

Environment

gruyaume commented 2 months ago

We should check if Vault is sealed before authenticating and trying anything. If it is sealed, we should return.