[vault] Refreshing vault from 1.7/stable to 1.8/stable cause vault to be in error state, and sealed

chanchiwai-ray commented 1 week ago

When COU refresh vault from 1.7/stable to 1.8/stable during Jammy/Yoga -> Jammy/Zed upgrade, the vault charm goes into error state with Connection Refused error:

unit-vault-0: 03:48:42 WARNING unit.vault/0.upgrade-charm Traceback (most recent call last):                                                 
unit-vault-0: 03:48:42 WARNING unit.vault/0.upgrade-charm   File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.10/site-packages/requests/adapters.py", line 439, in send                                                                                                           
unit-vault-0: 03:48:42 WARNING unit.vault/0.upgrade-charm     resp = conn.urlopen(                                                           
unit-vault-0: 03:48:42 WARNING unit.vault/0.upgrade-charm   File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 787, in urlopen                                                                                                   
unit-vault-0: 03:48:42 WARNING unit.vault/0.upgrade-charm     retries = retries.increment(                                                   
unit-vault-0: 03:48:42 WARNING unit.vault/0.upgrade-charm   File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.10/site-packages/urllib3/util/retry.py", line 592, in increment                                                                                                     
unit-vault-0: 03:48:42 WARNING unit.vault/0.upgrade-charm     raise MaxRetryError(_pool, url, error or ResponseError(cause))                 
unit-vault-0: 03:48:42 WARNING unit.vault/0.upgrade-charm urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='127.0.0.1', port=8220): Max retries exceeded with url: /v1/auth/approle/login (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f9bba586bf0>: Failed to establish a new connection: [Errno 111] Connection refused'))                                                                
unit-vault-0: 03:48:42 WARNING unit.vault/0.upgrade-charm                                                                                    
unit-vault-0: 03:48:42 WARNING unit.vault/0.upgrade-charm During handling of the above exception, another exception occurred:                
unit-vault-0: 03:48:42 WARNING unit.vault/0.upgrade-charm                                                                                    
unit-vault-0: 03:48:42 WARNING unit.vault/0.upgrade-charm Traceback (most recent call last):                                                 
unit-vault-0: 03:48:42 WARNING unit.vault/0.upgrade-charm   File "/var/lib/juju/agents/unit-vault-0/charm/hooks/upgrade-charm", line 22, in <module>                                                                                                                                      
unit-vault-0: 03:48:42 WARNING unit.vault/0.upgrade-charm     main()
unit-vault-0: 03:48:42 WARNING unit.vault/0.upgrade-charm   File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.10/site-packages/charms/reactive/__init__.py", line 74, in main                                                                                                     
unit-vault-0: 03:48:42 WARNING unit.vault/0.upgrade-charm     bus.dispatch(restricted=restricted_mode)                                       
unit-vault-0: 03:48:42 WARNING unit.vault/0.upgrade-charm   File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.10/site-packages/charms/reactive/bus.py", line 390, in dispatch                                                                                                     
unit-vault-0: 03:48:42 WARNING unit.vault/0.upgrade-charm     _invoke(other_handlers)                                                        
unit-vault-0: 03:48:42 WARNING unit.vault/0.upgrade-charm   File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.10/site-packages/charms/reactive/bus.py", line 359, in _invoke                                                                                                      
unit-vault-0: 03:48:42 WARNING unit.vault/0.upgrade-charm     handler.invoke()                                                               
unit-vault-0: 03:48:42 WARNING unit.vault/0.upgrade-charm   File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.10/site-packages/charms/reactive/bus.py", line 181, in invoke                                                                                                       
unit-vault-0: 03:48:42 WARNING unit.vault/0.upgrade-charm     self._action(*args)                                                            
unit-vault-0: 03:48:42 WARNING unit.vault/0.upgrade-charm   File "/var/lib/juju/agents/unit-vault-0/charm/reactive/vault_handlers.py", line 1044, in publish_ca_info                                                                                                                      
unit-vault-0: 03:48:42 WARNING unit.vault/0.upgrade-charm     if not client_approle_authorized():                                            
unit-vault-0: 03:48:42 WARNING unit.vault/0.upgrade-charm   File "/var/lib/juju/agents/unit-vault-0/charm/reactive/vault_handlers.py", line 987, in client_approle_authorized                                                                                                             
unit-vault-0: 03:48:42 WARNING unit.vault/0.upgrade-charm     vault.get_local_client()                                                       
unit-vault-0: 03:48:42 WARNING unit.vault/0.upgrade-charm   File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.10/site-packages/tenacity/__init__.py", line 339, in wrapped_f                                                                                                      
unit-vault-0: 03:48:42 WARNING unit.vault/0.upgrade-charm     return self(f, *args, **kw)                                                    
unit-vault-0: 03:48:42 WARNING unit.vault/0.upgrade-charm   File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.10/site-packages/tenacity/__init__.py", line 430, in __call__                                                                                                       
unit-vault-0: 03:48:42 WARNING unit.vault/0.upgrade-charm     do = self.iter(retry_state=retry_state)                                        
unit-vault-0: 03:48:42 WARNING unit.vault/0.upgrade-charm   File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.10/site-packages/tenacity/__init__.py", line 367, in iter                                                                                                           
unit-vault-0: 03:48:42 WARNING unit.vault/0.upgrade-charm     return fut.result()                                                            
unit-vault-0: 03:48:42 WARNING unit.vault/0.upgrade-charm   File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result                                                                                                                                                   
unit-vault-0: 03:48:42 WARNING unit.vault/0.upgrade-charm     return self.__get_result()                                                     
unit-vault-0: 03:48:42 WARNING unit.vault/0.upgrade-charm   File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result                                                                                                                                             
unit-vault-0: 03:48:42 WARNING unit.vault/0.upgrade-charm     raise self._exception                                                          
unit-vault-0: 03:48:42 WARNING unit.vault/0.upgrade-charm   File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.10/site-packages/tenacity/__init__.py", line 433, in __call__                                                                                                       
unit-vault-0: 03:48:42 WARNING unit.vault/0.upgrade-charm     result = fn(*args, **kwargs)                                                   
unit-vault-0: 03:48:42 WARNING unit.vault/0.upgrade-charm   File "/var/lib/juju/agents/unit-vault-0/charm/lib/charm/vault.py", line 284, in get_local_client                                                                                                                              
unit-vault-0: 03:48:42 WARNING unit.vault/0.upgrade-charm     client.auth.approle.login(app_role_id)                                                                                                                                                                                      
unit-vault-0: 03:48:42 WARNING unit.vault/0.upgrade-charm   File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.10/site-packages/hvac/api/auth_methods/approle.py", line 494, in login                                                                                              
unit-vault-0: 03:48:42 WARNING unit.vault/0.upgrade-charm     return self._adapter.login(                                                                                                                                                                                                 
unit-vault-0: 03:48:42 WARNING unit.vault/0.upgrade-charm   File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.10/site-packages/hvac/adapters.py", line 197, in login                                                                                                              
unit-vault-0: 03:48:42 WARNING unit.vault/0.upgrade-charm     response = self.post(url, **kwargs)                                                                                                                                                                                         
unit-vault-0: 03:48:42 WARNING unit.vault/0.upgrade-charm   File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.10/site-packages/hvac/adapters.py", line 126, in post                                                                                                               
unit-vault-0: 03:48:42 WARNING unit.vault/0.upgrade-charm     return self.request("post", url, **kwargs)                                     
unit-vault-0: 03:48:42 WARNING unit.vault/0.upgrade-charm   File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.10/site-packages/hvac/adapters.py", line 364, in request                                                                                                            
unit-vault-0: 03:48:42 WARNING unit.vault/0.upgrade-charm     response = super(JSONAdapter, self).request(*args, **kwargs)                   
unit-vault-0: 03:48:42 WARNING unit.vault/0.upgrade-charm   File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.10/site-packages/hvac/adapters.py", line 313, in request                                                                                                            
unit-vault-0: 03:48:42 WARNING unit.vault/0.upgrade-charm     response = self.session.request(                                               
unit-vault-0: 03:48:42 WARNING unit.vault/0.upgrade-charm   File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.10/site-packages/requests/sessions.py", line 542, in request                                                                                                        
unit-vault-0: 03:48:42 WARNING unit.vault/0.upgrade-charm     resp = self.send(prep, **send_kwargs)                                          
unit-vault-0: 03:48:42 WARNING unit.vault/0.upgrade-charm   File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.10/site-packages/requests/sessions.py", line 655, in send                                                                                                           
unit-vault-0: 03:48:42 WARNING unit.vault/0.upgrade-charm     r = adapter.send(request, **kwargs)                                            
unit-vault-0: 03:48:42 WARNING unit.vault/0.upgrade-charm   File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.10/site-packages/requests/adapters.py", line 516, in send                                                                                                           
unit-vault-0: 03:48:42 WARNING unit.vault/0.upgrade-charm     raise ConnectionError(e, request=request)                                      
unit-vault-0: 03:48:42 WARNING unit.vault/0.upgrade-charm requests.exceptions.ConnectionError: HTTPConnectionPool(host='127.0.0.1', port=8220): Max retries exceeded with url: /v1/auth/approle/login (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f9bba586bf0>: Failed to establish a new connection: [Errno 111] Connection refused'))

When manually running juju resolved --all, the vault is able to recover itself but it's sealed, and required the operator to unseal it before COU can upgrade the cloud again. It appears that it only happens during charm refresh

Environment: deployed with stsstack-bundle using ./generate-bundle.sh --name cou -r yoga -s jammy --ceph --run

jneo8 commented 2 days ago

The error status seems can be recovered automatically. We can try to add one resolve step to it.

But we need to discuss the solution how to unseal the vault. Since the vault snap is required and the unseal keys are unknown for COU. The possible solution could be

Option1: COU handle unseal

let the user to provide unseal keys and install snap vault, otherwise the cou upgrade will be blocked. This should be a pre-check step.
- The way to input unseal keys need to be security
COU run juju resolve and vault operator unseal to unseal vault.
The warning message need to be provided when on cou plan

Option2: The blocked, `sealed` status is the expected status of vault.

User need to unseal manually.
More thing to know is because vault is sealed, the nova-compute status will become error. And we can't fix it without unseal.

Information about seal/unseal:

jneo8 commented 2 days ago

Candidate solution

The vault should be the first upgrade application, before all the other applications. Since it's not in the upgrade order.
COU first analysis the cloud to check if the vault application is exists. If exists, COU ask user to give the unseal-keys like a password-input, which should be hidden and can't show on anywhere.
- The number of required unseal-keys could be found by vault status
After refresh the vault, COU unseal the vault by using the vault snap.
- juju exec/run can't be a option here to avoid the the unseal-keys be record in juju logs.

canonical / charmed-openstack-upgrader