Encounter a SecretNotFoundError when removing relation with certificates operator (when using CertHandler)

[shayancanonical]

Bug Description

We encounter a SecretNotFoundError when we remove relation with certificates operator and we are using CertHandler and are related to tempo-k8s

To Reproduce

In a microk8s model:

1. git clone git@github.com:canonical/cos-lite-bundle.git
2. tox -e render-edge
3. juju deploy ./bundle.yaml --edge
4. juju deploy -n 1 tempo-k8s --channel edge
5. juju deploy -n 1 self-signed-certificates
6. jhack imatrix fill
7. juju offer tempo-k8s:tracing
8. juju offer self-signed-certificates:certificates

In an lxd model:

1. juju deploy -n 1 mysql --channel 8.0/edge
2. juju consume :admin/.tempo-k8s
3. juju consume :admin/.self-signed-certificates
4. juju relate mysql:tracing tempo-k8s:tracing
5. juju relate mysql:tracing-certificates self-signed-certificates:certificates
6. wait till everything settles into active/idle
7. juju remove-relation mysql:tracing-certificates self-signed-certificates:certificates

Environment

juju: 3.4.2 lxd: 5.21.1 LTS MicroK8s v1.27.13 revision 6744 cert_handler.py: v1.8

Relevant log output

unit-mysql-0: 13:19:53 ERROR unit.mysql/0.juju-log tracing-certificates:4: Exception while exporting Span batch.                                                                                                   
Traceback (most recent call last):                                                                                                                                                                                 
  File "/var/lib/juju/agents/unit-mysql-0/charm/venv/opentelemetry/sdk/trace/export/__init__.py", line 368, in _export_batch                                                                                       
    self.span_exporter.export(self.spans_list[:idx])  # type: ignore                                                                                                                                               
  File "/var/lib/juju/agents/unit-mysql-0/charm/venv/opentelemetry/exporter/otlp/proto/http/trace_exporter/__init__.py", line 145, in export                                                                       
    resp = self._export(serialized_data)                                                                                                                                                                           
  File "/var/lib/juju/agents/unit-mysql-0/charm/venv/opentelemetry/exporter/otlp/proto/http/trace_exporter/__init__.py", line 114, in _export                                                                      
    return self._session.post(                                                                                                                                                                                     
  File "/var/lib/juju/agents/unit-mysql-0/charm/venv/requests/sessions.py", line 637, in post                                                                                                                      
    return self.request("POST", url, data=data, json=json, **kwargs)                                                                                                                                               
  File "/var/lib/juju/agents/unit-mysql-0/charm/venv/requests/sessions.py", line 589, in request                                                                                                                   
    resp = self.send(prep, **send_kwargs)                                                                                                                                                                          
  File "/var/lib/juju/agents/unit-mysql-0/charm/venv/requests/sessions.py", line 703, in send                                                                                                                      
    r = adapter.send(request, **kwargs)                                                                                                                                                                            
  File "/var/lib/juju/agents/unit-mysql-0/charm/venv/requests/adapters.py", line 458, in send                                                                                                                      
    self.cert_verify(conn, request.url, verify, cert)                                                                                                                                                              
  File "/var/lib/juju/agents/unit-mysql-0/charm/venv/requests/adapters.py", line 261, in cert_verify                                                                                                               
    raise OSError(                                                                                                                                                                                                 
OSError: Could not find a suitable TLS CA certificate bundle, invalid path: /var/snap/charmed-mysql/common/var/run/tracing-ca.crt                                                                                  
unit-mysql-0: 13:19:53 ERROR unit.mysql/0.juju-log tracing-certificates:4: Uncaught exception while in charm code:                                                                                                 
Traceback (most recent call last):                                                                                                                                                                                 
  File "/var/lib/juju/agents/unit-mysql-0/charm/venv/ops/model.py", line 2955, in _run                                                                                                                             
    result = subprocess.run(args, **kwargs)  # type: ignore                                                                                                                                                        
  File "/usr/lib/python3.10/subprocess.py", line 526, in run                                                                                                                                                       
    raise CalledProcessError(retcode, process.args,                                                                                                                                                                
subprocess.CalledProcessError: Command '('/var/lib/juju/tools/unit-mysql-0/secret-info-get', '--label', 'cert-handler-private-vault', '--format=json')' returned non-zero exit status 1.                           

The above exception was the direct cause of the following exception:                                                                                                                                               

Traceback (most recent call last):                                                                                                                                                                                 
  File "/var/lib/juju/agents/unit-mysql-0/charm/venv/ops/model.py", line 3321, in _run_for_secret                                                                                                                  
    return self._run(*args, return_output=return_output, use_json=use_json)                                                                                                                                        
  File "/var/lib/juju/agents/unit-mysql-0/charm/venv/ops/model.py", line 2957, in _run                                                                                                                             
    raise ModelError(e.stderr) from e                                                                                                                                                                              
ops.model.ModelError: ERROR secret "cert-handler-private-vault" not found                                                                                                                                          

The above exception was the direct cause of the following exception:                                                                                                                                               

Traceback (most recent call last):                                                                                                                                                                                 
  File "/var/lib/juju/agents/unit-mysql-0/charm/./src/charm.py", line 898, in <module>                                                                                                                             
    main(MySQLOperatorCharm)                                                                                                                                                                                       
  File "/var/lib/juju/agents/unit-mysql-0/charm/venv/ops/main.py", line 436, in main                                                                                                                               
    _emit_charm_event(charm, dispatcher.event_name)                                                                                                                                                                
  File "/var/lib/juju/agents/unit-mysql-0/charm/venv/ops/main.py", line 144, in _emit_charm_event                                                                                                                  
    event_to_emit.emit(*args, **kwargs)                                                                                                                                                                            
  File "/var/lib/juju/agents/unit-mysql-0/charm/venv/ops/framework.py", line 351, in emit                                                                                                                          
    framework._emit(event)                                                                                                                                                                                         
  File "/var/lib/juju/agents/unit-mysql-0/charm/venv/ops/framework.py", line 853, in _emit                                                                                                                         
    self._reemit(event_path)                                                                                                                                                                                       
  File "/var/lib/juju/agents/unit-mysql-0/charm/venv/ops/framework.py", line 942, in _reemit                                                                                                                       
    custom_handler(event)                                                                                                                                                                                          
  File "/var/lib/juju/agents/unit-mysql-0/charm/lib/charms/observability_libs/v1/cert_handler.py", line 572, in _on_certificates_relation_broken                                                                   
    self.vault.clear()                                                                                                                                                                                             
  File "/var/lib/juju/agents/unit-mysql-0/charm/lib/charms/observability_libs/v1/cert_handler.py", line 263, in clear                                                                                              
    self._backend.clear()                                                                                                                                                                                          
  File "/var/lib/juju/agents/unit-mysql-0/charm/lib/charms/observability_libs/v1/cert_handler.py", line 240, in clear                                                                                              
    self._secret.remove_all_revisions()                                                                                                                                                                            
  File "/var/lib/juju/agents/unit-mysql-0/charm/venv/ops/model.py", line 1420, in remove_all_revisions                                                                                                             
    self._id = self.get_info().id                                                                                                                                                                                  
  File "/var/lib/juju/agents/unit-mysql-0/charm/venv/ops/model.py", line 1314, in get_info                                                                                                                         
    return self._backend.secret_info_get(id=self.id, label=self.label)                                                                                                                                             
  File "/var/lib/juju/agents/unit-mysql-0/charm/venv/ops/model.py", line 3335, in secret_info_get                                                                                                                  
    result = self._run_for_secret('secret-info-get', *args, return_output=True, use_json=True)                                                                                                                     
  File "/var/lib/juju/agents/unit-mysql-0/charm/venv/ops/model.py", line 3324, in _run_for_secret                                                                                                                  
    raise SecretNotFoundError() from e                                                                                                                                                                             
ops.model.SecretNotFoundError                                                                                                                                                                                      
unit-mysql-0: 13:19:53 ERROR juju.worker.uniter.operation hook "tracing-certificates-relation-broken" (via hook dispatching script: dispatch) failed: exit status 1

Additional context

No response

[ca-scribner]

not sure I understand why, but this didn't close when #100 closed. @PietroPasotti should this be closed?

[PietroPasotti]

perhaps because it didn't merge to main but to leon's branch. Yes, closing.