[BUG] Non-leader tyirng to create secret

[juditnovak]

Steps to reproduce

Non-deterministic issue. See pipeline attached for onw way it was triggered.

Actual behavior

unit-opensearch-0: 11:23:31 ERROR unit.opensearch/0.juju-log certificates:6: Uncaught exception while in charm code:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-opensearch-0/charm/venv/ops/model.py", line 3180, in _run
    result = subprocess.run(args, **kwargs)  # type: ignore
  File "/usr/lib/python3.10/subprocess.py", line 526, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '('/var/lib/juju/tools/unit-opensearch-0/secret-add', '--label', 'opensearch:app:app-admin', '--owner', 'application', 'truststore-***')' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-opensearch-0/charm/./src/charm.py", line [213](https://github.com/canonical/opensearch-dashboards-operator/actions/runs/10007015106/job/27661171763#step:22:214), in <module>
    main(OpenSearchOperatorCharm)
  File "/var/lib/juju/agents/unit-opensearch-0/charm/venv/ops/main.py", line 548, in main
    manager.run()
  File "/var/lib/juju/agents/unit-opensearch-0/charm/venv/ops/main.py", line 527, in run
    self._emit()
  File "/var/lib/juju/agents/unit-opensearch-0/charm/venv/ops/main.py", line 516, in _emit
    _emit_charm_event(self.charm, self.dispatcher.event_name)
  File "/var/lib/juju/agents/unit-opensearch-0/charm/venv/ops/main.py", line 147, in _emit_charm_event
    event_to_emit.emit(*args, **kwargs)
  File "/var/lib/juju/agents/unit-opensearch-0/charm/venv/ops/framework.py", line 348, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-opensearch-0/charm/venv/ops/framework.py", line 860, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-opensearch-0/charm/venv/ops/framework.py", line 950, in _reemit
    custom_handler(event)
  File "/var/lib/juju/agents/unit-opensearch-0/charm/lib/charms/tls_certificates_interface/v3/tls_certificates.py", line 1811, in _on_relation_changed
    self.on.certificate_available.emit(
  File "/var/lib/juju/agents/unit-opensearch-0/charm/venv/ops/framework.py", line 348, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-opensearch-0/charm/venv/ops/framework.py", line 860, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-opensearch-0/charm/venv/ops/framework.py", line 950, in _reemit
    custom_handler(event)
  File "/var/lib/juju/agents/unit-opensearch-0/charm/lib/charms/opensearch/v0/opensearch_tls.py", line 209, in _on_certificate_available
    self.store_new_ca(self.charm.secrets.get_object(scope, cert_type.val))
  File "/var/lib/juju/agents/unit-opensearch-0/charm/lib/charms/opensearch/v0/opensearch_tls.py", line 446, in store_new_ca
    self._create_keystore_pwd_if_not_exists(Scope.APP, CertType.APP_ADMIN, "ca")
  File "/var/lib/juju/agents/unit-opensearch-0/charm/lib/charms/opensearch/v0/opensearch_tls.py", line 434, in _create_keystore_pwd_if_not_exists
    self.charm.secrets.put_object(
  File "/var/lib/juju/agents/unit-opensearch-0/charm/lib/charms/opensearch/v0/opensearch_secrets.py", line 359, in put_object
    self._add_or_update_juju_secret(scope, key, value, merge)
  File "/var/lib/juju/agents/unit-opensearch-0/charm/lib/charms/opensearch/v0/opensearch_secrets.py", line 272, in _add_or_update_juju_secret
    return self._add_juju_secret(scope, key, value)
  File "/var/lib/juju/agents/unit-opensearch-0/charm/lib/charms/opensearch/v0/opensearch_secrets.py", line [228](https://github.com/canonical/opensearch-dashboards-operator/actions/runs/10007015106/job/27661171763#step:22:229), in _add_juju_secret
    secret = scope_obj.add_secret(safe_value, label=label)
  File "/var/lib/juju/agents/unit-opensearch-0/charm/venv/ops/model.py", line 477, in add_secret
    id = self._backend.secret_add(
  File "/var/lib/juju/agents/unit-opensearch-0/charm/venv/ops/model.py", line 3624, in secret_add
    result = self._run('secret-add', *args, return_output=True)
  File "/var/lib/juju/agents/unit-opensearch-0/charm/venv/ops/model.py", line 3182, in _run
    raise ModelError(e.stderr) from e
ops.model.ModelError: ERROR this unit is not the leader

Versions

See all on the pipeline. https://github.com/canonical/opensearch-dashboards-operator/actions/runs/10007015106/job/27661171763#step:22:211

[github-actions[bot]]

https://warthogs.atlassian.net/browse/DPE-4900

[reneradoi]

Bugfix was published with revision 119.