[DPE-3928] - feat: secrets integration

[marcoppenheimer]

Features

feat: zookeeper, peer and client data secrets supported

• Includes abstracted unified unified secrets + peer data
• More resilient relation interface with ZooKeeper to support zookeeper interface, and upcoming zookeeper_client interface, optionally using secrets

  chore: bump data_interfaces to v34

• Introduces a whole bunch of new stuff, predominantly unified secrets + peer data

Refactoring

chore: support tls + tls-ca and ca + ca-cert

• Secret labels need to be >2 characters. For backwards compatibility, we now:
  • Look for ca-cert and ca in peer-data, cleaning up old ones when we get granted a new CA
  • Look for bool(tls-ca) from ZooKeeper, defaulting to original tls=enabled

    refactor: align with ZooKeeper charm patterns

• provider.update_connection_info() --> charm.update_client_data() with new state.clients + models.KafkaClient
• literals.Substrate --> core.models.SUBSTRATES + literals.SUBSTRATE
• state.broker --> state.unit_broker, mostly for clarity that this is 'the unit'

  test: replace model_full_name with ops_test

• Maybe it's easier to use, not sure, but removes amount of non-test code in tests

  test: updating int-test helpers

• Most pertinent change here is how we get Kafka + ZK data that both optionally use secrets, or Kafka provider data. Solved by getting every relation-data key and merging them with every secret key

  test: sync unit-tests with K8s

• I think they got a bit out of date, have updated and synced, especially the upgrade tests

Bug Fixes

fix: explicitly emit config_changed events where necessary

• Before, we were using self._on_config_changed(event) whenever we wanted to do things
• When config_changed deferred, it re-ran the whole event, which caused a whole lot of restarting issues and non-idempotent event handling and hard-to-grok event ordering
• For example, we ran certificates-available, which called config_changed, which deferred. In certificates-available, we have a check that says 'If I already have a cert, restart'. So after we deferred the event, we re-run it, have a cert, and restart, which we may not have wanted
• Explicitly emitting a config-changed event helps avoid issues like that, as only the 'changed' event gets deferred

  chore: avoid ruff warnings

• As ruff is still in active dev, need to update pyproject.toml to avoid warnings

  test: add async sleeps to int-tests

• It maybe a bug in the secrets implementation, but secrets seem to be really slow to update, or to trigger secret-changed events where before we had 'instant' relation-changed events
• This has caused a bunch of instability in the int-tests, so adding fast-forward: asyncio.sleep has helped somewhat. Will keep looking at this

[deusebio]

Just discussed offline with Marc. Actually I did not realized that this pattern is also like this in ZK, here. Although also Marc is not fond of this, we can just keep it as-is for the time being, but I would like to improve this in the next weeks when we are together in Madrid.