[DPE-5097] - test: add mtls int-tests

[marcoppenheimer]

Changes Made

test: add mtls int-tests

• Different to VM in several ways:
  • Because it's a container, needs to manually install Java + wget (apt) and Kafka (Apache download)
• The run-mtls-producer and get-offsets app-charm actions take mtls-nodeport instead of bootstrap-server, so the mTLS test assumes/requires NodePort to be enabled

  feat: add LRU cache for K8s api calls with 2m ttl

• Cache is in-memory, so does not persist across charm hook execution. Did explore pickle-ing this, but was fiddly so left it out
• get_ttl_hash() sets a 2m ttl for the in-memory cache, in-case of long-running event executions that may require recently-changed results
• Originally suspected that flakiness in int-tests were due to K8s API server slowdowns from spamming Pod/Node/Service requests
  • kube-apiserver has --max-requests-inflight=400 as the default, so it's unlikely that this is the issue
  • BUT, it would be an issue for deployments with lots of units, so the feature is suggested to stick around

[deusebio]

Oh, there is another thought I had:

It seems that we are only testing the trusted-certificates behaviour, and not the trusted-ca. Do we have a test for this? If not, I'd also like to test this scenario. If we can use a lot of code already written here, and this would be reasonably easy, I would make it part of this PR, otherwise I'm also fine to have a dedicated ticket. But I'd approach this next pulse

[marcoppenheimer]

@deusebio

Do we have a test for this?

No we don't, mostly to save headache of writing more fiddly mTLS tests for the 'easier' case of CA trusting (vs individual cert). Happy to for it to be a separate ticket, as both K8s + VM will need it 👍🏾

[zmraul]

It seems that we are only testing the trusted-certificates behaviour, and not the trusted-ca.

We don't, but the charm is executing the exact same code on both cases, minus a check on the relation name to get certificate or CA. It's good to have that test, but I don't think is critical IMO.