canonical / mysql-router-k8s-operator

Mysql router operator charm for kubernetes
https://charmhub.io/mysql-router-k8s
Apache License 2.0
2 stars 6 forks source link

[DPE-4173] Stabilize exporter tests by using listen-port to avoid ephemeral ports #277

Closed shayancanonical closed 1 month ago

shayancanonical commented 2 months ago

Issue

We are finding connections in TIME_WAIT between router exporter and mysql (still investigating what is happening here). Due to pebble's on-failure: restart, the exporter service eventually is able to start up when the connection in TIME_WAIT terminates after 60s - however, our retry in the test is 3 mins sometimes does not provide enough time for pebble's exponential backoff to work

Solution

Extend the timeout to 7 mins

shayancanonical commented 2 months ago

Responded in https://github.com/canonical/mysql-router-operator/pull/154#issuecomment-2203069565. I think it is still valid to have the test. We are allowing room for error (in particular the bind-address-in-use error) with the extended timeout, but it is still valuable to test integration with COS. I will create a ticket to continue investigating the issue, but there is an issue reported upstream from which I am still waiting to hear back