Scram over mtls phase3 - proper token wait

[rukai]

This PR replaces the hardcoded 4s wait with logic to check that all brokers have received the new delegation token. This is done by sending a DescribeDelegationToken request to all brokers until we receive a response that contains the delegation token we just created. This is done such that we confirm one node has received the token before moving onto the next node. This minimizes requests sent.

Alternatively we could wait until one node has received the token. (a node that isnt the node we created the token on) And then at that point send requests to all other nodes in parallel. This will cut off a bit of latency in token creation. If we need to improve latency in the future this would be useful to attempt, but I've left it out of this PR to keep this PR simple.

Further notes:

• A limitation of this PR is it requires all brokers to be included in the mtls_port_contact_points config field. This is needed because the token task does not yet have broker discovery implemented. That will be done in a follow up PR.
• connection creation is now done up front for all nodes to ensure that the connections are available later on when we check that the token has propagated to all nodes.
• I included an info level trace describing that a delegation token was created and how long it took, since token creation is kind of rare and should only happen once per user per shotover instance, it will be useful to have insight into token creation without being noisy.

[codspeed-hq[bot]]

CodSpeed Performance Report

Merging #1627 will not alter performance

_{Comparing rukai:scram_over_mtls_phase3 (01eb50a) with main (bafb058)}

Summary

✅ 37 untouched benchmarks