Broker warning reported from a client when running tests that pass

[vuldin]

Steps to reproduce:

1. checkout v2 branch
2. helm -n redpanda-ns install redpanda ./redpanda --create-namespace
3. stern -A redpanda- -i WARN
4. helm -n redpanda-ns test redpanda

If you don't have stern command (step 3 above) then just watch the logs for the redpanda broker pods during/after step 4. You will see:

redpanda-ns redpanda-1 redpanda WARN  2022-07-29 01:17:32,724 [shard 0] kafka/client - client.cc:156 - broker_error: kafka::client::broker_error ({ node: -1 }, { error_code: broker_not_available [8] })
redpanda-ns redpanda-2 redpanda WARN  2022-07-29 01:17:35,531 [shard 0] kafka/client - client.cc:156 - broker_error: kafka::client::broker_error ({ node: -1 }, { error_code: broker_not_available [8] })

I've seen this discussed before, pointing to this being the result of an issue around TLS and/or SASL (more investigation is needed). But neither TLS nor SASL was enabled in the above example, so something else is the issue.

The tests that cause these two WARN messages are:

• https://github.com/redpanda-data/helm-charts/blob/c0b88a77036448ed04d0715a1b81e473a10a4b7c/redpanda/templates/tests/test-pandaproxy-status.yaml
• https://github.com/redpanda-data/helm-charts/blob/c0b88a77036448ed04d0715a1b81e473a10a4b7c/redpanda/templates/tests/test-schemaregistry-status.yaml

JIRA Link: K8S-12

[vuldin]

This message also appears within github action workflows (or locally with ct install --config redpanda/ci/ct.yaml).

[joejulian]

Please retest this with 2.0.0