Bidirectional host:port discovery Kafka API - Admin API

[r-vasquez]

Who is this for and what problem do they have today?

Some rpk commands use the Kafka API (topic, group, cluster metadata, ...) and other commands use the Admin API (cluster maintenance, acl user, ...) and it's not intuitive to understand which commands require which API.

running a command that uses the Kafka API requires the flag --brokers so rpk knows where to issue the call:

$ rpk cluster info --brokers 127.0.0.1:36175
CLUSTER
=======
redpanda.fc7c5aa1-0ba8-4a81-9c41-932ce7d84d53

BROKERS
=======
ID    HOST       PORT
0*    127.0.0.1  46305
1     127.0.0.1  36175
2     127.0.0.1  38967

Using the --broker flag in a command that requires the admin API will result in an error (see how confusing it is to move from cluster info -> cluster health)

$ rpk cluster health --brokers 127.0.0.1:36175
Request error, trying another node: Get "http://127.0.0.1:9644/v1/cluster/health_overview": dial tcp 127.0.0.1:9644: connect: connection refused
unable to request cluster health: Get "http://127.0.0.1:9648/v1/cluster/health_overview": dial tcp 127.0.0.1:9648: connect: connection refused

To call a command that uses the admin API, the user would have to pass the --api-urls flag:

$ rpk cluster health --api-urls 127.0.0.1:33459
CLUSTER HEALTH OVERVIEW
=======================
Healthy:               true
Controller ID:         0
All nodes:             [2 0 1]
Nodes down:            []
Leaderless partitions: []

What are the success criteria?

To provide a way to discover other admin api host:port — if you query the admin API for a Kafka protocol API, redpanda will handle the request redirecting to the proper host:port, and the other way around too.

Why is solving this problem impactful?

This will solve a lot of the confusion (this is one of the most asked questions) in the rpk command space.

Additional notes

JIRA Link: CORE-1171

[xxchan]

I'm just beginning to play with redpanda and rpk. I met this issue and felt confused. Similar to REDPANDA_BROKERS, what about adding a REDPANDA_API_URLS env var as an easy first step? Guess the discovery is not easy, especially when using docker? 👀

[r-vasquez]

@xxchan You can use: REDPANDA_API_ADMIN_ADDRS env variable :smile:

We will document it once we address https://github.com/redpanda-data/redpanda/issues/4538

[joejulian]

Have you considered adding an env command?

$ export RPK_BROKERS=192.168.1.1:32001
$ rpk env
RPK_BROKERS=192.168.1.1:32001

Which would show environment variables that are set.

and

$ rpk env --all
RPK_ADMIN_API=127.0.0.1:9601
...
RPK_BROKERS=192.168.1.1:32001
...

Would show every available environment variable. It would show defaults if not configured, or whatever is configured even if configured by a configuration file.