Proxy Admin API commands

[patrickangeles]

Who is this for and what problem do they have today?

Requests via the Redpanda Admin API will cause the broker to redirect some requests to another broker. For example:

curl  -v https://redpanda-0/v1/security/users -X POST -H 'Content-Type: "application/vnd.kafka.json.v2+json"' -d '{"username":"xxx","password":"xxx","algorithm":"SCRAM-SHA-256"}'
....
< HTTP/1.1 307 Temporary Redirect
< Content-Length: 0
< Date: Thu, 19 May 2022 13:54:43 GMT
< Content-Type: application/json
< Server: Seastar httpd
< Location: https://redpanda-2/v1/security/users

In a Kubernetes environment, this means that Admin API host endpoints need to be exposed via LB per node. Ideally the behavior could be similar to Pandaproxy where the initial broker proxies an internal request to the correct broker, instead of doing an HTTP forward.

What are the success criteria?

Why is solving this problem impactful?

Additional notes

JIRA Link: CORE-916

[patrickangeles]

Related: https://github.com/redpanda-data/redpanda/issues/2865

[mattschumpert]

What's the underlying reason why we need to redirect to another broker for user creation?

[mattschumpert]

Is https://github.com/redpanda-data/redpanda/issues/1029 the alternative then?

[mattschumpert]

@jcsp can you comment on this? When and why are we forwarding API commands to other brokers

[dotnwat]

Some commands are only handled by the controller. However, clients may have incomplete knowledge and send requests to a non-controller node. In this case http redirects are used to guide the client to the controller.

[jcsp]

Some commands are only handled by the controller. However, clients may have incomplete knowledge and send requests to a non-controller node. In this case http redirects are used to guide the client to the controller.

Like Noah says - and the request in this ticket is to have Redpanda deal with that internally rather than redirecting the client.

The terminology gets messy around this, sometimes people say things they don't mean. The terms are:

• redirection: redpanda receives a command from a client, then asks the client to talk to a different redpanda node instead.
• proxying/forwarding: redpanda receives a command from a client, the internally talks to a different redpanda node to fulfil it

[mattschumpert]

Thanks @jcsp

@patrickangeles why is this particular environment not happy with the HTTP redirect (is the redirected IP/host incorrect from the client's perspective)? Also, if its a problem why isn't it a problem for every user deploying Redpanda on K8s?

[patrickangeles]

A few stated reasons:

• Requires an LB configured per admin endpoint
• Desire to funnel all admin requests through a single endpoint for security
• easier for tooling/scripts not to have to deal w/ redirects

[mattschumpert]

@alenkacz can you comment here? It seems like a generalized problem with anyone using K8s (all nodes need a public endpoint for admin API). DO you agree?

[alenkacz]

@patrickangeles I understand the part about having LB to funnel all requests to admin api via one endpoint. I am a bit lost about why we need LB per broker. We publish all admin api addresses in status so if you know, which one is the controller, you can talk to the node directly. Could you help me understand?

example status from cluster:

"externalAdmin": [
            "2.rp-x.xxx.byoc.vectorized.cloud:31124",
            "1.rp-x.xxx.byoc.vectorized.cloud:31124",
            "0.rp-x.xxx.byoc.vectorized.cloud:31124"
          ],

[michaelandrepearce]

because atm the admin api returns its instance as a redirect, as such a client is redirected, and a individual route (lb) to each instance has to exist and fw's etc. even if you knew which node is controller to avoid the redirect, that means any node could be controller and therefor a route directly to any in your infra has to exists, which does not work in secured envs and then scaling up and down, as you have to manage that with fw's manually and also security really dont like multiple large swathes of routes being allowed for admin functions. (e.g. we have push back on RP adoption because of this)

by having a single lb for the cluster and if a call hits any node the call is forwarded/proxied to correct node (as like PP and SR does) then you don't have to create routes for every instance, and maintain fw's etc for admin api.

and this way from a security/fw pov externally you only need to open up route/fw's etc, to that single lb endpoint, thus sec team happy, and also now you can scale up and down cluster in k8s without having to worry about having to add externally addtional routes and fws.

lasty then also configuration management tooling need to just curl/http request the api and not do anything fancy in logic, e.g. it would be in very basic a single curl without needing to add follow redirect avoiding issues where some tooling may not unfortunately support redirect, or have redirect disabled for sec reasons.

[emaxerrno]

digesting. let me think about this for a few. will post back.

[mattschumpert]

@graphcareful have you thought any more about how this could be implemented?

[DarranCooke]

Hi @mattschumpert is there an update on this, please?