Rook API server pod should be optional

[bassam]

The Rook API server should be exposed directly by the Operator Pods. Its unclear why we need an API pod to run in every rook cluster/namespace.

[travisn]

I think we actually want to keep the api service specific to a cluster to make it convenient for command-line users of rookctl. The operator just won't have a dependency on it per #703.

[bassam]

convenient in what way? Are you talking about Kuberenetes namespaces and RBAC?

[travisn]

It's really only a question of whether we want to support rookctl inside kubernetes. It's not really a scenario since a kubernetes user would be using TPRs other native primitives. But it does seem useful to support for consistent cross-platform support (kubernetes/standalone) for users trying the different platforms.

[bassam]

if we did want to continue support rookctl inside a K8S cluster, could we not serve the API endpoints from the Operator pods?

[travisn]

it would require a change to the api to require the cluster name in the routes, which doesn't seem like the right design. If the client has an endpoint, it should be able to assume it's talking to a single cluster.

[bassam]

I think there are pros and cons to both approaches. The convenience of defaulting to a single cluster can be solved this with default the cluster to be the current one rookctl pod is running in. For example rookctl --cluster foo <command> where the cluster default is based on the current namespace when running in K8S.

FWIW, ceph the tool support operating on multiple clusters.

[DanKerns]

Is multicluster an important use case? If it's just for multitenancy then maybe there is a more kubernetes-centric way to address that.

[travisn]

the kubernetes-centric way would seem to me that you have a service endpoint for each cluster. In our case, that would be an api service in each namespace where rook is running

[travisn]

With changes coming in #974 the dependency on the operator will be removed (#703). This makes the api service an optional component. Because it is still required for standalone and other scenarios, we can keep it as an option on the cluster CRD.

[bassam]

we can keep it as an option on the cluster CRD.

cluster or operator CRD?

[travisn]

the api service today belongs to the cluster instance. If we want the api to be single instance with the operator, it would require some redesign of the api service.

[bassam]

whats the plan for this?

[travisn]

this plan is to make it optional for now. It's still needed for the toolbox

[bassam]

which part of the toolbox needs it? health and status? if so can we switch these to use ceph tools directly.

[travisn]

specifically, rookctl uses the api. If we get rid of the api i think we're saying we get rid of the majority of functionality in rookctl

[bassam]

ahh ok, I'm an advocate of removing rookctl in favor of just ceph tools. and removing the rook API completely.

[travisn]

@bassam is removing the API completely really feasible? The crds and kubectl api are great for creating, deleting, and updating resources. They also have the ability to provide basic status info (which we aren't using right now), but they don't appear to be a fit for more detailed or dynamic status. If there is a client that needs an API, it's not feasible to go launch the toolbox to run ceph tools. The rest API gives us that ability to expose ceph commands when there is not another API. In the long term, we would certainly want to push this functionality into the ceph rest api or other apis, but until that happens I don't see how we avoid some api surface in Rook to serve the more complex requests.

[jbw976]

we discussed this in the Rook meeting this morning. We want to get to a place where we only rely on CRDs, Ceph tools, and the Ceph REST API. Those should be the only interaction points for users of Rook.

An unanswered question would be how best to expose metrics/status to prometheus. That is currently done through the Rook REST API's /metrics endpoint.

We should update the docs saying that rookctl and the Rook REST API are deprecated.

[bassam]

An unanswered question would be how best to expose metrics/status to prometheus. That is currently done through the Rook REST API's /metrics endpoint.

there is a ceph-mgr module to export prometheus metrics. is that an option?

[jbw976]

ah right, i forgot about that new ceph-mgr module. would we need to add that to our CMakeLists so it gets compiled and included in the image?

[DanKerns]

We also discussed coordinating with Sig-API as that makes sense.

[travisn]

No need for making the api optional since it is being removed in #1474