Create kubectl plugin as additional tool for configuring and troubleshooting Rook

[travisn]

Is this a bug report or feature request?

• Feature Request

What should the feature do: Create a Krew plugin to extend kubectl commands for Rook.

What is use case behind this feature: kubectl plugins provide additional conveniences for running kubectl commands. This can be a very helpful tool for troubleshooting or automating steps that today are manual or not obvious to collect. This is another tool that can help supplement our documentation around advanced configuration or disaster recovery, besides some common commands to simplify the life of Rook users.

Here are some commands we might consider:

• kubectl rook get-mon-endpoints
• kubectl rook set-mon-endpoints: For disaster recovery, update the rook-ceph-mon-endpoints configmap and restart the operator
• kubectl rook force-mon-quorum <mon>: Automate the disaster recovery guide for forcing the mon quorum to a single mon
• kubectl rook get-volumes [rwx|rwo|cephfs|rbd]
• kubectl rook ceph <args>: Run any ceph command in the toolbox
• kubectl rook operator restart: Restart the operator
• kubectl rook operator set <property> <value>: Set the property in the rook-ceph-operator-config configmap, for example to enable debug logging or configure the CSI driver
• kubectl rook health: Shortcut to “ceph status” command, or else pretty print the status on the CephCluster CR
• kubectl rook version: Print the version of Rook and the version of Ceph daemons from the CephCluster CR status
• kubectl rook status: Print the phase and/or conditions of every Ceph CR
• kubectl rook purge-osd <id>: Replace the purge-osd.yaml example job
• kubectl rook set-pool-failure-domain <pool> <domain>: Replace the advanced topic for updating a pool's failure domain

All commands would default to the rook-ceph namespace. For other namespaces pass -n <namespace>, or pick up the environment variable ROOK_NAMESPACE.

Likely we should create a new repo for this tool, no need to version it with the main Rook releases.

[subhamkrai]

@travisn can I take this?

[Madhu-1]

Maybe add commands for cephcsi also?

[BlaineEXE]

Rook is not synonymous with Ceph. I think we should have the commands be kubectl ceph, kubectl rook-ceph, or make the kubectl rook plugin require a backend selection kubectl rook ceph...

[subhamkrai]

Rook is not synonymous with Ceph. I think we should have the commands be kubectl ceph, kubectl rook-ceph, or make the kubectl rook plugin require a backend selection kubectl rook ceph...

currently how I have implemented is kubectl will be prefix and rook will be root command which will accept 2 arguments(till now will add more as more command is added) 1 is ceph for ceph related command likekubectl rook ceph status and 2nd is operator kubectl rook operator restart

[travisn]

The tool really seems like it would need to be specialized for a storage provider. This would imply that each storage provider implements its own tool in its own repo. And as awkward as it is to append storage provider names to rook, it seems the tool would need to be called rook-ceph, which would mean commands such as:

• kubectl rook-ceph operator restart
• kubectl rook-ceph ceph status

Another possibility is that the storage providers get a name simply of the storage provider, and drop the rook prefix. This would give commands like:

• kubectl ceph operator restart
• kubectl ceph status

The latter is cleaner, but we lose the naming that it belongs to rook. I'm leaning more towards the simplicity than keeping rook in the name.

As far as the repo name, the pattern for other krew plugins seems to be kubectl-<pluginName>.

[BlaineEXE]

I support calling it ceph also.

I think the only oddity about naming it ceph is for running ceph CLI commands. I think kubectl ceph run <command>... would be good. But not all ceph tools use the ceph CLI tool. Should we support running rbdtool/radosgw-admin also? It might be kubectl ceph run ceph status or kubectl ceph run radosgw-admin ..... We would want to allow-list the commands to run, though so people can't just run whatever they want.

[travisn]

What about this?

• The tool optimizes some commands and runs them in the operator pod, no toolbox necessary:
  • kubectl ceph status
• The toolbox must already be running, and any arbitrary command can be executed there.
  • kubectl ceph toolbox <any command that you could run in the toolbox>:

[BlaineEXE]

What about this?

* The tool optimizes some commands and runs them in the operator pod, no toolbox necessary:

  * `kubectl ceph status`

I think that makes some amount of sense. In reality, we may want to just pass through any unknown command as though it's a command directly to the ceph cli.

• We could just pass through status, osd pool ls to ceph, etc as long as we don't have special commands for those.
• A combined Rook and Ceph status could be kubectl ceph rook status, and the rook command could be a good umbrella for rook-related things like kubectl ceph rook resources (to list the Rook CRs).

* The toolbox must already be running, and any arbitrary command can be executed there.

  * `kubectl ceph toolbox <any command that you could run in the toolbox>`:

I think toolbox already running OR we can run the command in a toolbox job. :)

---

For example:

• kubectl ceph rook status - return status of all rook resources as well as ceph status
• kubectl ceph rook operator restart - restart rook-ceph operator
• kubectl ceph rook operator config set ROOK_LOG_LEVEL=DEBUG - set a rook config like the log level
• kubectl ceph rook operator config unset ROOK_LOG_LEVEL - unset a rook config to go back to the default
• kubectl ceph status - run ceph status in the operator pod
• kubectl ceph <any other command> - run ceph <any other command> in the operator pod

[subhamkrai]

created this checklist from the issue so that we have the update and any new suggestion can be added here FYI @travisn @leseb @BlaineEXE @sp98 @Madhu-1 @parth-gr

• [X] kubectl rook get-mon-endpoints
• [ ] kubectl rook set-mon-endpoints: For disaster recovery, update the rook-ceph-mon-endpoints configmap and restart the operator
• [ ] kubectl rook force-mon-quorum <mon>: Automate the disaster recovery guide for forcing the mon quorum to a single mon
• [ ] kubectl rook get-volumes [rwx|rwo|cephfs|rbd]
• [x] kubectl rook ceph <args>: Run any ceph command in the toolbox
• [x] kubectl rook operator restart: Restart the operator
• [x] kubectl rook operator set <property> <value>: Set the property in the rook-ceph-operator-config configmap, for example to enable debug logging or configure the CSI driver
• [ ] kubectl rook health: Shortcut to “ceph status” command, or else pretty print the status on the CephCluster CR
• [ ] kubectl rook version: Print the version of Rook and the version of Ceph daemons from the CephCluster CR status
• [ ] kubectl rook status: Print the phase and/or conditions of every Ceph CR
• [ ] kubectl rook purge-osd <id>: Replace the purge-osd.yaml example job
• [ ] kubectl rook set-pool-failure-domain <pool> <domain>: Replace the advanced topic for updating a pool's failure domain

[BlaineEXE]

I think this is a great discussion. Naming is one of the hardest problems in development, and here we are really determining what the user experience of this tool will be. Having the tool be as easy to use and understand for users is, in my opinion, the most important consideration for this development. That said, I think we have discussed 3 options for how the plugin can be named.

1. kubectl-rook
2. kubectl-rook-ceph
3. kubectl-ceph

I'll list out some of the commands the use might run below, and I'll try to offer a good pro-con for all of them.

I will start with some assumptions:

1. For running ceph CLI commands, it is best for users to make the experience as close to that documented by the Ceph docs as possible.
   • To me, this means that ... ceph <subcommand> (e.g., kubectl rook ceph osd pool ls) is the correct user experience.
   • Compare that to something like kubectl rook run <subcommand> (e.g., kubectl rook run osd pool ls)
2. For setting operator configs, it is best for users to have a config subcommand.
   • kubectl rook operator set is unclear what is being set on the CLI, and kubectl rook operator config set is much more clear. This also follows the kubectl config ... precedent.
3. This plugin is for Ceph only. We will not accept features to the plugin for NFS, Cassandra, or any future backends

kubectl-rook

• root command: kubectl rook
• restart operator: kubectl rook operator restart
• get operator configs: kubectl rook operator config get
• set operator config: kubectl rook operator config set <prop> <val>
• show health of cluster and CRs: kubectl rook health
• show status of all CRs: kubectl rook status
• run a ceph command (in operator): kubectl rook ceph <command> (e.g., ceph status)
• run any toolbox command: kubectl rook toolbox <command> (e.g., radosgw-admin ...)

PROS	CONS
kubectl rook is a simple and clear root command	This command will make the experience for future backends worse

kubectl-ceph

• root command: kubectl ceph
• restart operator: kubectl ceph operator restart OR kubectl ceph rook operator restart
• get operator configs: kubectl ceph operator config get OR kubectl ceph rook operator config get
• set operator config: kubectl ceph operator config set <prop> <val> OR kubectl ceph rook operator config set <prop> <val>
• show health of cluster and CRs: kubectl ceph health OR kubectl ceph rook health
• show status of all CRs: kubectl ceph status OR kubectl ceph rook status
• run a ceph command (in operator): kubectl ceph <command>
• run any toolbox command: kubectl ceph toolbox <command>

PROS	CONS
kubectl ceph is a simple root command that won't conflict with future backend plugins
this makes running ceph commands the first-class citizen	this makes running ceph commands the first-class citizen
	our own subcommands may override ceph cli commands (e.g. if kubectl ceph status has a special implementation)
	using kubectl ceph rook <command> to run commands to work around the above issue is more verbose

kubectl-rook-ceph

• root command: kubectl rook-ceph
• restart operator: kubectl rook-ceph operator restart
• get operator configs: kubectl rook-ceph operator config get
• set operator configs: kubectl rook-ceph operator config set <prop> <val>
• show health of cluster and CRs: kubectl rook-ceph health
• show status of all CRs: kubectl rook-ceph status
• run a ceph command (in operator): kubectl rook-ceph ceph <command>
• run any toolbox command: kubectl rook-ceph toolbox <command>

PROS	CONS
kubectl rook-ceph won't conflict with future backend plugins and is the most clear	it is also the most verbose root command
	kubectl rook-ceph ceph <cmd> stutters with "...-ceph ceph..."

[BlaineEXE]

After typing out all of the above, I think the best option is to use rook-ceph as the root command. It is the most clear in my opinion, and the extra verbosity is minor.

If the verbosity is an issue for anyone, that is a very easily fixed problem using aliases. In my own environment, I would likely use alias rookctl='kubectl rook-ceph' (or alias cephctl='kubectl rook-ceph'???), and we could even suggest that to users.

[additional] I suspect that once krew is more established, kubectl will start offering tab completion of the plugin name (root command), which will help users who might have multiple rook-<backend> plugins installed if that ever happens in the future.

[travisn]

I'm not worried about the verbosity either of rook-ceph vs ceph. It's more about what makes the most sense for usability. From our discussion, the clarifying usability question is really the difference between a control plane command and a raw ceph command that should be run directly (either in the toolbox or operator pod). I agree that the most clear option is to call the tool rook-ceph so that everything by default is a control plane command. Then if you want a raw ceph command, we have the toolbox keyword.

I would suggest we don't even offer a kubectl rook-ceph ceph option to run a command in the operator pod. That's just an optimization. If they want a raw ceph command, it seems most usable and clear if we stick with kubectl rook-ceph toolbox <command> and require the toolbox to be running if they want to use it. Or, if the tool sees the toolbox is not running, it could run the command in the operator pod with the required config arguments.

[sp98]

Have seen projects following the below format for the CLI:

<Top Level Command> <Command> <Resource> <Arguments> <Flags>

For example: Kubectl get pods -n rook-ceph

So can also have CLI commands like below :

kubectl rook-ceph get operator-config
kubectl rook-ceph get health
kubectl rook-ceph get version
kubectl rook-ceph get status 
kubectl rook-ceph get mon endpoints
kubectl rook-ceph get volumes [rwx|rwo|cephfs|rbd]

kubectl rook-ceph set pool-failure-domain <pool> <domain>
kubectl rook set mon-endpoints 
kubectl rook-ceph set operator-config <property> <value>
kubectl rook-ceph set mon-quorum <mon> 

kubectl rook restart operator

kubectl rook-ceph delete osd  <osd-id>