use case: detect & install iscsi

[AmitKumarDas]

ref - https://gist.github.com/jjo/1ccade2ea3ae05c5825fd37c8722d4f6 ref - https://github.com/go-cmd/cmd

Draft - 0

kind: Command
apiVersion: dao.mayadata.io/v1alpha1
metadata:
  name: install-cephfs
  annotations:
    cmd: &cmd apt-get update -qy && apt-get install -qy tgt ceph-fs-common ceph-common xfsprogs
spec:
  hostPID: true
  securityContext:
    privileged: true
  command:
    - nsenter
    - --mount=/proc/1/ns/mnt
    - --
    - sh
    - -c
    - *cmd
status:
  exitCode:
  reason:

[AmitKumarDas]

Thoughts 1

• An operator that listens to kind Command
• This operator spins up a K8s DaemonJob or Job with characteristics described in Command specs
  • DaemonJob is formed if nodeSelector is set else Job gets formed
• This image understands kind Command & executes the Command specs
• This image publishes the results as kind CommandResult
• Operator aggregates the CommandResult(s) & publishes the success, failure, counts, etc at Command's status field

Draft 1

kind: Command
metadata:
spec:
  hostPID:
  securityContext:
  nodeSelector:
  # image is the binary on which this command will be run
  # this image will be based on multi-stage Docker build consisting of:
  # 1/ any docker slim image as a base image
  # 2/ iscsi-util binary (in this case) or other 3rd party binaries based on requirement
  # 3/ a golang binary (that uses os.exec to invoke commands over std.in)
  # NOTE: golang binary will be developed by us
  image:
  # represents the order to execute commands
  order:
  - cmd-0
  - cmd-1 
  # commands is a map[string][]string
  # one can execute one or more commands in a particular order or in a random order
  # commands are run via golang os.exec programatically
  commands:
    cmd-0:
    cmd-1:
status:
  phase:
  runNodeCount:
  passNodeCount:
  failNodeCount:
  # errors due to execution of each command will be displayed here
  errors:
    cmd-0:
    cmd-1:
  # we avoid populating the success outputs in this custom resource which also
  # is acting as an aggregator of commands that might have run against
  # 100s of nodes
  #
  # NOTE: successful outputs can be found at CommandResult custom resource

# Provides the result of a command that is run on a particular host
# In other words, if the Command CR is run against more than 1 nodes
# e.g. n nodes, then there will be n CommandResult instances
kind: CommandResult
spec:
  commandRef:
    uid:
    name:
    namespace:
status:
  phase:
  hostName:
  # actual commands
  commands:
    cmd-0:
    cmd-1:
  # errors if any w.r.t command
  errors:
    cmd-0:
      exitCode:
      reason:
    cmd-1:
      exitCode:
      reason:
  # success output if any w.r.t command
  outputs:
    cmd-0:
    cmd-1:

[AmitKumarDas]

UseCases

• Find a way to detect presence of iscsi utils in k8s worker nodes
• Ability to install or upgrade iscsi utils in k8s worker nodes
• Ability to execute 3rd party cli tools
  • e.g. ability to execute https://github.com/sumitnagal/test-tools/pull/1

[vharsh]

A long time ago, I once looked at a PR in Ansible, which was being used to figure out the hostname, well it turns out not everyone follows the sane POSIX standards(take package management for example, it's a mess, yum, apt, dnf, if I exclude people from the ArchLinux and Gentoo world). Some organizations might even end up running their own software repositories thankfully some of them keep them in sync, some don't(I hope people don't like to rename packages to make it harder for us).

Besides, considering the number of packages out in the wild would we prefer to redo the work done by awesome tools like brew on MacOS, LinuxBrew on Linux, etc? Is building stuff from source by pulling it from somewhere the only ONE way to do it?

[AmitKumarDas]

@vharsh We would like to detect/monitor, & eventuall install/uninstall/upgrade host packages. This issue talks about iscsi-utils for example. Having said that, we would like to go for GitOps way for this requirement.

We wont be building source rather we want to put binaries (optional) in place to execute its commands. These commands are in turn specified in K8s CR i.e. Command. However to install iscsi, we don't have to get into above image building process. We can just use our go binary that will run binaries present on the host by running this go image inside a Pod with privileged status & so on.

Now think for example, running a custom python binary via this approach. In order to do so, we would have to get the python binary along with our base docker image & our own go based executor. Our go executor reads the Command CR that specifies the commands native to this python binary. OTH this entire stuff boils down to an image that needs to be provided to this Command CR. This Command CR would in-turn create a Job/Pod that makes use of:

• 1/ this image &
• 2/ the Command CR as a config.

[AmitKumarDas]

@vharsh to be more specific, we are not re-inventing anything here. We are packaging in a way that is native to K8s & is more GitOps friendly. So a Mac based k8s worker node can use brew xyz in the Command CR.

[vharsh]

Okay, from Director perspective a user might want to happen automatically(this is a dangerous word and IMO an end user in prod wouldn't trust any pod to take permissions to run stuff on Node by reading thing off a cfgmap, a rouge user like me might edit the cfgmap, add sudo shutdown now and kill the executor pod(conditioned to my permissions in the cluster) and then the stateful workloads will go for a toss(if shutdown doesn't have a graceful wait period).

[AmitKumarDas]

Yes. Agree. However this needs to be channelized via authentication and authorisation policies. I believe starting off with kubernetes namespaced scope and rbac should be a good start.

[AmitKumarDas]

On a different note, we can use this for chaos tests. 🤓

[AmitKumarDas]

Thinking further I guess we will expose the final product i.e. the docker image that does just one thing e.g. detect iscsi and install the same. Obviously there will be lots of docker images one for each distro.

The mechanics of the design will remain flexible though to build any kind of product/image.

[AmitKumarDas]

This final image depending on its criticality can be made to run only from kube-system namespace if this makes things secure. In other words an image only for cluster admins and no one else

[AmitKumarDas]

Some references:

• https://github.com/piontec/netperf-operator
• https://github.com/portworx/fio-tools
• https://github.com/pingcap/chaos-mesh/tree/master/controllers/iochaos
• https://github.com/rancher/system-upgrade-controller

[AmitKumarDas]

Draft specs:

kind: Command
spec:
  enabled:
    when: Once # Never, Always
  template:
    job:  # either job or daemonJob need to be set
    daemonJob: # either job or daemonJob need to be set
  env:
  commands:
  - name: # mandatory
    desc: # optional
    cmd: # single line command that is run as go exec []string args
    sh: # multi line bash script
status:
  phase:
  outputs:
    <commandname>:

How it works?

• Controller watches Command resource
• Controller create a Job (or DaemonJob) based on spec.template
• This Job's annotation will have the details of the Command resource
  • e.g. name, namespace, apiversion & kind of Command resource will be set as annotation
  • in the Job (or DaemonJob)
• This Job (or DaemonJob) makes use of a binary that picks the Command details from this annotations
• This binary runs the commands & remembers its outputs
• This binary updates the Command resource with the outputs
• A Command can be run only once or multiple times
• A Command should be run by a single controller instance at any point in time
• Once the Command status is updated, the completed Job (or DaemonJob) is deleted
• A Command can be re-run if it is meant to be run multiple times

[AmitKumarDas]

First commit that implements Command custom resource partially.