Support K8s resources

[windsource]

Description

For the acceptance of Ankaios some compatibility to Kubernetes is important. For that reason the CLI shall be extended to also support (a subset of the) Kubernetes pod spec.

Goals

• The CLI allows to apply a pod spec
• The CLI allows to delete a pod spec which results in stopping the workloads
• The CLI allows to list the pod specs that have been applied
• Podman runtime support is sufficient

Final result

Summary

In the course of work we decided to not only change the CLI, but directly add a new runtime - "podman-kube". The new runtime will instrument Podman to handle K8s manifests via the podman play kube command.

Special CLI commands will not be needed as a simple ank run --runtime podman-kube --agent X --config <k8s manifest content> can be used.

Tasks

• [x] Define how and which K8s resources are supported
• [x] Move runtime management related logic from AgentManager to the new RuntiumeManager
• [x] Move initial workload start with reusing Workloads from The PodmanAdapter to RuntiumeManager
• [x] Create new PodmanKubeRuntime implementing the business logic of handling the K8s manifests
• [x] Replace the PodmanAdapter with a a new PodmanRuntime implementing the business logic of handling containers with Podman. Use the CLI here (see comment below for details).
• [x] Replace podman_api crate with calls to the Podman CLI (target: reduce dependencies and unify the system to use one way of communicating with Podman; podman_api does not properly support play kube) - #54
• [x] Update SW design and requirement tracing
• [x] Add requirements for supported Podman features
• [x] Add requirements for supported podman play kube features (there are different options for the command; in case we don't support all, we need to specify which)
• [x] Fix the podman-kube system test
• [x] make sure podman-kube requirements are complete
• [x] document supported Podman versions - #97

[krucod3]

As agreed in the team, we need to check what parts of the pod specification we would like to support (and what is supported by Podman). We also need to check if we would like to also support deployments, configMaps and persistent volume claims.

[krucod3]

Podman supports Kubernetes manifests via the podman play kube command. Currently the following K8s resources are supported (according to the documentation liked above):

• Pod: A Podman pod is created which can include multiple containers. Init containers are also supported.
• Deployment: Unfortunately the replica count does not work as expected. It is always overwritten to 1 which kind of degrades the deployment to a pod resources. It is still helpful in case you have a deployment file.
• PersistentVolumeClaim: Volumes in general works for Pods and Deployments. If you use a PVC, a Podman named volume is created. By default the volumes created with the PVC is not delated when tearing down the Pod. This can still be forced with the --force option.
• Secret: Creates a Podman named secret which is persisted and can also be used by other Pods. The secret is deleted when tearing down the Pod
• ConfigMap The ConfigMap is not persisted in any way by Podman. The data inside is used to fill in variables on the fly while creating the pods in the manifest, but it is gone once the creation process is over. Unlike Secret it does not remain in the system.

Although deployments do not bring a lot of benefit, we should support them to ease up the transition from K8s. Regarding multiple resources in a manifest, it makes sense to allow ConfigMaps, PVCs and Secrets to be added beside Pod and Deployment. Still there are a couple of questions here:

• What to do if multiple resources creating pods are specified? For Ankaios one workload spec has one name, e.g. workloads.nginx. If we have now 2 pods managed by one workload things start getting weird. What is the state of this pods? Should I restart both if one fails? I think we should forbid more then two pod creating blocks in the config.
• What to do if only a PVC or only a Secret is specified? This could come in handy in certain situations, but we don't have a real workload behind it ... The name is then kind of misleading. Still, it would be very helpful, so I think we should allow it.
• How do we pass podman play kube specific options? We could have them in the runtime config, e.g.:

  runtimeConfig: |
  force-remove: true
  userns: host
  manifest: |
    apiVersion: v1
    kind: Secret
    metadata:
      name: foo
    data:
      foo: YmFy # base64 for bar

[windsource]

Besides workloads Ankaios knows also other resource types like configs and cronJobs which just have not been implemented yet. Maybe we could use another resource type besides workload if a PVC or a secret is created? But that would mean that existing YAML with several resources like pod, PVC and secret need to be split up when passing to Ankaios which is also not optimal.

[krucod3]

Yes, splitting would not be optimal. Another issue I see is that configs should be runtime agnostic. If we start making runtime specific config items, the complexity will rise very quickly and adding new runtimes could get very hard.

[krucod3]

As for cron jobs, they are currently though of as a reference to a workload with an interval specifying the execution frequency.

[krucod3]

Podman's documentation does not specify that the delete method for play/kube accepts a yaml/json file that specifies the resources that shall be deleted (I'll create a ticket for that). Because of the issue in the podman documentation, the podman-api crate that we use does not accept arguments specifying the yaml file or the options when tearing down the kube manifest. I'll create a ticket for that too. In my opinion the method also has a misleading name as it also deals with other typed of K8s resources, e.g. Secrets. Maybe the play_kubernetes_yaml method can be used for both creation and deletion (--down) as in the CLI and the REST API, although I personally don't like that interface of the Podman CLI either. The other option would be a delete_kubernetes_yaml method.

As for our implementation, I would suggest to directly send http calls to the Podman socket. It is not high effort for play kube and we don't have to wait for the changes suggested above. We can also call the Podman CLI.

[krucod3]

Podman issue: https://github.com/containers/podman/issues/19945 Podman-api issue: https://github.com/vv9k/podman-api-rs/issues/168

[krucod3]

As the podman-api rust crate does not support what we need, we have to go for the REST API or the CLI of Podman. We had a short discussion with @windsource and came to the conclusion to go for the CLI. It seems like play kube is either ways not really trimmed for speed for now see https://github.com/containers/podman/issues/19716, so the latency of loading Podman in memory should not play a major role in this case.

[krucod3]

If we are already switching to usage of the Podman CLI instead of the Rest API, we could think about changing the config for the Porman runtime to accept strings for create container and start container and pass them 1-to-1 to the Podman CLI. This way we support all features of Podman and don't need special documentation of our Podman runtime config.

[krucod3]

Most of the changes for the structure needed to support your second runtime are now done. All functionality should already be there including creating the control interface for each workload and passing it to the runtimes. Continuing with the generic state checker.

[maturar]

We were discussing how to implement new version of the "podman runtime" (for the containers). We came to the question whether to use podman run or the combination podman create + podman start. If I compare podman create and podman run, I can see following differences in command options:

podman create has additionally following arguments (comparing to the podman run):

      --init-ctr string                          Make this a pod init container.

podman run has additionally following arguments (comparing to the podman create):

  -d, --detach                                   Run container in background and print container ID
      --detach-keys [a-Z]                        Override the key sequence for detaching a container. Format is a single character [a-Z] or a comma separated sequence of `ctrl-<value>`, where `<value>` is one of: `a-cf`, `@`, `^`, `[`, `\`, `]`, `^` or `_` (default "ctrl-p,ctrl-q")
      --passwd                                   add entries to /etc/passwd and /etc/group (default true)
      --preserve-fds uint                        Pass a number of additional file descriptors into the container
      --rmi                                      Remove container image unless used by other containers
      --sig-proxy                                Proxy received signals to the process (default true)

Now the question is what we want to use in the podman_runtime.rs.

NOTE: the new implementation is also qood chance to change/improve configuration in the startConfig.yaml. We have an idea to pass the podman runtime configuration transparently (as much as possible) to the podman. Current approach is not transparent at all. If we want Ankaios to support new podman runtime configuration item, we have to extend the Ankaios. If we make the runtime configuration transparent (from the Ankaios point of view), we will make the situation simpler and more flexible.

[maturar]

One more note: podman create and podman run have the logic to pull the image in case the image is not available. It means we can call podman create or podman run directly. We do not need to pull images manually.

[krucod3]

The new structure is working fine, although there are some places that need polishing. The requirements tracing should also be almost fine now. What remains is to update the podman runtime requirements and to update the diagrams with the new structure and workflow.

[maturar]

I am nearly finished with the implementation of "create workload" in the podman_runtime.rs. It uses podman run instead of podman create + podman start. I have discussed it with @krucod3 and we do not see any necessity to call create + start. The latest commit ba9a33f2d21e7e1b97a5ae153bcf359b61cf49aa also contains new format of the PodmanRuntimeConfig. We have reviewed it (quickly) and it seems to be okay. The new runtime config is called PodmanRuntimeConfigCli. The old config remains as it is for now. We will remove the old config later on (as soon as we finish the implementation here).

[krucod3]

I'm currently working on the unit tests for the RuntimeFacade, but have trouble using mackall. It seems we will need to write our own mock runtime in order to porerly unit test the Runtimefacade and the Workload object.

[krucod3]

RuntimeFacade utests done, started with the workload object now.

[christoph-hamm]

Last task is completed.