Better integration with Kubernetes

[raballew]

Feature request description

I would like to lift and shift my development workloads from a local machine to a remote Kubernetes cluster. This would allow developers to seamlessly tap into more resources that are available in the cluster for development purposes in case they need special purpose hardware such as GPUs but a the same time share these expensive resources with other users if not required.

Suggest potential solution

While Podman currently supports creating resources on Kubernetes clusters through podman kube apply it is not possible to interact with these resources in a meaningful way.

So, when I want to start a Pod on Kubernetes and get all pods I would need to run the following:

podman kube apply --kubeconfig $KUBECONFIG pod.yaml
kubectl get pod

In this case, whats the point of using the podman kube apply command in the first place if I have to use kubectl afterwards?

In theory using podman should be enough. So running the following pseudo-code should lead to the same result and would imo be more consistent.

podman kube apply --kubeconfig $KUBECONFIG deployment.yaml
podman kube ps -a --kubeconfig $KUBECONFIG

Anyhow, there are other examples such as podman kube exec --kubeconfig $KUBECONFIG -it <POD> /bin/bash which would also be useful.

Additionally I am wondering if podman should have provider implementations for instantiating a remote instance on KubeVirt?

podman machine init --kubeconfig $KUBECONFIG

Have you considered any alternatives?

No response

Additional context

No response

[giuseppe]

The value of kube apply is that you can use it directly with podman objects and deploy them to Kubernetes, that then you can handle using existing tools.

I don't think Podman should grow into a fully featured client for Kubernetes. As you've pointed out, these features are already present in kubectl.

[raballew]

There is podman kube play and podman kube down for local usage. But according to https://docs.podman.io/en/stable/markdown/podman-kube-apply.1.html podman kube apply just takes Kubernetes YAML and applies it to a cluster. This can be replaced with kubectl apply completely. This is inconsistent and confusing.

I'd suggest two options:

1. remove podman kube apply completely
2. improve usability by allowing users to interact with resources created and manage their entire life cycle through podman without needing kubectl at all. so users would use podman CLI terminology to work on k8s resources.

# creates <VOLUME>
podman kube volume create 
# creates <POD>
podman kube pod create
podman kube pod ps
podman kube run --pod <POD> --volume <VOLUME>
podman kube pod rm <POD>
podman kube volume rm <VOLUME>

Alternatively there is also value in creating a machine on Kubernetes.

podman kube machine init
podman kube machine start
podman system connection add <VM on KubeVirt>
# interact with the system
podman kube machine stop
podman kube machine rm

[rhatdan]

We can have a discussion after the break, I would definitely lean more towards removing the functionality rather then chasing the support in kubectl.

[giuseppe]

podman kube apply just takes Kubernetes YAML and applies it to a cluster. This can be replaced with kubectl apply completely

you can still use it without using -f, right? Could we do something like podman kube apply --kubeconfig /tmp/kubeconfig myvol vol-test-1 using only kubectl?

[raballew]

Wouldn't it be:

podman generate kube <RESOURCE> ...
kubectl apply -f 

[giuseppe]

Yes that would do it. Now it's a matter if we care about having just one command to do it.

@umohnani8 any other advantage in having the feature in Podman?

[umohnani8]

The idea behind kube apply is to bridge the gap between podman deployments and k8s deployments. With kube apply the user can pass it a container or pod id as well as a kube yaml to be deployed in a k8s cluster. The user can then use the web console or kubectl via the terminal to manage their workloads. Podman is not trying to replace the kube functionality or commands, our idea is to bridge the gap between the two and make it as simple as possible for users (specifically developers and newbies) to migrate workloads back and forth. Adding this command added to that convenience by simple being able to pass in a container/pod id/name to be deployed in a k8s cluster. I think we should update the docs to clarify the exact use case of this command to avoid any confusion around the idea of podman commands replacing kubectl commands.

[raballew]

[...] as simple as possible for users (specifically developers and newbies) to migrate workloads back and forth.

Adding documentation partially resolves the confusion but it does not address the inconsistency. I would challenge the "as simple as possible" statement as one would need to use kubectl for a "back and forth" migration anyhow.

[rhatdan]

We have a cabal meeting tomorrow, should we discuss this there?