Allow to change k8s imagePullPolicy

[leozilla]

Is your feature request related to a problem? Please describe. With cloudflow 2.0.22 it is not possible to configure the imagePullPolicy for images when deploying to k8s. This makes it difficult/impossible to use some options for pushing/pulling images with minikube, see https://minikube.sigs.k8s.io/docs/handbook/pushing/ Most of the mentioned options requires a imagePullPolicy which is different from Always (which is what cloudflow uses per default)

Is your feature request related to a specific runtime of cloudflow or applicable for all runtimes? Applicable to all runtimes. As its a k8s deployment limitation.

Describe the solution you'd like I dont have a good solution in mind right now. All I can think of is extending the existing k8s configuration settings by adding the imagePullPolicy as another possible setting. eg:

cloudflow.streamlets.valid-logger.kubernetes.pods.pod.containers.container.imagePullPolicy = "Never"

Describe alternatives you've considered The only two alternatives I can think of are: 1) Using a dedicated image registry with minikube. Which is more effort than the handy minikube cache I typically use. 2) Updating/patching the streamlet k8s Deployment and changing the imagePullPolicyafter it was deployed. Quite an ugly solution.

Additional context None

[andreaTP]

Hi @leozilla , thanks for this proposal! For sure implementing support for custom imagePullPolicy is something we can take into consideration in our roadmap; if you want to push some effort into this direction we can try to provide you support as well 🙂 .

Meanwhile, you do have alternatives, the first that comes to my mind is the minikube registry . Another alternative is to avoid publishing the images during the buildApp phase:

ThisBuild / cloudflowDockerRegistry := None
ThisBuild / cloudflowDockerRepository := None

and manually push the images to the minikube cache

Does one of those solutions work for you?

[leozilla]

@andreaTP I would love to help u with this feature, we could discuss possible solutions together and if u want I can also provide a draft/PoC PR for it. Just to mention, this feature is not something I need asap, I am fine if u want to append it to the back of the roadmap.

Yeah, I think the minikube registry will work but I haven't tried it yet (cos its more cumbersome than other options). But actually I think its the only solution which works.

I actually tried with the minikube cache but this requires to use a imagePullPolicy != Always, otherwise minikube will always try to pull the image from a registry/repository. btw: I used sbt build instead of sbt buildApp cos this just builds the docker image w/o publishing it.

[andreaTP]

Thanks for the feedback @leozilla ! Let me try to understand:

I used sbt build instead of sbt buildApp cos this just builds the docker image w/o publishing it.

This is not the "standard" way of doing it, in this case, you should simply add this line to your build.sbt:

ThisBuild / cloudflowDockerRegistry := None

(or avoid at all to set it).

The image is going to be created, not published and the generated CR is going to refer to the image by name (e.g. canonical name:version) instead of by sha. I already used this mechanism for deploying to a microk8s cluster successfully (using the embedded registry).

[vkorenev]

Hi @andreaTP, thank you for you answers! I'm also facing this issue while trying to deploy to local Kubernetes (the one which is shipped with Docker Desktop on Mac). Indeed, I can build container images with local docker and generate a CR with the settings that you suggested. I can see an image for the app:

$ docker images
REPOSITORY                                                                  TAG                                                     IMAGE ID       CREATED         SIZE
mw-iv-streams-streamlets                                                    develop-0.0.0-185-d74d91c0-20210125-1933                15f57d1cfd6d   2 hours ago     333MB
...

Then I deploy my app:

$ kubectl cloudflow deploy target/mw-iv-streams-app.json --no-registry-credentials

However, the pods fail to create:

$ kubectl -n mw-iv-streams-app get pods
NAME                                                READY   STATUS             RESTARTS   AGE
mw-iv-streams-app-chub-77d7d8bc75-llz7c             0/1     ErrImagePull       0          45s
mw-iv-streams-app-dc-7455964499-frkhm               0/1     ImagePullBackOff   0          53s

It looks like Cloudflow operator is still trying to pull the image from a remote repo, although it is available locally.

[leozilla]

@vkorenev Yes this is exactly the problem we are taking about. The docker client inside minikube cannot pull the image. (minikube is also used for the local Kubernetes in Docker Desktop on Mac)

@vkorenev @andreaTP The only working solution I found so far is using the minikube registry. But as u can see this is a bit cumbersome to use. I also think that I configured something incorrect cos I cannot pull the image inside minikube via docker pull MINIKUBE_IP:5000/my-image:0.1. Which I think should actually work. But anyway I found this way that works for me on Ubuntu Linux. I think its almost the same on Mac.

U can find the official guide here: https://minikube.sigs.k8s.io/docs/handbook/pushing/#4-pushing-to-an-in-cluster-using-registry-addon

Enable insecure registries in docker daemon. This is probably the only thing which might be different between Linux and Mac. Add the following json config to the /etc/docker/daemon.json file, or create a new file with this content. 192.168.39.239 is the minikube ip which u can get via minikube ip.

{
  "insecure-registries": [ "192.168.39.239:5000" ]
}

edit: Dont forget to restart the docker daemon, on linux sudo service docker restart

Start minikube with insecure registries enabled and enable the registry addon.

minikube start --kubernetes-version=latest --insecure-registry "10.0.0.0/24,192.168.39.0/24"
minikube addons enable registry

In your build.sbt use the following config. Note that I use localhost:5000 here. The reason for this is that minikube couldn't otherwise pull the images.

ThisBuild / cloudflowDockerRegistry := None

lazy val my-app =  (project in file("."))
  .settings(
    cloudflowDockerImageName := Some(DockerImageName("localhost:5000/my-image", "0.1")), 
  )

Build the app.

sbt buildApp

Prepare and push docker images.

docker tag localhost:5000/my-image:0.1 $(minikube ip):5000/my-image:0.1
docker push $(minikube ip):5000/my-image:0.1

Deploy the app to minikube.

kubectl config set-context --current --namespace="" # this is necessary to work around a current bug in 2.0.22
kubectl cloudflow deploy target/my-app.json --no-registry-credentials

This way minikube can pull the images because we use localhost:5000 as the registry address.

Hope that helps.

[andreaTP]

Thanks a lot for wrapping it up all together @leozilla ! Yes that's more or less how I got it working on microk8s as well. We might want to improve the experience in the future, but, as of today, this is it, if you want/can add a section with those steps to the docs it will be great 🙂 .

A quick note on:

kubectl config set-context --current --namespace="" # this is necessary to work around a current bug in 2.0.22

This is fixed and new binaries are published, just re-download the CLI.

[leozilla]

@andreaTP yeah sure I can add something to the docs