Improve cycle time on Minikube by eliminating push step

[benweissmann]

I'm working on setting up a development environment for my company based on minikube, and draft seems like a great fit -- except that the cycle time from saving a file to having it running on the cluster is longer than it'd like. Of the ~15 seconds it takes to update the service, about 8 seconds is pushing the image (which is a 19MB image).

For now, i've rolled my own upgrade path that does:

eval "$(minikube docker-env)"
docker build -t my-app .

kubectl delete pod -l app=my-app --grace-period=0 --force

and have the image tag in my helm package be "latest". This is much faster, but loses all the other nice benefits of draft.

Would you be open to adding an option to use a flow like this (probably with a helm upgrade or even a kubectl set image) when minikube is detected, rather than the default push-to-registry flow? I'm happy to help with implementation, but wanted to check whether this is something you'd consider merging. Pointers to where to look to implement this change would also be much appreciated.

[bacongobbler]

Hi there!

Given Draft's current state, I'd be concerned about the user experience introducing this new workflow. How would this workflow work on different cloud providers like Azure or Google Kubernetes Engine, for example? We wouldn't want to introduce a workflow that works one way on one cloud provider that differs from the others. If we eliminate the push step, none of the kubelets (other than the one running the docker daemon we just built the image with) will be able to run the container.

That being said, we have been thinking about opening up the workflow to be scriptable using brigade as the backend for draft. That might open up new opportunities such as this. It's a much larger re-write than we may be able to tackle given the current roadmap for a 1.0 release, though.

In other words, I think this is a great use case to optimize for minikube, but we're not at the stage where we can start pulling out core functionality from Draft in certain situations yet. :)

[bacongobbler]

There is also a related discussion in https://github.com/Azure/draft/issues/277 that discusses how we could potentially improve the 8 second image push time. That might be a quicker solution than what I proposed above. If you feel like giving it a go, have at it! :)

[jstrachan]

if a cluster has only 1 node then it makes sense as an optimisation to push directly to the docker daemon of the 1 node to speed up the developer round trip. Maybe some env var or something could enable the 'optimal single node' developer flow?

[bacongobbler]

In certain instances, it’s not guaranteed that there will always be one node in the cluster. For example a lot of users trying out AKS are seeing the value in scaling down the cluster to a lower node count at night.

For minikube that might be possible, though I’d first try optimizing it via #277 first before doing cloud provider-specific optimizations.

[jstrachan]

maybe we can just provide a way to disable the push - then users can opt in (e.g. minikube users)?

avoiding the push also avoids the user having to also run a docker registry and figure out the docker registry URL and login credentials; which will help folks get started with draft + minikube too.

[bacongobbler]

Like I said, we want to optimize the workflow in other ways first before going this route. For example, we would like to compare the last build's SHA against the current and see if pushing to the registry/re-deploying the same code is necessary, or replace the "upload to draftd" step with a protocol that only ships a diff between builds. That would speed up this process immensely, and would eliminate the need to "disable" the push step in certain use cases.

Alternatively, we could look at contributing upstream to Docker to optimize layer pushes that are identical. Right now we are clocking that time at around 16 seconds to push the same image. I'm sure we can make that faster.

I'm not saying this is a bad idea, but I would rather start putting effort in these areas first so all users can enjoy the optimizations rather than just minikube users. If we're still seeing significant performance gains by disabling the registry, then with #168 we can optimize for the minikube case with a feature flag.

[bacongobbler]

FYI I have started parts of the performance improvement via #562. I'd love to hear feedback on whether that feature is important or not!

566 also tracks some discussion and proposals around sending incremental file changes to draftd, so if anyone's interested in discussing ideas there that would be lovely. :)

[bjornmagnusson]

This is also very much wanted in a DockerForMac / DockerForWindows environment

[thedrow]

Note that Minikube might not be a single node in the future. See https://github.com/kubernetes/minikube/pull/2565.

[bacongobbler]

done in #573