Azure / draft-classic

A tool for developers to create cloud-native applications on Kubernetes.
https://draft.sh
MIT License
3.92k stars 396 forks source link

[>.12 draft releases]: Long first remote push kills the release process #644

Open squillace opened 6 years ago

squillace commented 6 years ago

On Mac, draft version: &version.Version{SemVer:"canary", GitCommit:"78385afe65500ebb4f546341f229a5f200f1128a", GitTreeState:"clean"}

The image shows you what happens again and again IF you've never pushed before. In this case, pushing to docker.io/squillace. If you immediately rerun draft up, it all succeeds fine, and subsequent rebuilds with changes will push much faster.

image

bacongobbler commented 6 years ago

This one is interesting. I haven't the foggiest idea what caused it, though it looks like it was a connection timeout error with the port-forwarded connection to tiller. I'll see if I can reproduce. :)

Before that error occurred, were you able to confirm that tiller was indeed running? It's possible that the cluster was still trying to start up.

squillace commented 6 years ago

same cluster, every time. Here it happens again, with the first time deploying to the same AKS cluster as above but from ACR. Wondering whether we're maybe trying to deploy before the image is ready to be pulled? No idea; just guessing here.

image

squillace commented 6 years ago

Remember: I'm on a home network for which the upload is VASTLY slower than most people's.... might have something to do with it. Look at the original upload times.....

squillace commented 6 years ago

Yeah: this is a thing. If the push takes too long -- and the initial push often does on a slow upload connection -- the release step fails on this result. the tunnel context has been lost, I think, but it doesn't reconnect/reconstruct the ssh context.

bacongobbler commented 6 years ago

hmm good insight! we establish a connection to tiller the moment we invoke draft up. Perhaps there's a way to keep the connection alive.

squillace commented 6 years ago

Or let me set the timeout somewhere..... don't do work FOR me, but in the small number of cases when upload is KNOWN slow, a hard to find timeout setting can be expanded...... ?

bacongobbler commented 6 years ago

labeling as a good first issue. :)