search

teamhephy / controller

Hephy Workflow Controller (API)
https://teamhephy.com
MIT License
14 stars 26 forks source link

deis run command taking over 60 seconds to complete #49

Open Cryptophobia opened 6 years ago

Cryptophobia commented 6 years ago

From @vdice on October 5, 2016 21:37

CI is seeing failures when running deis run env -a <appName> due to said command not returning within the default timeout of 60 seconds. (Original issue/background info found in https://github.com/deis/workflow-e2e/issues/329)

As it is reasonable to expect such a command would return in a more reasonable amount of time, I've filed the issue here. cc @kmala

If issue is addressed, a TODO on the potential fix would be to decrease e2e's bumped threshold for any test involving deis run.

Copied from original issue: deis/controller#1106

Cryptophobia commented 6 years ago

From @helgi on October 5, 2016 21:46

That seems like a problem with the e2e setup and not Controller - deis run has a 20-minute timeout loop itself on top of waiting for the component to come up in Kubernetes.

I'm suspecting any run that has deis run failures also has additional failures. Kubernetes could be too slow for the e2e 60 second wait period.

Cryptophobia commented 6 years ago

From @bacongobbler on October 5, 2016 22:2

I think the ask is "is 60 seconds too long for deis run ls?" in which case, yes it can be. The problem is that the image needs to be fetched onto the worker node before the command is run. If there was a way we could pre-warm the worker nodes with the image, that may help speed things up.

I think it's both e2e's fault (the controller has a 20 minute timeout for deis run operations, so maybe the e2e run should do the same) and it's also the controller's implementation of deis run. If we can plug into kubectl exec instead of dropping a pod every time we call deis run, that might help make this situation more ideal.

Cryptophobia commented 6 years ago

From @helgi on October 5, 2016 22:28

@bacongobbler kubectl into an existing pod? I think the argument against that would be that you use those pods to serve traffic or do alternative things where as run may be a bit destructive / intense operations. Not sure tho.

Cryptophobia commented 6 years ago

From @bacongobbler on October 5, 2016 23:1

That is true. There can be an argument made for and against using kubectl exec for deis run.

Cryptophobia commented 6 years ago

From @helgi on October 5, 2016 23:9

Yeah - Agreed. We could make both work... exec mode picks a random pod of the a type (a particular proc type needs to be picked) to run