cloudfoundry-incubator / kubo-release

Kubernetes BOSH release
https://www.cloudfoundry.org/container-runtime/
Apache License 2.0
161 stars 76 forks source link

Add maximum timeout property to curl method #378

Closed miro-grapeup closed 4 years ago

miro-grapeup commented 4 years ago

There is no max-time property set in the curl method. Sometimes it may cause to command to run forever. Please take a look on the example below where it was hanging for ~120h.

#bosh tasks
ID     State       Started At                    Last Activity At              User   Deployment  Description                                 Result
64244  processing  Thu Feb 20 17:20:02 UTC 2020  Thu Feb 20 17:20:02 UTC 2020  admin  k8s         run errand smoke-tests from deployment k8s  -
#ps -ef 
root     31613 20789  0 Feb20 ?        00:00:00 /bin/bash /var/vcap/jobs/smoke-tests/bin/run
root     31614 31613  0 Feb20 ?        00:00:21 /var/vcap/packages/smoke-tests/run-smoke-tests -ginkgo.randomizeAllSpecs -ginkgo.failOnPending -ginkgo.v
root     31755 31614  0 Feb20 ?        00:00:47 curl --head http://10.10.54.200:30486
#after killing curl process
Task 64244 | 17:20:02 | Preparing deployment: Preparing deployment (00:00:01)
Task 64244 | 17:20:03 | Running errand: k8s-master/7bf023a3-4848-444a-8bbc-0cd7a2c8c83a (1)
Task 64244 | 17:20:03 | Running errand: k8s-master/ac38930e-af05-4cbd-9a86-43df6ade0f5d (0)
Task 64244 | 17:20:03 | Running errand: k8s-master/ed58c9c1-c6fd-4ea3-809d-d9f27089fbec (2) (00:00:16)
Task 64244 | 17:20:19 | Fetching logs for k8s-master/ed58c9c1-c6fd-4ea3-809d-d9f27089fbec (2): Finding and packing log files (00:00:08)
Task 64244 | 17:20:36 | Running errand: k8s-master/ac38930e-af05-4cbd-9a86-43df6ade0f5d (0) (00:00:33)
Task 64244 | 17:20:36 | Fetching logs for k8s-master/ac38930e-af05-4cbd-9a86-43df6ade0f5d (0): Finding and packing log files (00:00:11)
Task 64244 | 15:06:32 | Running errand: k8s-master/7bf023a3-4848-444a-8bbc-0cd7a2c8c83a (1) (117:46:29)
Task 64244 | 15:07:06 | Fetching logs for k8s-master/7bf023a3-4848-444a-8bbc-0cd7a2c8c83a (1): Finding and packing log files

And finally the output from smoketest:

CFCR Smoke Tests Deployment     
  allows access to pod logs 
  /var/vcap/data/compile/smoke-tests/src/smoke-tests/smoke_test.go:86   
kubectl run --generator=deployment/apps.v1 is DEPRECATED and will be removed in a future version. Use kubectl run --generator=run-pod/v1 or kubectl create instead. 
deployment.apps/iafegicahd created  
service/iafegicahd exposed  
Waiting for deployment "iafegicahd" rollout to finish: 0 of 1 updated replicas are available... 
deployment "iafegicahd" successfully rolled out 
iafegicahd-75fbf49c96-6n7n410.10.54.20030486deployment.extensions "iafegicahd" deleted  

• Failure [423959.836 seconds]  
CFCR Smoke Tests    
/var/vcap/data/compile/smoke-tests/src/smoke-tests/smoke_test.go:33 
  Deployment    
  /var/vcap/data/compile/smoke-tests/src/smoke-tests/smoke_test.go:45   
    allows access to pod logs [It]  
    /var/vcap/data/compile/smoke-tests/src/smoke-tests/smoke_test.go:86 

    Timed out after 423956.417s.    
    Error: Unexpected non-nil/non-zero extra argument at index 1:   
        <*exec.ExitError>: &exec.ExitError{ProcessState:(*os.ProcessState)(0xc00000e020), Stderr:[]uint8(nil)}  

    /var/vcap/data/compile/smoke-tests/src/smoke-tests/smoke_test.go:101

What this PR does / why we need it:

It prevents the smoke-test from running forever

How can this PR be verified? It is a tricky part, as I do not know what is the root cause of curl hanging forever. Looks like it waits forever for the response.

Is there any change in kubo-deployment? No

Is there any change in kubo-ci? No

Does this affect upgrade, or is there any migration required? No

Which issue(s) this PR fixes:

Release note:

Add --max-time to the smoke-test curl method.
linux-foundation-easycla[bot] commented 4 years ago

CLA Check
The committers are authorized under a signed CLA.

svrc commented 4 years ago

lgtm