deis / workflow-e2e

End-to-end tests for Deis Workflow
MIT License
12 stars 18 forks source link

git_push_test.go - deis run spec flakiness #329

Closed vdice closed 8 years ago

vdice commented 8 years ago

CI is seeing the following two specs fail intermittently -- but consistently enough to fail ~1 out of 5 test runs.

Deis Workflow.git push deis master with an existing user who has added their public key and who has a local git repo containing buildpack source code and has run deis apps:create from within that repo and can execute deis run successfully

Deis Workflow.git push deis master with an existing user who has added their public key and who has a local git repo containing dockerfile source code and has run deis apps:create from within that repo and can execute deis run successfully

The first buildpack variant seems to be even more prevalent. Here are examples of jobs failed by either/both of these two:

https://ci.deis.io/job/workflow-test-pr/7677 https://ci.deis.io/job/workflow-test-pr/7677/artifact/7677/logs/

https://ci.deis.io/job/workflow-test/3241/ https://ci.deis.io/job/workflow-test/3241/artifact/3241/logs/

Example output:

10:41:04 ------------------------------
10:44:14 $ deis auth:register http://deis.k8s.local --username=test-481972137 --password=asdf1234 --email=test-481972137@deis.io
10:44:14 Registered test-481972137
10:44:14 Logged in as test-481972137
10:44:14 Configuration file written to /tmp/deis-workflow-home399849476/.deis/test-481972137.json
10:44:14 $ deis keys:add /tmp/deis-workflow-home399849476/.ssh/deiskey-457.pub
10:44:14 Uploading deiskey-457.pub to deis... done
10:44:14 $ deis apps:create test-231794756 
10:44:14 Creating Application... ...done, created test-231794756
10:44:14 Git remote deis successfully created for app test-231794756.
10:44:14 $ GIT_SSH=/tmp/deis-workflow-home399849476/.ssh/git-ssh GIT_KEY=/tmp/deis-workflow-home399849476/.ssh/deiskey-457 git push deis master
10:44:14 Warning: Permanently added '[deis-builder.k8s.local]:2222,[10.15.254.209]:2222' (ECDSA) to the list of known hosts.
10:44:14 remote: ---> Starting build... but first, coffee!        
10:44:14 remote: -----> Restoring cache...        
10:44:14 remote:        No cache file found. If this is the first deploy, it will be created now.        
10:44:14 remote: -----> Go app detected        
10:44:14 remote: -----> Checking Godeps/Godeps.json file.        
10:44:14 remote: -----> Installing go1.7.1... done        
10:44:14 remote: -----> Running: go install -v -tags heroku .        
10:44:14 remote:        github.com/deis/example-go        
10:44:14 remote: -----> Discovering process types        
10:44:14 remote:        Procfile declares types -> web        
10:44:14 remote: -----> Checking for changes inside the cache directory...        
10:44:14 remote:        Files inside cache folder changed, uploading new cache...        
10:44:14 remote:        Done: Uploaded cache (79M)        
10:44:14 remote: -----> Compiled slug size is 1.9M        
10:44:14 remote: ---> ---> Build complete.        
10:44:14 remote: Launching App...        
10:44:14 remote: ...        
10:44:14 remote: ...        
10:44:14 remote: ...        
10:44:14 remote: ...        
10:44:14 remote: ...        
10:44:14 remote: ...        
10:44:14 remote: ...        
10:44:14 remote: ...        
10:44:14 remote: ...        
10:44:14 remote: ---> Done, test-231794756:v2 deployed to Workflow        
10:44:14 remote:         
10:44:14 remote: ---> Use 'deis open' to view this application in your browser        
10:44:14 remote:         
10:44:14 remote: ---> To learn more, use 'deis help' or visit https://deis.com/        
10:44:14 remote:         
10:44:14 To ssh://git@deis-builder.k8s.local:2222/test-231794756.git
10:44:14  * [new branch]      master -> master
10:44:14 Waiting up to 60 seconds for `curl -sL -w "%{http_code}\\n" "http://test-231794756.k8s.local" -o /dev/null` to return 200...
10:44:14 $ curl -sL -w "%{http_code}\\n" "http://test-231794756.k8s.local" -o /dev/null
10:44:14 503
10:44:14 $ curl -sL -w "%{http_code}\\n" "http://test-231794756.k8s.local" -o /dev/null
10:44:14 503
10:44:14 $ curl -sL -w "%{http_code}\\n" "http://test-231794756.k8s.local" -o /dev/null
10:44:14 503
10:44:14 $ curl -sL -w "%{http_code}\\n" "http://test-231794756.k8s.local" -o /dev/null
10:44:14 503
10:44:14 $ curl -sL -w "%{http_code}\\n" "http://test-231794756.k8s.local" -o /dev/null
10:44:14 503
10:44:14 $ curl -sL -w "%{http_code}\\n" "http://test-231794756.k8s.local" -o /dev/null
10:44:14 503
10:44:14 $ curl -sL -w "%{http_code}\\n" "http://test-231794756.k8s.local" -o /dev/null
10:44:14 503
10:44:14 $ curl -sL -w "%{http_code}\\n" "http://test-231794756.k8s.local" -o /dev/null
10:44:14 200
10:44:14 Waiting up to 60 seconds for `curl -sL "http://test-231794756.k8s.local"` to return Powered by Deis...
10:44:14 $ curl -sL "http://test-231794756.k8s.local"
10:44:14 Powered by Deis
10:44:14 Release v2 on test-231794756-web-1059387270-mr482
10:44:14 $ deis run env -a test-231794756
10:44:14 Running 'env'...
10:44:14 $ deis apps:destroy --app=test-231794756 --confirm=test-231794756
10:44:14 Destroying test-231794756...
10:44:14 done in 0s
10:44:14 $ deis auth:cancel --username=test-481972137 --password=asdf1234 --yes
10:44:14 Please log in again in order to cancel this account
10:44:14 Logged in as test-481972137
10:44:14 Configuration file written to /tmp/deis-workflow-home399849476/.deis/test-481972137.json
10:44:14 Account cancelled
10:44:14 
10:44:14 
10:44:14 • Failure [190.064 seconds]
10:44:14 git push deis master
10:44:14 /go/src/github.com/deis/workflow-e2e/tests/git_push_test.go:284
10:44:14   with an existing user
10:44:14   /go/src/github.com/deis/workflow-e2e/tests/git_push_test.go:282
10:44:14     who has added their public key
10:44:14     /go/src/github.com/deis/workflow-e2e/tests/git_push_test.go:280
10:44:14       and who has a local git repo containing buildpack source code
10:44:14       /go/src/github.com/deis/workflow-e2e/tests/git_push_test.go:166
10:44:14         and has run `deis apps:create` from within that repo
10:44:14         /go/src/github.com/deis/workflow-e2e/tests/git_push_test.go:164
10:44:14           and can execute deis run successfully [It]
10:44:14           /go/src/github.com/deis/workflow-e2e/vendor/github.com/onsi/ginkgo/ginkgo_dsl.go:357
10:44:14 
10:44:14           Timed out after 60.014s.
10:44:14           Expected process to exit.  It did not.
10:44:14 
10:44:14           /go/src/github.com/deis/workflow-e2e/tests/git_push_test.go:161
10:44:14 ------------------------------
vdice commented 8 years ago

A timeout bump has been added to the tests as of https://github.com/deis/workflow-e2e/issues/330 (controller issue remains); closing.