Test Flake: cluster up fails

spadgett commented 7 years ago

https://ci.openshift.redhat.com/jenkins/job/test_pull_request_origin_web_console/42/console

-- Login to server ... FAIL
   Error: error: The server was unable to respond - verify you have provided the correct host and port and that the server is currently running.
++ export status=FAILURE
++ status=FAILURE
+ set +o xtrace
########## FINISHED STAGE: FAILURE: START OPENSHIFT SERVER [00h 02m 59s] ##########
Build step 'Execute shell' marked build as failure

spadgett commented 7 years ago

https://ci.openshift.redhat.com/jenkins/job/test_pull_request_origin_web_console/48/console

+ oc cluster up --version=latest --public-hostname=localhost
Starting OpenShift using openshift/origin:latest ...
-- Checking OpenShift client ... OK
-- Checking Docker client ... OK
-- Checking Docker version ... OK
-- Checking for existing OpenShift container ... OK
-- Checking for openshift/origin:latest image ... OK
-- Checking Docker daemon configuration ... OK
-- Checking for available ports ... OK
-- Checking type of volume mount ... 
   Using nsenter mounter for OpenShift volumes
-- Creating host directories ... OK
-- Finding server IP ... 
   Using 127.0.0.1 as the server IP
-- Starting OpenShift container ... 
   Creating initial OpenShift configuration
   Starting OpenShift using container 'origin'
   Waiting for API server to start listening
   OpenShift server started
-- Adding default OAuthClient redirect URIs ... OK
-- Installing registry ... OK
-- Installing router ... OK
-- Importing image streams ... OK
-- Importing templates ... OK
-- Installing template service broker ... FAIL
   Error: failed to start the template service broker apiserver: the server cannot complete the requested operation at this time, try again later (get daemonsets.extensions apiserver)
++ export status=FAILURE
++ status=FAILURE

spadgett commented 7 years ago

https://ci.openshift.redhat.com/jenkins/job/test_pull_request_origin_web_console/59/ https://ci.openshift.redhat.com/jenkins/job/test_pull_request_origin_web_console/61/ https://ci.openshift.redhat.com/jenkins/job/test_pull_request_origin_web_console/62/

spadgett commented 7 years ago

cc @deads2k @csrwng @bparees @stevekuznetsov

spadgett commented 7 years ago

This fails most of the time starting the template broker now.

deads2k commented 7 years ago

This fails most of the time starting the template broker now.

Can you crank up the log level on oc cluster up so we can get a better idea of what fails? Also, the container logs would be really helpful.

stevekuznetsov commented 7 years ago

We need a higher log level, or we need y'all to run on openshift-ansible so our logs are useful. Unclear the best way to get Origin Bash things (like container logs & cleanup) to show up in this repo...

deads2k commented 7 years ago

I looked into this in more detail. I really need master logs, container logs, or pod resources to debug this. I tried reproducing locally and didn't see the same problem.

stevekuznetsov commented 7 years ago

OK, container logs should be at artifacts/scripts/bash/logs/containers now -- pushed up https://github.com/openshift/aos-cd-jobs/commit/3ca303e0b047dbdc780a3231ddb6f211c90af5fe

spadgett commented 7 years ago

I opened a pull that also increases the log level

https://github.com/openshift/aos-cd-jobs/pull/574

spadgett commented 7 years ago

Lowering the severity since we've stop seeing this

stevekuznetsov commented 7 years ago

@spadgett did we get logs showing up as expected?

spadgett commented 7 years ago

@stevekuznetsov I'm not seeing them unless I'm looking in the wrong place. Here's a recent run

https://ci.openshift.redhat.com/jenkins/job/test_pull_request_origin_web_console/76/

stevekuznetsov commented 7 years ago

We downloaded the artifact directory before we generated the logs... whoops. I'll fix

spadgett commented 7 years ago

I'm no longer seeing this.

openshift / origin-web-console

Test Flake: cluster up fails #1937