Expose controller errors in vertx app(Vert.x HTTP Booster)

[jaseemabid]

Expose controller is something we believed we weren't even using in production, but I'm noticing errors related to ec in the deployment step. Relevant part of the logs are

Running: echo | exposecontroller --watch-namespace jabid-stage --exposer Route
command: echo | exposecontroller --watch-namespace jabid-stage --exposer Route out: Please enter Username: Please enter Password:
command: echo | exposecontroller --watch-namespace jabid-stage --exposer Route err: I0301 14:10:02.276861     470 exposecontroller.go:56] Using build: '2.3.27'
I0301 14:10:02.301360     470 config.go:20] No /etc/exposecontroller/config.yml file found.  Will try to figure out defaults
I0301 14:10:02.301413     470 exposecontroller.go:125] Running in : `jabid-stage`
W0301 14:10:02.358563     470 controller.go:478] Failed to load url https://openshift.default.svc/.well-known/oauth-authorization-server got: Get https://openshift.default.svc/.well-known/oauth-authorization-server: x509: certificate signed by unknown authority
E0301 14:10:02.359929     470 controller.go:186] Failed to list nodes to detect minishift: the server does not allow access to the requested resource (get nodes)
I0301 14:10:02.359951     470 controller.go:118] Using OAuth Authorize URL:
W0301 14:10:02.359959     470 controller.go:120] Please use $OAUTH_AUTHORIZE_URL to define the OAuth Authorize URL!
W0301 14:10:02.361334     470 controller.go:251] Could not find kubernetes service in the default namespace so we could not detect whether to use http or https as the apiserver protocol. Error: the server does not allow access to the requested resource (get services kubernetes)
I0301 14:10:02.361379     470 controller.go:534] starting expose controller
I0301 14:10:07.361572     470 controller.go:542] stopping expose controller

1. I have to admit that I don't understand what exposecontroller do, but I was under the impression that its not part of osio anymore. Now we have builds failing because of it.
2. The build reports success, even though it did not.
3. The app never got deployed to http://vertxapp-jabid-stage.8a09.starter-us-east-2.openshiftapps.com

cc: @maxandersen.

@kbsingh I did this immediately after a tenant reset, so this probably affects every user on osio.

Complete logs:

OpenShift Build jabid/vertxapp-3 from https://github.com/jaseemabid/vertxapp.git
Connecting to https://api.github.com using jaseemabid/****** (cd-github)
Obtained Jenkinsfile from a836a698b646f7869906d572af76de7771ae43f7
Loading library github.com/fabric8io/fabric8-pipeline-library@master
Attempting to resolve master from remote references...
 > git --version # timeout=10
 > git ls-remote -h -t https://github.com/fabric8io/fabric8-pipeline-library.git # timeout=10
Found match: refs/heads/master revision ed80ca334cd25477b1045a329e5c21b6b7eade12
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/fabric8io/fabric8-pipeline-library.git # timeout=10
Fetching without tags
Fetching upstream changes from https://github.com/fabric8io/fabric8-pipeline-library.git
 > git --version # timeout=10
 > git fetch --no-tags --progress https://github.com/fabric8io/fabric8-pipeline-library.git +refs/heads/*:refs/remotes/origin/*
Checking out Revision ed80ca334cd25477b1045a329e5c21b6b7eade12 (master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f ed80ca334cd25477b1045a329e5c21b6b7eade12
Commit message: "Merge pull request #382 from rupalibehera/token-env-planner-team"
 > git rev-list ed80ca334cd25477b1045a329e5c21b6b7eade12 # timeout=10
[Pipeline] echo
Kubernetes Plugin Version 013
[Pipeline] echo
Loaded PipelineConfiguration PipelineConfiguration{jobNameToKindMap={}, ciBranchPatterns=[PR-.*], cdBranchPatterns=[master], cdGitHostAndOrganisationToBranchPatterns={}, disableITestsCD=false, disableITestsCI=false, useDockerSocketFlag=false}
[Pipeline] echo
Loaded the useDockerSocket flag false
[Pipeline] podTemplate
[Pipeline] {
[Pipeline] node
Still waiting to schedule task
jenkins-slave-qtgsq-754wb is offline
Running on jenkins-slave-qtgsq-754wb in /home/jenkins/workspace/jaseemabid_vertxapp_master-H32ZJFBTIYTHVJ3ER6OMNP2WKR2G54WF6CUM3E3DAZZLYSGOCDWQ
[Pipeline] {
[Pipeline] checkout
Cloning the remote Git repository
Cloning with configured refspecs honoured and without tags
Cloning repository https://github.com/jaseemabid/vertxapp.git
 > git init /home/jenkins/workspace/jaseemabid_vertxapp_master-H32ZJFBTIYTHVJ3ER6OMNP2WKR2G54WF6CUM3E3DAZZLYSGOCDWQ # timeout=10
Fetching upstream changes from https://github.com/jaseemabid/vertxapp.git
 > git --version # timeout=10
using GIT_ASKPASS to set credentials cd-github
 > git fetch --no-tags --progress https://github.com/jaseemabid/vertxapp.git +refs/heads/master:refs/remotes/origin/master
 > git config remote.origin.url https://github.com/jaseemabid/vertxapp.git # timeout=10
 > git config --add remote.origin.fetch +refs/heads/master:refs/remotes/origin/master # timeout=10
 > git config remote.origin.url https://github.com/jaseemabid/vertxapp.git # timeout=10
Fetching without tags
Fetching upstream changes from https://github.com/jaseemabid/vertxapp.git
using GIT_ASKPASS to set credentials cd-github
 > git fetch --no-tags --progress https://github.com/jaseemabid/vertxapp.git +refs/heads/master:refs/remotes/origin/master
Checking out Revision a836a698b646f7869906d572af76de7771ae43f7 (master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f a836a698b646f7869906d572af76de7771ae43f7
Commit message: "Initial import"
 > git rev-list a836a698b646f7869906d572af76de7771ae43f7 # timeout=10
[Pipeline] echo
Using branch master
[Pipeline] echo
Using branch master
[Pipeline] echo
NOTE: running pipelines for the first time will take longer as build and base docker images are pulled onto the node
[Pipeline] container
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Build Release)
[Pipeline] sh
[jaseemabid_vertxapp_master-H32ZJFBTIYTHVJ3ER6OMNP2WKR2G54WF6CUM3E3DAZZLYSGOCDWQ] Running shell script
+ git checkout -b jaseemabid/vertxapp/master-1.0.3
Switched to a new branch 'jaseemabid/vertxapp/master-1.0.3'
[Pipeline] sh
[jaseemabid_vertxapp_master-H32ZJFBTIYTHVJ3ER6OMNP2WKR2G54WF6CUM3E3DAZZLYSGOCDWQ] Running shell script
+ mvn org.codehaus.mojo:versions-maven-plugin:2.2:set -U -DnewVersion=1.0.3
Picked up _JAVA_OPTIONS: -Duser.home=/root/ -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -Dsun.zip.disableMemoryMapping=true -XX:+UseParallelGC -XX:MinHeapFreeRatio=5 -XX:MaxHeapFreeRatio=10 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -Xms10m -Xmx192m
[INFO] Scanning for projects...
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building Vert.x - HTTP 1.0.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- versions-maven-plugin:2.2:set (default-cli) @ vertxapp ---
[INFO] Searching for local aggregator root...
[INFO] Local aggregation root: /home/jenkins/workspace/jaseemabid_vertxapp_master-H32ZJFBTIYTHVJ3ER6OMNP2WKR2G54WF6CUM3E3DAZZLYSGOCDWQ
[INFO] Processing change of io.openshift.booster:vertxapp:1.0.0-SNAPSHOT -> 1.0.3
[INFO] Processing io.openshift.booster:vertxapp
[INFO]     Updating project io.openshift.booster:vertxapp
[INFO]         from version 1.0.0-SNAPSHOT to 1.0.3
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 4.778 s
[INFO] Finished at: 2018-03-01T14:08:09+00:00
[INFO] Final Memory: 7M/10M
[INFO] ------------------------------------------------------------------------
[Pipeline] sh
[jaseemabid_vertxapp_master-H32ZJFBTIYTHVJ3ER6OMNP2WKR2G54WF6CUM3E3DAZZLYSGOCDWQ] Running shell script
+ mvn clean -B -e -U deploy -Dmaven.test.skip=false -P openshift
Picked up _JAVA_OPTIONS: -Duser.home=/root/ -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -Dsun.zip.disableMemoryMapping=true -XX:+UseParallelGC -XX:MinHeapFreeRatio=5 -XX:MaxHeapFreeRatio=10 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -Xms10m -Xmx192m
[INFO] Error stacktraces are turned on.
[INFO] Scanning for projects...
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building Vert.x - HTTP 1.0.3
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-clean-plugin:3.0.0:clean (default-clean) @ vertxapp ---
[INFO]
[INFO] --- maven-enforcer-plugin:3.0.0-M1:enforce (enforce-java-version) @ vertxapp ---
[INFO]
[INFO] --- maven-enforcer-plugin:3.0.0-M1:enforce (enforce-maven-version) @ vertxapp ---
[INFO]
[INFO] --- buildnumber-maven-plugin:1.4:create (get-scm-revision) @ vertxapp ---
[INFO] Executing: /bin/sh -c cd '/home/jenkins/workspace/jaseemabid_vertxapp_master-H32ZJFBTIYTHVJ3ER6OMNP2WKR2G54WF6CUM3E3DAZZLYSGOCDWQ' && 'git' 'rev-parse' '--verify' 'HEAD'
[INFO] Working directory: /home/jenkins/workspace/jaseemabid_vertxapp_master-H32ZJFBTIYTHVJ3ER6OMNP2WKR2G54WF6CUM3E3DAZZLYSGOCDWQ
[INFO] Storing buildNumber: a836a698b646f7869906d572af76de7771ae43f7 at timestamp: 1519913302108
[INFO] Storing buildScmBranch: jaseemabid/vertxapp/master-1.0.3
[INFO]
[INFO] --- vertx-maven-plugin:1.0.13:initialize (vmp) @ vertxapp ---
[INFO]
[INFO] --- maven-resources-plugin:3.0.2:resources (default-resources) @ vertxapp ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 1 resource
[INFO]
[INFO] --- fabric8-maven-plugin:3.5.28:resource (fmp) @ vertxapp ---
[INFO] F8: Running in OpenShift mode
[INFO] F8: Using docker image name of namespace: jabid-jenkins
[INFO] F8: Running generator vertx
[INFO] F8: vertx: Using Docker image registry.access.redhat.com/redhat-openjdk-18/openjdk18-openshift as base / builder
[INFO] F8: using resource templates from /home/jenkins/workspace/jaseemabid_vertxapp_master-H32ZJFBTIYTHVJ3ER6OMNP2WKR2G54WF6CUM3E3DAZZLYSGOCDWQ/src/main/fabric8
[INFO] F8: fmp-controller: Adding a default Deployment
[INFO] F8: fmp-service: Adding a default service 'vertxapp' with ports [8080]
[INFO] F8: vertx-health-check: Adding readiness probe on port 8080, path='/', scheme='HTTP', with initial delay 10 seconds
[INFO] F8: vertx-health-check: Adding liveness probe on port 8080, path='/', scheme='HTTP', with initial delay 180 seconds
[INFO] F8: fmp-revision-history: Adding revision history limit to 2
[INFO] F8: f8-icon: Adding icon for deployment
[INFO] F8: f8-icon: Adding icon for service
[INFO] F8: validating /home/jenkins/workspace/jaseemabid_vertxapp_master-H32ZJFBTIYTHVJ3ER6OMNP2WKR2G54WF6CUM3E3DAZZLYSGOCDWQ/target/classes/META-INF/fabric8/openshift/vertxapp-svc.yml resource
[INFO] F8: validating /home/jenkins/workspace/jaseemabid_vertxapp_master-H32ZJFBTIYTHVJ3ER6OMNP2WKR2G54WF6CUM3E3DAZZLYSGOCDWQ/target/classes/META-INF/fabric8/openshift/vertxapp-deploymentconfig.yml resource
[INFO] F8: validating /home/jenkins/workspace/jaseemabid_vertxapp_master-H32ZJFBTIYTHVJ3ER6OMNP2WKR2G54WF6CUM3E3DAZZLYSGOCDWQ/target/classes/META-INF/fabric8/openshift/vertxapp-route.yml resource
[INFO] F8: validating /home/jenkins/workspace/jaseemabid_vertxapp_master-H32ZJFBTIYTHVJ3ER6OMNP2WKR2G54WF6CUM3E3DAZZLYSGOCDWQ/target/classes/META-INF/fabric8/kubernetes/vertxapp-svc.yml resource
[INFO] F8: validating /home/jenkins/workspace/jaseemabid_vertxapp_master-H32ZJFBTIYTHVJ3ER6OMNP2WKR2G54WF6CUM3E3DAZZLYSGOCDWQ/target/classes/META-INF/fabric8/kubernetes/vertxapp-deployment.yml resource
[INFO]
[INFO] --- maven-compiler-plugin:3.7.0:compile (default-compile) @ vertxapp ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 1 source file to /home/jenkins/workspace/jaseemabid_vertxapp_master-H32ZJFBTIYTHVJ3ER6OMNP2WKR2G54WF6CUM3E3DAZZLYSGOCDWQ/target/classes
[INFO]
[INFO] --- maven-resources-plugin:3.0.2:testResources (default-testResources) @ vertxapp ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 1 resource
[INFO]
[INFO] --- maven-compiler-plugin:3.7.0:testCompile (default-testCompile) @ vertxapp ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 2 source files to /home/jenkins/workspace/jaseemabid_vertxapp_master-H32ZJFBTIYTHVJ3ER6OMNP2WKR2G54WF6CUM3E3DAZZLYSGOCDWQ/target/test-classes
[INFO]
[INFO] --- maven-surefire-plugin:2.20:test (default-test) @ vertxapp ---
[INFO] Surefire report directory: /home/jenkins/workspace/jaseemabid_vertxapp_master-H32ZJFBTIYTHVJ3ER6OMNP2WKR2G54WF6CUM3E3DAZZLYSGOCDWQ/target/surefire-reports
[INFO]
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running io.openshift.booster.HttpApplicationTest
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Mar 01, 2018 2:08:46 PM io.netty.util.internal.MacAddressUtil defaultMachineId
WARNING: Failed to find a usable hardware address from the network interfaces; using random bytes: 36:84:47:cd:2c:ad:36:6f
Server started on port 8081
Server started on port 8081
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.508 s - in io.openshift.booster.HttpApplicationTest
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO]
[INFO] --- maven-jar-plugin:3.0.2:jar (default-jar) @ vertxapp ---
[INFO] Building jar: /home/jenkins/workspace/jaseemabid_vertxapp_master-H32ZJFBTIYTHVJ3ER6OMNP2WKR2G54WF6CUM3E3DAZZLYSGOCDWQ/target/vertxapp-1.0.3.jar
[INFO]
[INFO] --- vertx-maven-plugin:1.0.13:package (vmp) @ vertxapp ---
[INFO] Executing: /bin/sh -c cd '/home/jenkins/workspace/jaseemabid_vertxapp_master-H32ZJFBTIYTHVJ3ER6OMNP2WKR2G54WF6CUM3E3DAZZLYSGOCDWQ' && 'git' 'whatchanged' '--until=2018-03-01 14:08:48 +0000' '--date=iso' '--' '/home/jenkins/workspace/jaseemabid_vertxapp_master-H32ZJFBTIYTHVJ3ER6OMNP2WKR2G54WF6CUM3E3DAZZLYSGOCDWQ'
[INFO] Working directory: /home/jenkins/workspace/jaseemabid_vertxapp_master-H32ZJFBTIYTHVJ3ER6OMNP2WKR2G54WF6CUM3E3DAZZLYSGOCDWQ
[INFO]
[INFO] --- fabric8-maven-plugin:3.5.28:build (fmp) @ vertxapp ---
[INFO] F8: Using OpenShift build with strategy S2I
[INFO] F8: Running generator vertx
[INFO] F8: vertx: Using Docker image registry.access.redhat.com/redhat-openjdk-18/openjdk18-openshift as base / builder
[INFO] Copying files to /home/jenkins/workspace/jaseemabid_vertxapp_master-H32ZJFBTIYTHVJ3ER6OMNP2WKR2G54WF6CUM3E3DAZZLYSGOCDWQ/target/docker/vertxapp/1.0.3/build/maven
[INFO] Building tar: /home/jenkins/workspace/jaseemabid_vertxapp_master-H32ZJFBTIYTHVJ3ER6OMNP2WKR2G54WF6CUM3E3DAZZLYSGOCDWQ/target/docker/vertxapp/1.0.3/tmp/docker-build.tar
[INFO] F8: [vertxapp:1.0.3] "vertx": Created docker source tar /home/jenkins/workspace/jaseemabid_vertxapp_master-H32ZJFBTIYTHVJ3ER6OMNP2WKR2G54WF6CUM3E3DAZZLYSGOCDWQ/target/docker/vertxapp/1.0.3/tmp/docker-build.tar
[INFO] F8: Creating BuildServiceConfig vertxapp-s2i for Source build
[INFO] F8: Creating ImageStream vertxapp
[INFO] F8: Starting Build vertxapp-s2i
[INFO] F8: Waiting for build vertxapp-s2i-1 to complete...
[INFO] Current reconnect backoff is 1000 milliseconds (T0)
[INFO] Current reconnect backoff is 2000 milliseconds (T1)
Mar 01, 2018 2:09:03 PM okhttp3.internal.platform.Platform log
WARNING: A connection to https://kubernetes.default.svc/ was leaked. Did you forget to close a response body? To see where this was allocated, set the OkHttpClient logger level to FINE: Logger.getLogger(OkHttpClient.class.getName()).setLevel(Level.FINE);
[INFO] Current reconnect backoff is 4000 milliseconds (T2)
[INFO] Current reconnect backoff is 8000 milliseconds (T3)
[INFO] Current reconnect backoff is 16000 milliseconds (T4)
[INFO] Current reconnect backoff is 32000 milliseconds (T5)
[INFO] F8: Build vertxapp-s2i-1 Complete
[WARNING] Executor didn't terminate in time after shutdown in close(), killing it in: io.fabric8.kubernetes.client.dsl.internal.WatchHTTPManager@efabd6
[INFO] F8: Found tag on ImageStream vertxapp tag: sha256:23e26d6475e04fc9e0bd550e175a217794baa9ef3eec0097fc98ed11c81ee651
[INFO] F8: ImageStream vertxapp written to /home/jenkins/workspace/jaseemabid_vertxapp_master-H32ZJFBTIYTHVJ3ER6OMNP2WKR2G54WF6CUM3E3DAZZLYSGOCDWQ/target/vertxapp-is.yml
[INFO]
[INFO] --- maven-install-plugin:2.5.2:install (default-install) @ vertxapp ---
[INFO] Installing /home/jenkins/workspace/jaseemabid_vertxapp_master-H32ZJFBTIYTHVJ3ER6OMNP2WKR2G54WF6CUM3E3DAZZLYSGOCDWQ/target/vertxapp-1.0.3.jar to /root/.mvnrepository/io/openshift/booster/vertxapp/1.0.3/vertxapp-1.0.3.jar
[INFO] Installing /home/jenkins/workspace/jaseemabid_vertxapp_master-H32ZJFBTIYTHVJ3ER6OMNP2WKR2G54WF6CUM3E3DAZZLYSGOCDWQ/pom.xml to /root/.mvnrepository/io/openshift/booster/vertxapp/1.0.3/vertxapp-1.0.3.pom
[INFO] Installing /home/jenkins/workspace/jaseemabid_vertxapp_master-H32ZJFBTIYTHVJ3ER6OMNP2WKR2G54WF6CUM3E3DAZZLYSGOCDWQ/target/classes/META-INF/fabric8/openshift.yml to /root/.mvnrepository/io/openshift/booster/vertxapp/1.0.3/vertxapp-1.0.3-openshift.yml
[INFO] Installing /home/jenkins/workspace/jaseemabid_vertxapp_master-H32ZJFBTIYTHVJ3ER6OMNP2WKR2G54WF6CUM3E3DAZZLYSGOCDWQ/target/classes/META-INF/fabric8/openshift.json to /root/.mvnrepository/io/openshift/booster/vertxapp/1.0.3/vertxapp-1.0.3-openshift.json
[INFO] Installing /home/jenkins/workspace/jaseemabid_vertxapp_master-H32ZJFBTIYTHVJ3ER6OMNP2WKR2G54WF6CUM3E3DAZZLYSGOCDWQ/target/classes/META-INF/fabric8/kubernetes.yml to /root/.mvnrepository/io/openshift/booster/vertxapp/1.0.3/vertxapp-1.0.3-kubernetes.yml
[INFO] Installing /home/jenkins/workspace/jaseemabid_vertxapp_master-H32ZJFBTIYTHVJ3ER6OMNP2WKR2G54WF6CUM3E3DAZZLYSGOCDWQ/target/classes/META-INF/fabric8/kubernetes.json to /root/.mvnrepository/io/openshift/booster/vertxapp/1.0.3/vertxapp-1.0.3-kubernetes.json
[INFO] Installing /home/jenkins/workspace/jaseemabid_vertxapp_master-H32ZJFBTIYTHVJ3ER6OMNP2WKR2G54WF6CUM3E3DAZZLYSGOCDWQ/target/vertxapp-is.yml to /root/.mvnrepository/io/openshift/booster/vertxapp/1.0.3/vertxapp-1.0.3-is.yml
[INFO]
[INFO] --- maven-deploy-plugin:2.8.2:deploy (default-deploy) @ vertxapp ---
[INFO] Using alternate deployment repository local-nexus::default::http://content-repository/content/repositories/staging/
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 01:21 min
[INFO] Finished at: 2018-03-01T14:09:34+00:00
[INFO] Final Memory: 41M/67M
[INFO] ------------------------------------------------------------------------
[Pipeline] findFiles
[Pipeline] step
Recording test results
[Pipeline] findFiles
[Pipeline] echo
Looking for matching Build vertxapp-3
[Pipeline] echo
Looking for matching Build vertxapp-3
[Pipeline] echo
Adding annotation 'fabric8.io/jenkins.testReportUrl: nulltestReport' to Build vertxapp-3
looking for vertxapp-3 in namespace jabid
[Pipeline] echo
[Pipeline] echo
Checking bayesian-link exists
[Pipeline] sh
[jaseemabid_vertxapp_master-H32ZJFBTIYTHVJ3ER6OMNP2WKR2G54WF6CUM3E3DAZZLYSGOCDWQ] Running shell script
+ mvn io.github.stackinfo:stackinfo-maven-plugin:0.2:prepare
Picked up _JAVA_OPTIONS: -Duser.home=/root/ -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -Dsun.zip.disableMemoryMapping=true -XX:+UseParallelGC -XX:MinHeapFreeRatio=5 -XX:MaxHeapFreeRatio=10 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -Xms10m -Xmx192m
[INFO] Scanning for projects...
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building Vert.x - HTTP 1.0.3
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- stackinfo-maven-plugin:0.2:prepare (default-cli) @ vertxapp ---
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 1.696 s
[INFO] Finished at: 2018-03-01T14:09:48+00:00
[INFO] Final Memory: 4M/9M
[INFO] ------------------------------------------------------------------------
[Pipeline] retry
[Pipeline] {
[Pipeline] bayesianAnalysis
Running Bayesian stack analysis...
Bayesian API URL is https://recommender.api.openshift.io/api/v1
[Pipeline] echo
Looking for matching Build vertxapp-3
[Pipeline] echo
Adding annotation 'fabric8.io/bayesian.analysisUrl: https://recommender.api.openshift.io/api/v1/stack-analyses/3194fc1620fe4f61835665b996cb7f0c' to Build vertxapp-3
[Pipeline] echo
looking for vertxapp-3 in namespace jabid
[Pipeline] }
[Pipeline] // retry
[Pipeline] echo
Checking sonarqube exists
[Pipeline] echo
Code validation service: sonarqube not available
[Pipeline] echo
s2i mode: true
[Pipeline] echo
Checking content-repository exists
[Pipeline] echo
mvn site disabled
[Pipeline] stash
Stashed 11 file(s)
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Rollout to Stage)
[Pipeline] unstash
[Pipeline] kubernetesApply
Loading OpenShift ImageStreams file: vertxapp-is.yml
Found namespace jabid
Found namespace jabid-jenkins
Found namespace jabid-run
Found namespace jabid-stage
Found existing environment jabid-stage
No fabric8.yml so unable to add environment pod annotations
No fabric8.yml so unable to add environment pod annotations
No fabric8.yml so unable to add environment pod annotations
No fabric8.yml so unable to add environment pod annotations
No fabric8.yml so unable to add environment pod annotations
Found BuildConfig name vertxapp namespace jabid from BuildConfigProjectProperty
Found Build name vertxapp-3 from BuildCause
Found BuildConfig: vertxapp namespace: jabid Build: vertxapp-3
Applying Service name: Service data: Service(apiVersion=v1, kind=Service, metadata=ObjectMeta(annotations={fabric8.io/git-commit=a836a698b646f7869906d572af76de7771ae43f7, fabric8.io/iconUrl=img/icons/vertx.svg, fabric8.io/git-branch=jaseemabid/vertxapp/master-1.0.3, prometheus.io/scrape=true, prometheus.io/port=9779}, clusterName=null, creationTimestamp=null, deletionGracePeriodSeconds=null, deletionTimestamp=null, finalizers=[], generateName=null, generation=null, initializers=null, labels={expose=true, space=vertx_app, app=vertxapp, provider=fabric8, version=1.0.3, group=io.openshift.booster}, name=vertxapp, namespace=null, ownerReferences=[], resourceVersion=null, selfLink=null, uid=null, additionalProperties={}), spec=ServiceSpec(clusterIP=null, externalIPs=[], externalName=null, externalTrafficPolicy=null, healthCheckNodePort=null, loadBalancerIP=null, loadBalancerSourceRanges=[], ports=[ServicePort(name=http, nodePort=null, port=8080, protocol=TCP, targetPort=IntOrString(IntVal=8080, Kind=null, StrVal=null, additionalProperties={}), additionalProperties={})], selector={app=vertxapp, provider=fabric8, group=io.openshift.booster}, sessionAffinity=null, type=null, additionalProperties={}), status=null, additionalProperties={})
Applying ImageStream name: ImageStream data: ImageStream(apiVersion=v1, kind=ImageStream, metadata=ObjectMeta(annotations=null, clusterName=null, creationTimestamp=null, deletionGracePeriodSeconds=null, deletionTimestamp=null, finalizers=[], generateName=null, generation=null, initializers=null, labels=null, name=vertxapp, namespace=null, ownerReferences=[], resourceVersion=null, selfLink=null, uid=null, additionalProperties={}), spec=ImageStreamSpec(dockerImageRepository=null, lookupPolicy=null, tags=[TagReference(annotations=null, from=ObjectReference(apiVersion=null, fieldPath=null, kind=ImageStreamImage, name=vertxapp@sha256:23e26d6475e04fc9e0bd550e175a217794baa9ef3eec0097fc98ed11c81ee651, namespace=jabid-jenkins, resourceVersion=null, uid=null, additionalProperties={}), generation=null, importPolicy=null, name=1.0.3, reference=null, referencePolicy=null, additionalProperties={})], additionalProperties={}), status=null, additionalProperties={})
Applying DeploymentConfig name: DeploymentConfig data: DeploymentConfig(apiVersion=v1, kind=DeploymentConfig, metadata=ObjectMeta(annotations={fabric8.io/git-commit=a836a698b646f7869906d572af76de7771ae43f7, fabric8.io/iconUrl=img/icons/vertx.svg, fabric8.io/git-branch=jaseemabid/vertxapp/master-1.0.3, fabric8.io/metrics-path=dashboard/file/kubernetes-pods.json/?var-project=vertxapp&var-version=1.0.3}, clusterName=null, creationTimestamp=null, deletionGracePeriodSeconds=null, deletionTimestamp=null, finalizers=[], generateName=null, generation=null, initializers=null, labels={space=vertx_app, app=vertxapp, provider=fabric8, version=1.0.3, group=io.openshift.booster}, name=vertxapp, namespace=null, ownerReferences=[], resourceVersion=null, selfLink=null, uid=null, additionalProperties={}), spec=DeploymentConfigSpec(minReadySeconds=null, paused=null, replicas=1, revisionHistoryLimit=2, selector={app=vertxapp, provider=fabric8, group=io.openshift.booster}, strategy=DeploymentStrategy(activeDeadlineSeconds=null, annotations=null, customParams=null, labels=null, recreateParams=null, resources=null, rollingParams=RollingDeploymentStrategyParams(intervalSeconds=null, maxSurge=null, maxUnavailable=null, post=null, pre=null, timeoutSeconds=3600, updatePeriodSeconds=null, additionalProperties={}), type=Rolling, additionalProperties={}), template=PodTemplateSpec(metadata=ObjectMeta(annotations={fabric8.io/git-commit=a836a698b646f7869906d572af76de7771ae43f7, fabric8.io/iconUrl=img/icons/vertx.svg, fabric8.io/git-branch=jaseemabid/vertxapp/master-1.0.3, fabric8.io/metrics-path=dashboard/file/kubernetes-pods.json/?var-project=vertxapp&var-version=1.0.3}, clusterName=null, creationTimestamp=null, deletionGracePeriodSeconds=null, deletionTimestamp=null, finalizers=[], generateName=null, generation=null, initializers=null, labels={space=vertx_app, app=vertxapp, provider=fabric8, version=1.0.3, group=io.openshift.booster}, name=null, namespace=null, ownerReferences=[], resourceVersion=null, selfLink=null, uid=null, additionalProperties={}), spec=PodSpec(activeDeadlineSeconds=null, affinity=null, automountServiceAccountToken=null, containers=[Container(args=[], command=[], env=[EnvVar(name=KUBERNETES_NAMESPACE, value=null, valueFrom=EnvVarSource(configMapKeyRef=null, fieldRef=ObjectFieldSelector(apiVersion=null, fieldPath=metadata.namespace, additionalProperties={}), resourceFieldRef=null, secretKeyRef=null, additionalProperties={}), additionalProperties={})], envFrom=[], image=vertxapp:1.0.3, imagePullPolicy=IfNotPresent, lifecycle=null, livenessProbe=Probe(exec=null, failureThreshold=null, httpGet=HTTPGetAction(host=null, httpHeaders=[], path=/, port=IntOrString(IntVal=8080, Kind=null, StrVal=null, additionalProperties={}), scheme=HTTP, additionalProperties={}), initialDelaySeconds=180, periodSeconds=null, successThreshold=null, tcpSocket=null, timeoutSeconds=null, additionalProperties={}), name=vertx, ports=[ContainerPort(containerPort=8080, hostIP=null, hostPort=null, name=http, protocol=TCP, additionalProperties={}), ContainerPort(containerPort=9779, hostIP=null, hostPort=null, name=prometheus, protocol=TCP, additionalProperties={}), ContainerPort(containerPort=8778, hostIP=null, hostPort=null, name=jolokia, protocol=TCP, additionalProperties={})], readinessProbe=Probe(exec=null, failureThreshold=null, httpGet=HTTPGetAction(host=null, httpHeaders=[], path=/, port=IntOrString(IntVal=8080, Kind=null, StrVal=null, additionalProperties={}), scheme=HTTP, additionalProperties={}), initialDelaySeconds=10, periodSeconds=null, successThreshold=null, tcpSocket=null, timeoutSeconds=null, additionalProperties={}), resources=null, securityContext=SecurityContext(capabilities=null, privileged=false, readOnlyRootFilesystem=null, runAsNonRoot=null, runAsUser=null, seLinuxOptions=null, additionalProperties={}), stdin=null, stdinOnce=null, terminationMessagePath=null, terminationMessagePolicy=null, tty=null, volumeMounts=[], workingDir=null, additionalProperties={})], dnsPolicy=null, hostAliases=[], hostIPC=null, hostNetwork=null, hostPID=null, hostname=null, imagePullSecrets=[], initContainers=[], nodeName=null, nodeSelector=null, restartPolicy=null, schedulerName=null, securityContext=null, serviceAccount=null, serviceAccountName=null, subdomain=null, terminationGracePeriodSeconds=null, tolerations=[], volumes=[], additionalProperties={}), additionalProperties={}), test=null, triggers=[DeploymentTriggerPolicy(imageChangeParams=null, type=ConfigChange, additionalProperties={}), DeploymentTriggerPolicy(imageChangeParams=DeploymentTriggerImageChangeParams(automatic=true, containerNames=[vertx], from=ObjectReference(apiVersion=null, fieldPath=null, kind=ImageStreamTag, name=vertxapp:1.0.3, namespace=null, resourceVersion=null, uid=null, additionalProperties={}), lastTriggeredImage=null, additionalProperties={}), type=ImageChange, additionalProperties={})], additionalProperties={}), status=null, additionalProperties={})
Applying Route name: Route data: Route(apiVersion=v1, kind=Route, metadata=ObjectMeta(annotations=null, clusterName=null, creationTimestamp=null, deletionGracePeriodSeconds=null, deletionTimestamp=null, finalizers=[], generateName=null, generation=null, initializers=null, labels={app=vertxapp, provider=fabric8, version=1.0.3, group=io.openshift.booster}, name=vertxapp, namespace=null, ownerReferences=[], resourceVersion=null, selfLink=null, uid=null, additionalProperties={}), spec=RouteSpec(alternateBackends=[], host=null, path=null, port=RoutePort(targetPort=IntOrString(IntVal=8080, Kind=null, StrVal=null, additionalProperties={}), additionalProperties={}), tls=null, to=RouteTargetReference(kind=Service, name=vertxapp, weight=null, additionalProperties={}), wildcardPolicy=null, additionalProperties={}), status=null, additionalProperties={})
Running: echo | exposecontroller --watch-namespace jabid-stage --exposer Route
command: echo | exposecontroller --watch-namespace jabid-stage --exposer Route out: Please enter Username: Please enter Password:
command: echo | exposecontroller --watch-namespace jabid-stage --exposer Route err: I0301 14:10:02.276861     470 exposecontroller.go:56] Using build: '2.3.27'
I0301 14:10:02.301360     470 config.go:20] No /etc/exposecontroller/config.yml file found.  Will try to figure out defaults
I0301 14:10:02.301413     470 exposecontroller.go:125] Running in : `jabid-stage`
W0301 14:10:02.358563     470 controller.go:478] Failed to load url https://openshift.default.svc/.well-known/oauth-authorization-server got: Get https://openshift.default.svc/.well-known/oauth-authorization-server: x509: certificate signed by unknown authority
E0301 14:10:02.359929     470 controller.go:186] Failed to list nodes to detect minishift: the server does not allow access to the requested resource (get nodes)
I0301 14:10:02.359951     470 controller.go:118] Using OAuth Authorize URL:
W0301 14:10:02.359959     470 controller.go:120] Please use $OAUTH_AUTHORIZE_URL to define the OAuth Authorize URL!
W0301 14:10:02.361334     470 controller.go:251] Could not find kubernetes service in the default namespace so we could not detect whether to use http or https as the apiserver protocol. Error: the server does not allow access to the requested resource (get services kubernetes)
I0301 14:10:02.361379     470 controller.go:534] starting expose controller
I0301 14:10:07.361572     470 controller.go:542] stopping expose controller

exposecontroller completed
Service vertxapp in environment Stage namespace jabid-stage is at URL: http://vertxapp-jabid-stage.8a09.starter-us-east-2.openshiftapps.com
Annotated Build vertxapp-3 in namespace jabid with version 1.0.3 environment yaml jabid-stage=---
environmentName: "Stage"
serviceUrls:
  vertxapp: "http://vertxapp-jabid-stage.8a09.starter-us-east-2.openshiftapps.com"
deploymentVersions:
  vertxapp: "1.0.3"

[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // container
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // podTemplate
[Pipeline] End of Pipeline

GitHub has been notified of this commit’s build result

Finished: SUCCESS

[jaseemabid]

@hrishin Are you sure you could access the app? I cant.

[jaseemabid]

This took quite some frustrating effort to dig into. We forked the kubernetes-pipeline-plugin added a lot of expose controller specific code into it. I was looking at the upstream code for a long time :man_shrugging:

@maxandersen I think we broke vertx quickstarts on prod a long time ago and nobody even noticed. I don't know enough about this to take a call on this issue right now.

/ cc: @pradeepto @kbsingh

[aslakknutsen]

@jaseemabid so according to you this has been broken since October 2017?

[jaseemabid]

@aslakknutsen Let me try to reproduce this with some other account.

[jaseemabid]

@aslakknutsen I tried the same with @jarifibrahim's account, but we never got past All nodes of label ‘mavenvertx2_2’ are offline even after several attempts. @rupalibehera Can you please try it out once?

[rupalibehera]

@jaseemabid, I see the same error but, I am able access the app's deployed in stage and run environment.

command: echo | exposecontroller --watch-namespace rbehera-1-run --exposer Route out: Please enter Username: Please enter Password: 
command: echo | exposecontroller --watch-namespace rbehera-1-run --exposer Route err: I0301 15:29:35.178807     756 exposecontroller.go:56] Using build: '2.3.27'
I0301 15:29:35.191601     756 config.go:20] No /etc/exposecontroller/config.yml file found.  Will try to figure out defaults
I0301 15:29:35.191635     756 exposecontroller.go:125] Running in : `rbehera-1-run`
W0301 15:29:35.222531     756 controller.go:478] Failed to load url https://openshift.default.svc/.well-known/oauth-authorization-server got: Get https://openshift.default.svc/.well-known/oauth-authorization-server: x509: certificate signed by unknown authority
E0301 15:29:35.223477     756 controller.go:186] Failed to list nodes to detect minishift: the server does not allow access to the requested resource (get nodes)
I0301 15:29:35.223491     756 controller.go:118] Using OAuth Authorize URL: 
W0301 15:29:35.223497     756 controller.go:120] Please use $OAUTH_AUTHORIZE_URL to define the OAuth Authorize URL!
W0301 15:29:35.224265     756 controller.go:251] Could not find kubernetes service in the default namespace so we could not detect whether to use http or https as the apiserver protocol. Error: the server does not allow access to the requested resource (get services kubernetes)
I0301 15:29:35.224303     756 controller.go:534] starting expose controller
I0301 15:29:40.224491     756 controller.go:542] stopping expose controller```

[ldimaggi]

Am also seeing this same error:

I0302 14:39:44.973538    1033 exposecontroller.go:125] Running in : `ldimaggi-stage`
W0302 14:39:45.006297    1033 controller.go:478] Failed to load url https://openshift.default.svc/.well-known/oauth-authorization-server got: Get https://openshift.default.svc/.well-known/oauth-authorization-server: x509: certificate signed by unknown authority
E0302 14:39:45.007310    1033 controller.go:186] Failed to list nodes to detect minishift: the server does not allow access to the requested resource (get nodes)
I0302 14:39:45.007329    1033 controller.go:118] Using OAuth Authorize URL: 
W0302 14:39:45.007335    1033 controller.go:120] Please use $OAUTH_AUTHORIZE_URL to define the OAuth Authorize URL!
W0302 14:39:45.008074    1033 controller.go:251] Could not find kubernetes service in the default namespace so we could not detect whether to use http or https as the apiserver protocol. Error: the server does not allow access to the requested resource (get services kubernetes)
I0302 14:39:45.008110    1033 controller.go:534] starting expose controller
I0302 14:39:50.008333    1033 controller.go:542] stopping expose controller

But - the app is deployed, is reachable, and runs on both stage and run.

[hrishin]

yes @jaseemabid http://vertxhttpbooster-hshinde-stage.8a09.starter-us-east-2.openshiftapps.com

Here is the app, like others I'm also facing expose controller error logs but the app is getting deployed in the staging environment.

[jaseemabid]

Now I've to find out why my deployments are failing. How long does it take usually for a deployment to come up @hrishin?

[joshuawilson]

@ldimaggi why is this not a bug? What type should it be?

[jaseemabid]

I can deploy a 'Vertx health check' app, but the logs show very similar errors.

I0305 09:21:35.698243     956 config.go:20] No /etc/exposecontroller/config.yml file found.  Will try to figure out defaults
I0305 09:21:35.698291     956 exposecontroller.go:125] Running in : `jabid-run`
W0305 09:21:35.731148     956 controller.go:478] Failed to load url https://openshift.default.svc/.well-known/oauth-authorization-server got: Get https://openshift.default.svc/.well-known/oauth-authorization-server: x509: certificate signed by unknown authority
E0305 09:21:35.732167     956 controller.go:186] Failed to list nodes to detect minishift: the server does not allow access to the requested resource (get nodes)
I0305 09:21:35.732211     956 controller.go:118] Using OAuth Authorize URL: 
W0305 09:21:35.732217     956 controller.go:120] Please use $OAUTH_AUTHORIZE_URL to define the OAuth Authorize URL!
W0305 09:21:35.732993     956 controller.go:251] Could not find kubernetes service in the default namespace so we could not detect whether to use http or https as the apiserver protocol. Error: the server does not allow access to the requested resource (get services kubernetes)
I0305 09:21:35.733032     956 controller.go:534] starting expose controller
I0305 09:21:40.733226     956 controller.go:542] stopping expose controller

@ldimaggi Out dependency on exposecontroller is most probably a bug and should be fixed.

[chmouel]

@jaseemabid here @joshuawilson is pointing you out that we need to add a label on issue for classifications when you log an issue on openshift.io/

[joshuawilson]

Thanks @chmouel that is correct. Now that you made it a type/bug, what is the severity?

[jaseemabid]

@joshuawilson This is not blocking anyone from deploying code, but is definitely pointing towards a few other problems. SEV2 or 3. Someone from build team needs to investigate this and I don't when we will be able to.

[joshuawilson]

Is the issue noticeable and annoying but you can work around it? then SEV2 Is the issue noticeable but you can live with it? then SEV3 is the issue not obvious or cosmetic? then SEV4

[hrishin]

How long does it take usually for a deployment to come up?

Around 12-14 minutes for the entire build. After that app is immediately getting available to access.

[jaseemabid]

@sthaha I've updated the title and description. This is not a blocker anymore, just something wrong related to deployments.

[sthaha]

I took a measurement of how long it takes to execute these statemetns and fail and it seems the saving is only 6 seconds

00:05:52.855 Running: echo | exposecontroller --watch-namespace sunil-thaha-stage --exposer Route
00:05:58.095 command: echo | exposecontroller --watch-namespace sunil-thaha-stage --exposer Route out: Please enter Username: Please enter Password:
00:05:58.096 command: echo | exposecontroller --watch-namespace sunil-thaha-stage --exposer Route err: I0315 09:03:47.926726     619 exposecontroller.go:56] Using build: '2.3.27'
00:05:58.096 I0315 09:03:47.948940     619 config.go:20] No /etc/exposecontroller/config.yml file found.  Will try to figure out defaults
00:05:58.096 I0315 09:03:47.948976     619 exposecontroller.go:125] Running in : `sunil-thaha-stage`
00:05:58.096 W0315 09:03:48.030514     619 controller.go:478] Failed to load url https://openshift.default.svc/.well-known/oauth-authorization-server got: Get https://openshift.default.svc/.well-known/oauth-authorization-server: x509: certificate signed by unknown authority
00:05:58.096 E0315 09:03:48.031446     619 controller.go:186] Failed to list nodes to detect minishift: the server does not allow access to the requested resource (get nodes)
00:05:58.096 I0315 09:03:48.031470     619 controller.go:118] Using OAuth Authorize URL:
00:05:58.096 W0315 09:03:48.031476     619 controller.go:120] Please use $OAUTH_AUTHORIZE_URL to define the OAuth Authorize URL!
00:05:58.096 W0315 09:03:48.032236     619 controller.go:251] Could not find kubernetes service in the default namespace so we could not detect whether to use http or https as the apiserver protocol. Error: the server does not allow access to the requested resource (get services kubernetes)
00:05:58.096 I0315 09:03:48.032268     619 controller.go:534] starting expose controller
00:05:58.096 I0315 09:03:53.032450     619 controller.go:542] stopping expose controller
00:05:58.096
00:05:58.096 exposecontroller completed

@pradeepto I am going move this task back to backlog since I am working on the sev-1 : https://github.com/openshiftio/openshift.io/issues/2427

[jaseemabid]

@sthaha To make my point clear, I'm not worried about the performance implications, this is about us doing something obviously wrong with a software we aren't even suppose to use, but somehow the build succeeded because of a retry logic somewhere in the Jenkinsfile.

[chmouel]

@jaseemabid I was one worried if there was perf issues from this, we all agree we should still fix it since it's ugly as...

[hrishin]

This would go away with OSIO pipeline library. As sooner or later we will move Java projects to new us new pipeline library.

cc: @sthaha @rupalibehera

[sthaha]

Going to mark this as "won't fix" since this will be solved in by the new pipeline - https://github.com/fabric8io/osio-pipeline