cf bulk push from Jenkins intermittently errors with No API endpoint set. Use 'cf login' or 'cf api' to target an endpoint.

[mchada]

The CF_HOMEenvironemnt veariable is set to unique path for each job and all the jobs use same user id. When doing bulk deployment, the config.json is somehow getting replaced/modified. While cf push fail, the ls CF_HOME/.cf throws no such file or directory saying the file is deleted by something. Pivotal ticket #33622 is created for this issue.

CLI 6.21.1 api_version:"2.43.0" cf 1.6

• export 'CF_HOME=/proj/gforce-jenkins/.jenkins/workspace/2. Config PROD Deploy Numbered PROD Version'
• CF_HOME='/proj/gforce-jenkins/.jenkins/workspace/2. Config PROD Deploy Numbered PROD Version'
• ./cf_login.sh pfmc GFORCEV3-PROD ls -l CF_HOME/.cf ls: cannot access /proj/gforce-jenkins/.jenkins/workspace/2. Config PROD Deploy Numbered PROD Version/.cf: No such file or directory

  detailed log:

Intermittent push failure report

Date 12-aug-2016 Time 12:00pm ORG: GFORCEV3.REDACTED.NA SPACE: GFORCEV3-Alpha cf Server: PreProd FMC Number of failed pushes: 1 Components that failed to push: yakback

Data that follows: Push logs for each micro service that failed. Console output Config.json

Cf push log for yakback: FAILED No API endpoint set. Use 'cf login' or 'cf api' to target an endpoint.

Console output:

Started by upstream project "ETL_Release_Set" build number 2047 originally caused by: Started by an SCM change [EnvInject] - Loading node environment variables. Building in workspace /proj/gforce-jenkins/.jenkins/workspace/ETL_Acceptance_Tests [WS-CLEANUP] Deleting project workspace... [WS-CLEANUP] Done Cloning the remote Git repository Cloning repository ssh://gfrcjnkd@wwwdev.gforce-gitblit.REDACTED.com:29418/devops/cf.git

git init /proj/gforce-jenkins/.jenkins/workspace/ETL_AcceptanceTests # timeout=10 Fetching upstream changes from ssh://gfrcjnkd@wwwdev.gforce-gitblit.REDACTED.com:29418/devops/cf.git git --version # timeout=10 git -c core.askpass=true fetch --tags --progress ssh://gfrcjnkd@wwwdev.gforce-gitblit.REDACTED.com:29418/devops/cf.git +refs/heads/:refs/remotes/origin/ git config remote.origin.url ssh://gfrcjnkd@wwwdev.gforce-gitblit.REDACTED.com:29418/devops/cf.git # timeout=10 git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 git config remote.origin.url ssh://gfrcjnkd@wwwdev.gforce-gitblit.REDACTED.com:29418/devops/cf.git # timeout=10 Fetching upstream changes from ssh://gfrcjnkd@wwwdev.gforce-gitblit.REDACTED.com:29418/devops/cf.git git -c core.askpass=true fetch --tags --progress ssh://gfrcjnkd@wwwdev.gforce-gitblit.REDACTED.com:29418/devops/cf.git +refs/heads/:refs/remotes/origin/_ git rev-parse refs/remotes/origin/master^{commit} # timeout=10 git rev-parse refs/remotes/origin/refs/heads/master^{commit} # timeout=10 Checking out Revision 85e040e44af9c0fc5cdc9a205ba721281dd6ce87 (refs/remotes/origin/master) git config core.sparsecheckout # timeout=10 git checkout -f 85e040e44af9c0fc5cdc9a205ba721281dd6ce87 # timeout=10 git branch -a -v --no-abbrev # timeout=10 git checkout -b master 85e040e44af9c0fc5cdc9a205ba721281dd6ce87 git rev-list 85e040e44af9c0fc5cdc9a205ba721281dd6ce87 # timeout=10 Copied 5,090 artifacts from "ETL_Release_Set" build number 2047 [EnvInject] - Injecting environment variables from a build step. [EnvInject] - Injecting as environment variables the properties file path 'ReleaseSet/build.properties' [EnvInject] - Variables injected successfully. [ETL_Acceptance_Tests] $ /bin/sh -xe /tmp/hudson6300977711290753543.sh

• cp ReleaseSet/MicroServices/gforce-authorization-0.0.1-SNAPSHOT.war ReleaseSet/MicroServices/gforce-biztalk-buddy-0.0.1-SNAPSHOT.war ReleaseSet/MicroServices/gforce-etl-0.0.1-SNAPSHOT.war ReleaseSet/MicroServices/gforce-mdd-bakery-0.0.1-SNAPSHOT.war ReleaseSet/MicroServices/gforce-mdd-big-bakery-0.0.1-SNAPSHOT.war ReleaseSet/MicroServices/gforce-mdd-store-0.0.1-SNAPSHOT.war ReleaseSet/MicroServices/gforce-notification-0.0.1-SNAPSHOT.war ReleaseSet/MicroServices/gforce-pdo-proxy-0.0.1-SNAPSHOT.war ReleaseSet/MicroServices/gforce-soe-proxy-0.0.1-SNAPSHOT.war . [ETL_Acceptance_Tests] $ /bin/sh -xe /tmp/hudson8041510028849013659.sh
• cp ReleaseSet/AlphaOnly/gforce-yakback-0.0.1-SNAPSHOT.war . [ETL_Acceptance_Tests] $ /bin/sh -xe /tmp/hudson6235573743738733376.sh
• mkdir -p stage
• cp ReleaseSet/MicroServices/gforce-authorization-0.0.1-SNAPSHOT.war ReleaseSet/MicroServices/gforce-biztalk-buddy-0.0.1-SNAPSHOT.war ReleaseSet/MicroServices/gforce-etl-0.0.1-SNAPSHOT.war ReleaseSet/MicroServices/gforce-mdd-bakery-0.0.1-SNAPSHOT.war ReleaseSet/MicroServices/gforce-mdd-big-bakery-0.0.1-SNAPSHOT.war ReleaseSet/MicroServices/gforce-mdd-store-0.0.1-SNAPSHOT.war ReleaseSet/MicroServices/gforce-notification-0.0.1-SNAPSHOT.war ReleaseSet/MicroServices/gforce-pdo-proxy-0.0.1-SNAPSHOT.war ReleaseSet/MicroServices/gforce-soe-proxy-0.0.1-SNAPSHOT.war stage [ETL_Acceptance_Tests] $ /bin/sh -xe /tmp/hudson8604730327992201783.sh
• cp gforce-yakback-0.0.1-SNAPSHOT.war stage [ETL_Acceptance_Tests] $ /bin/sh -xe /tmp/hudson3119764734139121929.sh
• cat [ETL_Acceptance_Tests] $ /bin/sh -xe /tmp/hudson9116196282484738306.sh
• cat [ETL_Acceptance_Tests] $ /bin/sh -xe /tmp/hudson8609869021489434967.sh
• unset http_proxy
• unset https_proxy
• export CF_HOME=/proj/gforce-jenkins/.jenkins/workspace/ETL_Acceptance_Tests
• CF_HOME=/proj/gforce-jenkins/.jenkins/workspace/ETL_Acceptance_Tests
• ./cf_login.sh ppfmc GFORCEV3-Alpha Logging into Cloud Foundry with server_code ppfmc space GFORCEV3-Alpha

Setting api endpoint to https://api.sys-q01.cfqafmc.REDACTED.com... OK

API endpoint: https://api.sys-q01.cfqafmc.REDACTED.com (API version: 2.43.0) Not logged in. Use 'cf login' to log in. API endpoint: https://api.sys-q01.cfqafmc.REDACTED.com Authenticating... OK Use 'cf target' to view or set your target org and space

API endpoint: https://api.sys-q01.cfqafmc.REDACTED.com (API version: 2.43.0) User: gforce-ci Org: GFORCEV3.REDACTED.NA Space: GFORCEV3-Alpha

• ./cf_delete_all.sh etl-alpha

Checking for running Micro Services for environment etl-alpha.

Found running Micro Services, deleting...

running_apps=authorization-etl-alpha biztalk-buddy-etl-alpha etl-etl-alpha mdd-bakery-etl-alpha mdd-big-bakery-etl-alpha mdd-store-etl-alpha notification-etl-alpha pdo-proxy-etl-alpha soe-proxy-etl-alpha yakback-etl-alpha Deleting app authorization-etl-alpha in org GFORCEV3.REDACTED.NA / space GFORCEV3-Alpha as gforce-ci... OK Deleting app biztalk-buddy-etl-alpha in org GFORCEV3.REDACTED.NA / space GFORCEV3-Alpha as gforce-ci... OK Deleting app etl-etl-alpha in org GFORCEV3.REDACTED.NA / space GFORCEV3-Alpha as gforce-ci... OK Deleting app mdd-bakery-etl-alpha in org GFORCEV3.REDACTED.NA / space GFORCEV3-Alpha as gforce-ci... OK Deleting app mdd-big-bakery-etl-alpha in org GFORCEV3.REDACTED.NA / space GFORCEV3-Alpha as gforce-ci... OK Deleting app mdd-store-etl-alpha in org GFORCEV3.REDACTED.NA / space GFORCEV3-Alpha as gforce-ci... OK Deleting app notification-etl-alpha in org GFORCEV3.REDACTED.NA / space GFORCEV3-Alpha as gforce-ci... OK Deleting app pdo-proxy-etl-alpha in org GFORCEV3.REDACTED.NA / space GFORCEV3-Alpha as gforce-ci... OK Deleting app soe-proxy-etl-alpha in org GFORCEV3.REDACTED.NA / space GFORCEV3-Alpha as gforce-ci... OK Deleting app yakback-etl-alpha in org GFORCEV3.REDACTED.NA / space GFORCEV3-Alpha as gforce-ci... OK Verifying Mcroservices have been deleted.

All Micro Services for environment etl-alpha have been deleted. Deleting orphaned routes...Getting routes as gforce-ci ...

Deleting route biztalk-buddy-etl-alpha.apps-q01.cfqafmc.REDACTED.com... Deleting route authorization-etl-alpha.apps-q01.cfqafmc.REDACTED.com... Deleting route mdd-big-bakery-etl-alpha.apps-q01.cfqafmc.REDACTED.com... Deleting route etl-etl-alpha.apps-q01.cfqafmc.REDACTED.com... Deleting route mdd-bakery-etl-alpha.apps-q01.cfqafmc.REDACTED.com... Deleting route pdo-proxy-etl-alpha.apps-q01.cfqafmc.REDACTED.com... Deleting route notification-etl-alpha.apps-q01.cfqafmc.REDACTED.com... Deleting route mdd-store-etl-alpha.apps-q01.cfqafmc.REDACTED.com... Deleting route soe-proxy-etl-alpha.apps-q01.cfqafmc.REDACTED.com... Deleting route yakback-etl-alpha.apps-q01.cfqafmc.REDACTED.com... OK done.[ETL_Acceptance_Tests] $ /bin/sh -xe /tmp/hudson3623794385913353744.sh

• /proj/gforce-jenkins/flyway-3.2.1/flyway -configFile=/proj/gforce-jenkins/.jenkins/workspace/ETL_Acceptance_Tests/build-scripts/flyway-linux.conf clean Flyway 3.2.1 by Boxfuse

Database: jdbc:jtds:sqlserver://ahivdb03162.ahi.REDACTED.com:1433/ETL_ALPHA_DB (Microsoft SQL Server 10.50) Cleaned schema [dbo](execution time 00:00.110s) [ETL_Acceptance_Tests] $ /bin/sh -xe /tmp/hudson8144892580192646608.sh

• /proj/gforce-jenkins/flyway-3.2.1/flyway -configFile=/proj/gforce-jenkins/.jenkins/workspace/ETL_Acceptance_Tests/build-scripts/flyway-linux.conf migrate Flyway 3.2.1 by Boxfuse

Database: jdbc:jtds:sqlserver://ahivdb03162.ahi.REDACTED.com:1433/ETL_ALPHA_DB (Microsoft SQL Server 10.50) Validated 20 migrations (execution time 00:00.038s) Creating Metadata table: [dbo].[schema_version] Current version of schema [dbo]: << Empty Schema >> WARNING: outOREDACTEDer mode is active. Migration of schema [dbo] may not be reproducible. Migrating schema [dbo] to version 101 - create-approval-notification-table Migrating schema [dbo] to version 102 - create-vehicle-line-lookup-table Migrating schema [dbo] to version 103 - create-biztalk-notification-table Migrating schema [dbo] to version 104 - create mdd store table Migrating schema [dbo] to version 109 - create-DANS-notification-table Migrating schema [dbo] to version 111 - create mdd white list table Migrating schema [dbo] to version 113 - create-market-type-table Migrating schema [dbo] to version 202 - approval notification inserts Migrating schema [dbo] to version 207 - vehicle line lookup inserts Migrating schema [dbo] to version 210 - dans notification data Migrating schema [dbo] to version 211 - initial mddstore inserts Migrating schema [dbo] to version 212 - mddstore update publishing date to null Migrating schema [dbo] to version 213 - whitelist for test Migrating schema [dbo] to version 214 - market type inserts Migrating schema [dbo] to version 2110 - insert mdd store CRUD TEST DATA Migrating schema [dbo] to version 2111 - insert WANAB-CC7-2016 732591 Migrating schema [dbo] to version 2112 - insert GEORGE-JSON-2062 Migrating schema [dbo] to version 2113 - insert WANAB-TY3-2016 Migrating schema [dbo] to version 2114 - insert WANAB-CC7-2016 813491 Migrating schema [dbo] to version 2115 - insert WANAB-CZG-2018 Successfully applied 20 migrations to schema [dbo](execution time 00:01.032s). [ETL_Acceptance_Tests] $ /bin/sh -xe /tmp/hudson7169589460106700353.sh

• unset http_proxy
• unset https_proxy
• export CF_HOME=/proj/gforce-jenkins/.jenkins/workspace/ETL_Acceptance_Tests
• CF_HOME=/proj/gforce-jenkins/.jenkins/workspace/ETL_Acceptance_Tests
• ./cf_login.sh ppfmc GFORCEV3-Alpha Logging into Cloud Foundry with server_code ppfmc space GFORCEV3-Alpha

Setting api endpoint to https://api.sys-q01.cfqafmc.REDACTED.com... OK

API endpoint: https://api.sys-q01.cfqafmc.REDACTED.com (API version: 2.43.0) User: gforce-ci Org: GFORCEV3.REDACTED.NA Space: GFORCEV3-Alpha API endpoint: https://api.sys-q01.cfqafmc.REDACTED.com Authenticating... OK Use 'cf target' to view or set your target org and space

API endpoint: https://api.sys-q01.cfqafmc.REDACTED.com (API version: 2.43.0) User: gforce-ci Org: GFORCEV3.REDACTED.NA Space: GFORCEV3-Alpha

• ./cf_bulk_deploy.sh etl-alpha Configuring for profile and environment etl-alpha processing stage/gforce-authorization-0.0.1-SNAPSHOT.war authorization 0 Creating manifest.authorization.yml cf push -f manifest.authorization.yml processing stage/gforce-biztalk-buddy-0.0.1-SNAPSHOT.war biztalk-buddy 1 Creating manifest.biztalk-buddy.yml cf push -f manifest.biztalk-buddy.yml processing stage/gforce-etl-0.0.1-SNAPSHOT.war etl 2 Creating manifest.etl.yml cf push -f manifest.etl.yml processing stage/gforce-mdd-bakery-0.0.1-SNAPSHOT.war mdd-bakery 3 Creating manifest.mdd-bakery.yml cf push -f manifest.mdd-bakery.yml processing stage/gforce-mdd-big-bakery-0.0.1-SNAPSHOT.war mdd-big-bakery 4 cf push -f manifest.mdd-big-bakery.yml processing stage/gforce-mdd-store-0.0.1-SNAPSHOT.war mdd-store 5 Creating manifest.mdd-store.yml cf push -f manifest.mdd-store.yml processing stage/gforce-notification-0.0.1-SNAPSHOT.war notification 6 Creating manifest.notification.yml cf push -f manifest.notification.yml processing stage/gforce-pdo-proxy-0.0.1-SNAPSHOT.war pdo-proxy 7 Creating manifest.pdo-proxy.yml cf push -f manifest.pdo-proxy.yml processing stage/gforce-soe-proxy-0.0.1-SNAPSHOT.war soe-proxy 8 Creating manifest.soe-proxy.yml cf push -f manifest.soe-proxy.yml processing stage/gforce-yakback-0.0.1-SNAPSHOT.war yakback 9 Creating manifest.yakback.yml cf push -f manifest.yakback.yml Started 10 wars pid_array = 1947 1959 1970 1981 1996 2008 2018 2032 2044 2059 component_array = authorization biztalk-buddy etl mdd-bakery mdd-big-bakery mdd-store notification pdo-proxy soe-proxy yakback Wating for all components to start Waiting for Component=authorization PID=1947...PID= SUCCESS exit code=0 Waiting for Component=biztalk-buddy PID=1959...PID= SUCCESS exit code=0 Waiting for Component=etl PID=1970...PID= SUCCESS exit code=0 Waiting for Component=mdd-bakery PID=1981...PID= SUCCESS exit code=0 Waiting for Component=mdd-big-bakery PID=1996...PID= SUCCESS exit code=0 Waiting for Component=mdd-store PID=2008...PID= SUCCESS exit code=0 Waiting for Component=notification PID=2018...PID= FAILURE exit code=1 Waiting for Component=pdo-proxy PID=2032...PID= FAILURE exit code=1 Waiting for Component=soe-proxy PID=2044...PID= SUCCESS exit code=0 Waiting for Component=yakback PID=2059...PID= FAILURE exit code=1 retcode=1 Build step 'Execute shell' marked build as failure Skipping publisher since build result is FAILURE Recording test results ERROR: Step ‘Publish JUnit test result report’ failed: No test report files were found. Configuration error? Archiving artifacts Started calculate disk usage of build Finished Calculation of disk usage of build in 0 seconds Started calculate disk usage of workspace Finished Calculation of disk usage of workspace in 0 seconds Warning: you have no plugins providing access control for builds, so falling back to legacy behavior of permitting any downstream builds to be triggered Finished: FAILURE

$WORKSPACE/.cf/config.json

/ 20160812132550 // http://wwwqa.gforce-jenkins.REDACTED.com:8080/view/ETL%20Build/job/ETL_Acceptance_Tests/ws/.cf/config.json/*view*/

{ "ConfigVersion": 3, "Target": "https://api.sys-q01.cfqafmc.REDACTED.com", "APIVersion": "2.43.0", "AuthorizationEndpoint": "https://login.sys-q01.cfqafmc.REDACTED.com", "LoggregatorEndPoint": "wss://loggregator.sys-q01.cfqafmc.REDACTED.com:443", "DopplerEndPoint": "wss://doppler.sys-q01.cfqafmc.REDACTED.com:443", "UaaEndpoint": "https://uaa.sys-q01.cfqafmc.REDACTED.com", "RoutingAPIEndpoint": "https://api.sys-q01.cfqafmc.REDACTED.com/routing", "AccessToken": "REDACTED", "SSHOAuthClient": "ssh-proxy", "RefreshToken": "REDACTEDQ", "OrganizationFields": { "GUID": "d49449a8-2ec8-467b-88f7-c508ae651868", "Name": "GFORCEV3.REDACTED.NA", "QuotaDefinition": { "guid": "03ce8e5c-ed2e-4ca9-b7c5-bd7d0d8fcf2d", "name": "Extended", "memory_limit": 262144, "instance_memory_limit": -1, "total_routes": 1000, "total_services": -1, "non_basic_services_allowed": false, "app_instance_limit": -1 } }, "SpaceFields": { "GUID": "c65a7c0e-351e-4f39-addb-e0021c83370f", "Name": "GFORCEV3-Alpha", "AllowSSH": true }, "SSLDisabled": false, "AsyncTimeout": 0, "Trace": "", "ColorEnabled": "", "Locale": "", "PluginRepos": [ { "Name": "CF-Community", "URL": "https://plugins.cloudfoundry.org" } ], "MinCLIVersion": "6.7.0", "MinRecommendedCLIVersion": "6.11.2" }

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/128829195

The labels on this github issue will be updated when the story is started.

[dkoper]

Thanks for raising the issue. We'll take a look if we can fix this.

[dkoper]

I heard from the team they tried to reach out to ask for more details as they couldn't reproduce it. Feel free to reopen or submit a new issue if you can provide more details like the OS and filesystem.