search

cloudfoundry / disaster-recovery-acceptance-tests

Acceptance tests for disaster recovery of Cloud Foundry
Apache License 2.0
8 stars 33 forks source link

DRATs failing in cf-deployment pipeline #326

Open jochenehret opened 5 months ago

jochenehret commented 5 months ago

Hi BBR Team,

the DRATs are currently failing in the cf-deployment pipeline: https://concourse.wg-ard.ci.cloudfoundry.org/teams/main/pipelines/cf-deployment/jobs/bbr-run-drats/builds/1339

It could be related to the removal of the CF CLI v6 from the BOSH cf-cli package: https://github.com/cloudfoundry/cf-deployment/pull/1184

Can you please check? This is blocking the cf-deployment pipeline. I can provide access to the bbl env, if needed.

Thanks for your support and best regards,

Jochen.

cf-gitbot commented 5 months ago

We have created an issue in Pivotal Tracker to manage this. Unfortunately, the Pivotal Tracker project is private so you may be unable to view the contents of the story.

The labels on this github issue will be updated when the story is started.

jochenehret commented 5 months ago

Test is green again: https://concourse.wg-ard.ci.cloudfoundry.org/teams/main/pipelines/cf-deployment/jobs/bbr-run-drats/builds/1343 Possibly from the update to https://github.com/cloudfoundry/backup-and-restore-sdk-release/releases/tag/v1.19.22 ? I would observe the next test runs and if there are no more failures, we can close this issue.

jochenehret commented 5 months ago

Test has failed again three times in a row: https://concourse.wg-ard.ci.cloudfoundry.org/teams/main/pipelines/cf-deployment/jobs/bbr-run-drats/builds/1348 Can you please check?

jochenehret commented 4 months ago

Here's the full log output from a failed attempt: https://concourse.wg-ard.ci.cloudfoundry.org/teams/main/pipelines/cf-deployment/jobs/bbr-run-drats/builds/1369

selected worker: concourse-worker-8646cc9fc8-jsk25
streaming volume disaster-recovery-acceptance-tests from concourse-worker-8646cc9fc8-jz4k2
Agent pid 19
Identity added: ssh.pem (ssh.pem)
Configured BOSH_ALL_PROXY to ssh+socks5://jumpbox@35.198.161.5:22?private-key=/tmp/build/b7784222/ssh.pem
/tmp/build/b7784222/bbr-binary-release /tmp/build/b7784222
./
./releases/
./releases/bbr-s3-config-validator.README.md
./releases/bbr
./releases/bbr-s3-config-validator
./releases/checksum.sha256
./releases/bbr-mac
/tmp/build/b7784222
/tmp/build/b7784222/src/github.com/cloudfoundry/disaster-recovery-acceptance-tests /tmp/build/b7784222
go: downloading github.com/onsi/ginkgo/v2 v2.19.0
go: downloading github.com/go-task/slim-sprig/v3 v3.0.0
go: downloading golang.org/x/tools v0.21.0
go: downloading github.com/google/pprof v0.0.0-20240424215950-a892ee059fd6
go: downloading github.com/onsi/gomega v1.33.1
go: downloading github.com/go-logr/logr v1.4.1
go: downloading code.cloudfoundry.org/routing-api v0.0.0-20240517181550-f97bdd7c2328
go: downloading golang.org/x/mod v0.19.0
go: downloading golang.org/x/net v0.25.0
go: downloading golang.org/x/sys v0.20.0
go: downloading github.com/google/go-cmp v0.6.0
go: downloading code.cloudfoundry.org/cfhttp/v2 v2.0.0
go: downloading github.com/tedsuo/rata v1.0.0
go: downloading github.com/vito/go-sse v1.0.0
go: downloading github.com/nu7hatch/gouuid v0.0.0-20131221200532-179d4d0c4d8d
go: downloading gopkg.in/yaml.v3 v3.0.1
go: downloading github.com/bmizerany/pat v0.0.0-20210406213842-e4b6760bdd6f
go: downloading golang.org/x/text v0.15.0

Running Suite: DisasterRecoveryAcceptanceTests Suite - /tmp/build/b7784222/src/github.com/cloudfoundry/disaster-recovery-acceptance-tests/acceptance
====================================================================================================================================================
Random Seed: 1720682750

Will run 1 of 1 specs
------------------------------
backing up Cloud Foundry backs up and restores a CF
/tmp/build/b7784222/src/github.com/cloudfoundry/disaster-recovery-acceptance-tests/runner/runner.go:52

CF CLI version:
  cf version 8.7.10+5b7ce3c.2024-04-04

Running test cases:
cf-routing
app-uptime
cf-app
cf-uaa
cf-networking
cf-nfsbroker
cf-smbbroker
cf-credhub

Checking deployment has been set up for test cases...
  STEP: checking if the NFS service is registered @ 07/11/24 07:26:01.059
  Setting API endpoint to https://api.cf.baba-yaga.env.wg-ard.ci.cloudfoundry.org...
  OK

  API endpoint:   https://api.cf.baba-yaga.env.wg-ard.ci.cloudfoundry.org
  API version:    3.169.0

  Not logged in. Use 'cf login' or 'cf login --sso' to log in.
  API endpoint: https://api.cf.baba-yaga.env.wg-ard.ci.cloudfoundry.org

  Authenticating...
  OK

  Use 'cf target' to view or set your target org and space.
  Getting service access for service offering nfs as admin...

  broker: nfsbroker
     offering   plan       access   orgs
     nfs        Existing   none     
  STEP: checking if the SMB service is registered @ 07/11/24 07:26:01.837
  Setting API endpoint to https://api.cf.baba-yaga.env.wg-ard.ci.cloudfoundry.org...
  OK

  API endpoint:   https://api.cf.baba-yaga.env.wg-ard.ci.cloudfoundry.org
  API version:    3.169.0

  Not logged in. Use 'cf login' or 'cf login --sso' to log in.
  API endpoint: https://api.cf.baba-yaga.env.wg-ard.ci.cloudfoundry.org

  Authenticating...
  OK

  Use 'cf target' to view or set your target org and space.
  Getting service access for service offering smb as admin...

  broker: smbbroker
     offering   plan       access   orgs
     smb        Existing   none     
  STEP: setting up the test context @ 07/11/24 07:26:02.43
  STEP: populating state in environment to be backed up @ 07/11/24 07:26:02.442
  STEP: running the BeforeBackup step for cf-routing @ 07/11/24 07:26:02.442
  STEP: Getting CF OAuth Token @ 07/11/24 07:26:02.442
  API endpoint: https://api.cf.baba-yaga.env.wg-ard.ci.cloudfoundry.org

  Authenticating...
  OK

  Targeted org system.

  Select a space:
  1. nfs-broker-space
  2. smb-broker-space

  Space (enter to skip): API endpoint:   https://api.cf.baba-yaga.env.wg-ard.ci.cloudfoundry.org
  API version:    3.169.0
  user:           admin
  org:            system
  space:          No space targeted, use 'cf target -s SPACE'
  STEP: Creating a pre-backup router group backup @ 07/11/24 07:26:03.006
  STEP: running the BeforeBackup step for app-uptime @ 07/11/24 07:26:18.036
  Setting API endpoint to https://api.cf.baba-yaga.env.wg-ard.ci.cloudfoundry.org...
  OK

  API endpoint:   https://api.cf.baba-yaga.env.wg-ard.ci.cloudfoundry.org
  API version:    3.169.0

  Not logged in. Use 'cf login' or 'cf login --sso' to log in.
  API endpoint: https://api.cf.baba-yaga.env.wg-ard.ci.cloudfoundry.org

  Authenticating...
  OK

  Use 'cf target' to view or set your target org and space.
  Creating org acceptance-test-org-85viY2TapHNUxkzPv3eUFA as admin...
  OK

  TIP: Use 'cf target -o "acceptance-test-org-85viY2TapHNUxkzPv3eUFA"' to target new org
  Creating space acceptance-test-space-85viY2TapHNUxkzPv3eUFA in org acceptance-test-org-85viY2TapHNUxkzPv3eUFA as admin...
  OK

  Assigning role SpaceManager to user admin in org acceptance-test-org-85viY2TapHNUxkzPv3eUFA / space acceptance-test-space-85viY2TapHNUxkzPv3eUFA as admin...
  OK

  Assigning role SpaceDeveloper to user admin in org acceptance-test-org-85viY2TapHNUxkzPv3eUFA / space acceptance-test-space-85viY2TapHNUxkzPv3eUFA as admin...
  OK

  TIP: Use 'cf target -o "acceptance-test-org-85viY2TapHNUxkzPv3eUFA" -s "acceptance-test-space-85viY2TapHNUxkzPv3eUFA"' to target new space
  API endpoint:   https://api.cf.baba-yaga.env.wg-ard.ci.cloudfoundry.org
  API version:    3.169.0
  user:           admin
  org:            acceptance-test-org-85viY2TapHNUxkzPv3eUFA
  space:          acceptance-test-space-85viY2TapHNUxkzPv3eUFA
  Pushing app test_app_85viY2TapHNUxkzPv3eUFA to org acceptance-test-org-85viY2TapHNUxkzPv3eUFA / space acceptance-test-space-85viY2TapHNUxkzPv3eUFA as admin...
  Packaging files to upload...
  Uploading files...
 374 B / 374 B [========================================================================================================================================================================================================================================================================================================================================================================================================================================================================================] 100.00% 1s

  Waiting for API to complete processing files...

  Staging app and tracing logs...
     Downloading nodejs_buildpack...
     Downloading go_buildpack...
     Downloading php_buildpack...
     Downloading binary_buildpack...
     Downloading python_buildpack...
     Downloaded binary_buildpack
     Downloading nginx_buildpack...
     Downloaded go_buildpack (5.1M)
     Downloading r_buildpack...
     Downloaded nodejs_buildpack (5.1M)
     Downloading ruby_buildpack...
     Downloaded python_buildpack (5.1M)
     Downloading staticfile_buildpack...
     Downloaded nginx_buildpack (7.5M)
     Downloading java_buildpack...
     Downloaded ruby_buildpack (4.3M)
     Downloading dotnet_core_buildpack...
     Downloaded php_buildpack (7.9M)
     Downloaded java_buildpack (247.4K)
     Downloaded r_buildpack (4M)
     Downloaded staticfile_buildpack (5.4M)
     Downloaded dotnet_core_buildpack (5.3M)
     Cell ba06a74c-baf5-4da6-b81e-cef9bf0fc80d creating container for instance a8aae43f-6753-4061-8743-64523e2f497e
     Security group rules were updated
     Cell ba06a74c-baf5-4da6-b81e-cef9bf0fc80d successfully created container for instance a8aae43f-6753-4061-8743-64523e2f497e
     Downloading app package...
     Downloaded app package (374B)
     -----> Staticfile Buildpack version 1.6.13
     -----> Installing nginx
     Using nginx version 1.25.5
     -----> Installing nginx 1.25.5
     Download [https://buildpacks.cloudfoundry.org/dependencies/nginx-static/nginx-static_1.25.5_linux_x64_cflinuxfs4_e1fcd6a3.tgz]
     -----> Root folder /tmp/app
     -----> Copying project files into public
     -----> Configuring nginx
     Exit status 0
     Uploading droplet, build artifacts cache...
     Uploading droplet...
     Uploading build artifacts cache...
     Uploaded build artifacts cache (2.3M)

  Waiting for app test_app_85viY2TapHNUxkzPv3eUFA to start...

  Instances starting...
  Instances starting...
  Instances starting...

  name:              test_app_85viY2TapHNUxkzPv3eUFA
  requested state:   started
  routes:            test_app_85viY2TapHNUxkzPv3eUFA.cf.baba-yaga.env.wg-ard.ci.cloudfoundry.org
  last uploaded:     Thu 11 Jul 07:26:35 UTC 2024
  stack:             cflinuxfs4
  buildpacks:        
    name                   version   detect output   buildpack name
    staticfile_buildpack   1.6.13    staticfile      staticfile

  type:            web
  sidecars:        
  instances:       1/1
  memory usage:    1024M
  start command:   $HOME/boot.sh
       state     since                  cpu    memory     disk       logging        details
  #0   running   2024-07-11T07:26:43Z   0.0%   0B of 0B   0B of 0B   0B/s of 0B/s   
  STEP: checking the app stays up @ 07/11/24 07:26:45.399
  Showing health and status for app test_app_85viY2TapHNUxkzPv3eUFA in org acceptance-test-org-85viY2TapHNUxkzPv3eUFA / space acceptance-test-space-85viY2TapHNUxkzPv3eUFA as admin...

  name:              test_app_85viY2TapHNUxkzPv3eUFA
  requested state:   started
  routes:            test_app_85viY2TapHNUxkzPv3eUFA.cf.baba-yaga.env.wg-ard.ci.cloudfoundry.org
  last uploaded:     Thu 11 Jul 07:26:35 UTC 2024
  stack:             cflinuxfs4
  buildpacks:        
    name                   version   detect output   buildpack name
    staticfile_buildpack   1.6.13    staticfile      staticfile

  type:           web
  sidecars:       
  instances:      1/1
  memory usage:   1024M
       state     since                  cpu    memory     disk       logging        details
  #0   running   2024-07-11T07:26:42Z   0.0%   0B of 0B   0B of 0B   0B/s of 0B/s   
  STEP: running the BeforeBackup step for cf-app @ 07/11/24 07:26:45.685
  STEP: creating new orgs and spaces @ 07/11/24 07:26:45.685
  Setting API endpoint to https://api.cf.baba-yaga.env.wg-ard.ci.cloudfoundry.org...
  OK

  API endpoint:   https://api.cf.baba-yaga.env.wg-ard.ci.cloudfoundry.org
  API version:    3.169.0

  Not logged in. Use 'cf login' or 'cf login --sso' to log in.
  API endpoint: https://api.cf.baba-yaga.env.wg-ard.ci.cloudfoundry.org

  Authenticating...
  OK

  Use 'cf target' to view or set your target org and space.
  Creating org acceptance-test-org-Jj7qDp5mLE0PN8bFqWNJlw as admin...
  OK

  TIP: Use 'cf target -o "acceptance-test-org-Jj7qDp5mLE0PN8bFqWNJlw"' to target new org
  Creating space acceptance-test-space-Jj7qDp5mLE0PN8bFqWNJlw in org acceptance-test-org-Jj7qDp5mLE0PN8bFqWNJlw as admin...
  OK

  Assigning role SpaceManager to user admin in org acceptance-test-org-Jj7qDp5mLE0PN8bFqWNJlw / space acceptance-test-space-Jj7qDp5mLE0PN8bFqWNJlw as admin...
  OK

  Assigning role SpaceDeveloper to user admin in org acceptance-test-org-Jj7qDp5mLE0PN8bFqWNJlw / space acceptance-test-space-Jj7qDp5mLE0PN8bFqWNJlw as admin...
  OK

  TIP: Use 'cf target -o "acceptance-test-org-Jj7qDp5mLE0PN8bFqWNJlw" -s "acceptance-test-space-Jj7qDp5mLE0PN8bFqWNJlw"' to target new space
  API endpoint:   https://api.cf.baba-yaga.env.wg-ard.ci.cloudfoundry.org
  API version:    3.169.0
  user:           admin
  org:            acceptance-test-org-Jj7qDp5mLE0PN8bFqWNJlw
  space:          acceptance-test-space-Jj7qDp5mLE0PN8bFqWNJlw
  Pushing app test_app_Jj7qDp5mLE0PN8bFqWNJlw to org acceptance-test-org-Jj7qDp5mLE0PN8bFqWNJlw / space acceptance-test-space-Jj7qDp5mLE0PN8bFqWNJlw as admin...
  Packaging files to upload...
  Uploading files...
 374 B / 374 B [========================================================================================================================================================================================================================================================================================================================================================================================================================================================================================] 100.00% 1s

  Waiting for API to complete processing files...

  Staging app and tracing logs...
     Downloading binary_buildpack...
     Downloading nodejs_buildpack...
     Downloading go_buildpack...
     Downloading staticfile_buildpack...
     Downloading java_buildpack...
     Downloaded java_buildpack (247.4K)
     Downloading ruby_buildpack...
     Downloaded go_buildpack (5.1M)
     Downloading dotnet_core_buildpack...
     Downloaded binary_buildpack (6.3M)
     Downloading nginx_buildpack...
     Downloaded staticfile_buildpack (5.4M)
     Downloading python_buildpack...
     Downloaded ruby_buildpack (4.3M)
     Downloading php_buildpack...
     Downloaded nodejs_buildpack (5.1M)
     Downloading r_buildpack...
     Downloaded dotnet_core_buildpack (5.3M)
     Downloaded r_buildpack (4M)
     Downloaded nginx_buildpack (7.5M)
     Downloaded python_buildpack (5.1M)
     Downloaded php_buildpack (7.9M)
     Cell f8d91ecd-e4c2-4594-84e5-6099e14f053c creating container for instance 05e3f7ca-c618-4a3f-b1ad-a20d3dea7630
     Security group rules were updated
     Cell f8d91ecd-e4c2-4594-84e5-6099e14f053c successfully created container for instance 05e3f7ca-c618-4a3f-b1ad-a20d3dea7630
     Downloading app package...
     Downloaded app package (374B)
     -----> Staticfile Buildpack version 1.6.13
     -----> Installing nginx
     Using nginx version 1.25.5
     -----> Installing nginx 1.25.5
     Download [https://buildpacks.cloudfoundry.org/dependencies/nginx-static/nginx-static_1.25.5_linux_x64_cflinuxfs4_e1fcd6a3.tgz]
     -----> Root folder /tmp/app
     -----> Copying project files into public
     -----> Configuring nginx
     Exit status 0
     Uploading droplet, build artifacts cache...
     Uploading droplet...
     Uploading build artifacts cache...
     Uploaded build artifacts cache (2.3M)
     Uploaded droplet (2.3M)
     Uploading complete
     Cell f8d91ecd-e4c2-4594-84e5-6099e14f053c stopping instance 05e3f7ca-c618-4a3f-b1ad-a20d3dea7630
     Cell f8d91ecd-e4c2-4594-84e5-6099e14f053c destroying container for instance 05e3f7ca-c618-4a3f-b1ad-a20d3dea7630
     Cell f8d91ecd-e4c2-4594-84e5-6099e14f053c successfully destroyed container for instance 05e3f7ca-c618-4a3f-b1ad-a20d3dea7630

  Waiting for app test_app_Jj7qDp5mLE0PN8bFqWNJlw to start...

  Instances starting...
  Instances starting...
  Instances starting...

  name:              test_app_Jj7qDp5mLE0PN8bFqWNJlw
  requested state:   started
  routes:            test_app_Jj7qDp5mLE0PN8bFqWNJlw.cf.baba-yaga.env.wg-ard.ci.cloudfoundry.org
  last uploaded:     Thu 11 Jul 07:27:04 UTC 2024
  stack:             cflinuxfs4
  buildpacks:        
    name                   version   detect output   buildpack name
    staticfile_buildpack   1.6.13    staticfile      staticfile

  type:            web
  sidecars:        
  instances:       1/1
  memory usage:    1024M
  start command:   $HOME/boot.sh
       state     since                  cpu    memory     disk       logging        details
  #0   running   2024-07-11T07:27:13Z   0.0%   0B of 1G   0B of 1G   0B/s of 0B/s   
  Pushing app stopped_test_app_Jj7qDp5mLE0PN8bFqWNJlw to org acceptance-test-org-Jj7qDp5mLE0PN8bFqWNJlw / space acceptance-test-space-Jj7qDp5mLE0PN8bFqWNJlw as admin...
  Packaging files to upload...
  Uploading files...
 374 B / 374 B [========================================================================================================================================================================================================================================================================================================================================================================================================================================================================================] 100.00% 1s

  Waiting for API to complete processing files...

  Staging app and tracing logs...
     Downloading binary_buildpack...
     Downloading dotnet_core_buildpack...
     Downloading nodejs_buildpack...
     Downloading ruby_buildpack...
     Downloading staticfile_buildpack...
     Downloaded ruby_buildpack (4.3M)
     Downloading java_buildpack...
     Downloaded nodejs_buildpack (5.1M)
     Downloading nginx_buildpack...
     Downloaded dotnet_core_buildpack (5.3M)
     Downloading go_buildpack...
     Downloaded staticfile_buildpack (5.4M)
     Downloading python_buildpack...
     Downloaded java_buildpack (247.4K)
     Downloading php_buildpack...
     Downloaded binary_buildpack (6.3M)
     Downloading r_buildpack...
     Downloaded go_buildpack (5.1M)
     Downloaded python_buildpack (5.1M)
     Downloaded r_buildpack (4M)
     Downloaded nginx_buildpack (7.5M)
     Downloaded php_buildpack (7.9M)
     Cell a0c47c10-0789-4519-bddb-987d9449778e creating container for instance 129cecb9-ec6a-48f6-9774-d5af090fd4fc
     Security group rules were updated
     Cell a0c47c10-0789-4519-bddb-987d9449778e successfully created container for instance 129cecb9-ec6a-48f6-9774-d5af090fd4fc
     Downloading app package...
     Downloaded app package (374B)
     -----> Staticfile Buildpack version 1.6.13
     -----> Installing nginx
     Using nginx version 1.25.5
     -----> Installing nginx 1.25.5
     Download [https://buildpacks.cloudfoundry.org/dependencies/nginx-static/nginx-static_1.25.5_linux_x64_cflinuxfs4_e1fcd6a3.tgz]
     -----> Root folder /tmp/app
     -----> Copying project files into public
     -----> Configuring nginx
     Exit status 0
     Uploading droplet, build artifacts cache...
     Uploading droplet...
     Uploading build artifacts cache...
     Uploaded build artifacts cache (2.3M)
     Uploaded droplet (2.3M)
     Uploading complete

  Waiting for app stopped_test_app_Jj7qDp5mLE0PN8bFqWNJlw to start...

  Instances starting...
  Instances starting...
  Instances starting...

  name:              stopped_test_app_Jj7qDp5mLE0PN8bFqWNJlw
  requested state:   started
  routes:            stopped_test_app_Jj7qDp5mLE0PN8bFqWNJlw.cf.baba-yaga.env.wg-ard.ci.cloudfoundry.org
  last uploaded:     Thu 11 Jul 07:27:28 UTC 2024
  stack:             cflinuxfs4
  buildpacks:        
    name                   version   detect output   buildpack name
    staticfile_buildpack   1.6.13    staticfile      staticfile

  type:            web
  sidecars:        
  instances:       1/1
  memory usage:    1024M
  start command:   $HOME/boot.sh
       state     since                  cpu    memory     disk       logging        details
  #0   running   2024-07-11T07:27:39Z   0.0%   0B of 0B   0B of 0B   0B/s of 0B/s   
  Pushing app running_test_app_Jj7qDp5mLE0PN8bFqWNJlw to org acceptance-test-org-Jj7qDp5mLE0PN8bFqWNJlw / space acceptance-test-space-Jj7qDp5mLE0PN8bFqWNJlw as admin...
  Packaging files to upload...
  Uploading files...
 374 B / 374 B [========================================================================================================================================================================================================================================================================================================================================================================================================================================================================================] 100.00% 1s

  Waiting for API to complete processing files...

  Staging app and tracing logs...
     Downloading binary_buildpack...
     Downloading staticfile_buildpack...
     Downloading java_buildpack...
     Downloading ruby_buildpack...
     Downloading dotnet_core_buildpack...
     Downloaded staticfile_buildpack
     Downloaded dotnet_core_buildpack
     Downloading nodejs_buildpack...
     Downloading go_buildpack...
     Downloaded ruby_buildpack
     Downloading python_buildpack...
     Downloaded java_buildpack
     Downloading php_buildpack...
     Downloaded binary_buildpack
     Downloading nginx_buildpack...
     Downloaded nodejs_buildpack
     Downloading r_buildpack...
     Downloaded php_buildpack
     Downloaded go_buildpack
     Downloaded python_buildpack
     Downloaded nginx_buildpack
     Downloaded r_buildpack
     Cell ba06a74c-baf5-4da6-b81e-cef9bf0fc80d creating container for instance 18eba071-0cad-4ecb-a2a9-1922ebac88c4
     Security group rules were updated
     Cell ba06a74c-baf5-4da6-b81e-cef9bf0fc80d successfully created container for instance 18eba071-0cad-4ecb-a2a9-1922ebac88c4
     Downloading app package...
     Downloaded app package (374B)
     -----> Staticfile Buildpack version 1.6.13
     -----> Installing nginx
     Using nginx version 1.25.5
     -----> Installing nginx 1.25.5
     Download [https://buildpacks.cloudfoundry.org/dependencies/nginx-static/nginx-static_1.25.5_linux_x64_cflinuxfs4_e1fcd6a3.tgz]
     -----> Root folder /tmp/app
     -----> Copying project files into public
     -----> Configuring nginx
     Exit status 0
     Uploading droplet, build artifacts cache...
     Uploading droplet...
     Uploading build artifacts cache...
     Uploaded build artifacts cache (2.3M)
     Uploaded droplet (2.3M)
     Uploading complete

  Waiting for app running_test_app_Jj7qDp5mLE0PN8bFqWNJlw to start...

  Instances starting...
  Instances starting...
  Instances starting...

  name:              running_test_app_Jj7qDp5mLE0PN8bFqWNJlw
  requested state:   started
  routes:            running_test_app_Jj7qDp5mLE0PN8bFqWNJlw.cf.baba-yaga.env.wg-ard.ci.cloudfoundry.org
  last uploaded:     Thu 11 Jul 07:27:51 UTC 2024
  stack:             cflinuxfs4
  buildpacks:        
    name                   version   detect output   buildpack name
    staticfile_buildpack   1.6.13    staticfile      staticfile

  type:            web
  sidecars:        
  instances:       1/1
  memory usage:    1024M
  start command:   $HOME/boot.sh
       state     since                  cpu    memory     disk       logging        details
  #0   running   2024-07-11T07:27:59Z   0.0%   0B of 0B   0B of 0B   0B/s of 0B/s   
  Setting env variable MY_SPECIAL_VAR for app test_app_Jj7qDp5mLE0PN8bFqWNJlw in org acceptance-test-org-Jj7qDp5mLE0PN8bFqWNJlw / space acceptance-test-space-Jj7qDp5mLE0PN8bFqWNJlw as admin...
  OK

  TIP: Use 'cf restage test_app_Jj7qDp5mLE0PN8bFqWNJlw' to ensure your env variable changes take effect.
  Setting env variable MY_STOPPED_SPECIAL_VAR for app stopped_test_app_Jj7qDp5mLE0PN8bFqWNJlw in org acceptance-test-org-Jj7qDp5mLE0PN8bFqWNJlw / space acceptance-test-space-Jj7qDp5mLE0PN8bFqWNJlw as admin...
  OK

  TIP: Use 'cf restage stopped_test_app_Jj7qDp5mLE0PN8bFqWNJlw' to ensure your env variable changes take effect.
  Setting env variable MY_RUNNING_SPECIAL_VAR for app running_test_app_Jj7qDp5mLE0PN8bFqWNJlw in org acceptance-test-org-Jj7qDp5mLE0PN8bFqWNJlw / space acceptance-test-space-Jj7qDp5mLE0PN8bFqWNJlw as admin...
  OK

  TIP: Use 'cf restage running_test_app_Jj7qDp5mLE0PN8bFqWNJlw' to ensure your env variable changes take effect.
  Stopping app stopped_test_app_Jj7qDp5mLE0PN8bFqWNJlw in org acceptance-test-org-Jj7qDp5mLE0PN8bFqWNJlw / space acceptance-test-space-Jj7qDp5mLE0PN8bFqWNJlw as admin...
  OK

  STEP: running the BeforeBackup step for cf-uaa @ 07/11/24 07:28:02.094
  STEP: we create a user and can login @ 07/11/24 07:28:02.094
  Setting API endpoint to https://api.cf.baba-yaga.env.wg-ard.ci.cloudfoundry.org...
  OK

  API endpoint:   https://api.cf.baba-yaga.env.wg-ard.ci.cloudfoundry.org
  API version:    3.169.0

  Not logged in. Use 'cf login' or 'cf login --sso' to log in.
  API endpoint: https://api.cf.baba-yaga.env.wg-ard.ci.cloudfoundry.org

  Authenticating...
  OK

  Use 'cf target' to view or set your target org and space.
  Creating user uaa-test-user-DpbrpcUGcTO9GICuROHrYA...
  OK

  TIP: Assign roles with 'cf set-org-role' and 'cf set-space-role'.
  Setting API endpoint to https://api.cf.baba-yaga.env.wg-ard.ci.cloudfoundry.org...
  OK

  API endpoint:   https://api.cf.baba-yaga.env.wg-ard.ci.cloudfoundry.org
  API version:    3.169.0

  Not logged in. Use 'cf login' or 'cf login --sso' to log in.
  API endpoint: https://api.cf.baba-yaga.env.wg-ard.ci.cloudfoundry.org

  Authenticating...
  OK

  Use 'cf target' to view or set your target org and space.
  Logging out uaa-test-user-DpbrpcUGcTO9GICuROHrYA...
  OK

  STEP: running the BeforeBackup step for cf-networking @ 07/11/24 07:28:03.33
  STEP: creating new orgs and spaces @ 07/11/24 07:28:03.331
  Setting API endpoint to https://api.cf.baba-yaga.env.wg-ard.ci.cloudfoundry.org...
  OK

  API endpoint:   https://api.cf.baba-yaga.env.wg-ard.ci.cloudfoundry.org
  API version:    3.169.0

  Not logged in. Use 'cf login' or 'cf login --sso' to log in.
  API endpoint: https://api.cf.baba-yaga.env.wg-ard.ci.cloudfoundry.org

  Authenticating...
  OK

  Use 'cf target' to view or set your target org and space.
  Creating org acceptance-test-org-A7sCNPOovD1MyNkgu4mrkA as admin...
  OK

  TIP: Use 'cf target -o "acceptance-test-org-A7sCNPOovD1MyNkgu4mrkA"' to target new org
  Creating space acceptance-test-space-A7sCNPOovD1MyNkgu4mrkA in org acceptance-test-org-A7sCNPOovD1MyNkgu4mrkA as admin...
  OK

  Assigning role SpaceManager to user admin in org acceptance-test-org-A7sCNPOovD1MyNkgu4mrkA / space acceptance-test-space-A7sCNPOovD1MyNkgu4mrkA as admin...
  OK

  Assigning role SpaceDeveloper to user admin in org acceptance-test-org-A7sCNPOovD1MyNkgu4mrkA / space acceptance-test-space-A7sCNPOovD1MyNkgu4mrkA as admin...
  OK

  TIP: Use 'cf target -o "acceptance-test-org-A7sCNPOovD1MyNkgu4mrkA" -s "acceptance-test-space-A7sCNPOovD1MyNkgu4mrkA"' to target new space
  API endpoint:   https://api.cf.baba-yaga.env.wg-ard.ci.cloudfoundry.org
  API version:    3.169.0
  user:           admin
  org:            acceptance-test-org-A7sCNPOovD1MyNkgu4mrkA
  space:          acceptance-test-space-A7sCNPOovD1MyNkgu4mrkA
  Pushing app test_app_A7sCNPOovD1MyNkgu4mrkA to org acceptance-test-org-A7sCNPOovD1MyNkgu4mrkA / space acceptance-test-space-A7sCNPOovD1MyNkgu4mrkA as admin...
  Packaging files to upload...
  Uploading files...
 374 B / 374 B [========================================================================================================================================================================================================================================================================================================================================================================================================================================================================================] 100.00% 1s

  Waiting for API to complete processing files...

  Staging app and tracing logs...
     Downloading binary_buildpack...
     Downloading java_buildpack...
     Downloading nodejs_buildpack...
     Downloading staticfile_buildpack...
     Downloading go_buildpack...
     Downloaded binary_buildpack
     Downloading python_buildpack...
     Downloaded nodejs_buildpack
     Downloading php_buildpack...
     Downloaded java_buildpack
     Downloading nginx_buildpack...
     Downloaded staticfile_buildpack
     Downloading r_buildpack...
     Downloaded go_buildpack
     Downloading ruby_buildpack...
     Downloaded php_buildpack
     Downloading dotnet_core_buildpack...
     Downloaded python_buildpack
     Downloaded r_buildpack
     Downloaded nginx_buildpack
     Downloaded ruby_buildpack
     Downloaded dotnet_core_buildpack
     Cell a0c47c10-0789-4519-bddb-987d9449778e creating container for instance e403de08-5e59-4660-b8a9-0745f63da3e7
     Security group rules were updated
     Cell a0c47c10-0789-4519-bddb-987d9449778e successfully created container for instance e403de08-5e59-4660-b8a9-0745f63da3e7
     Downloading app package...
     Downloaded app package (374B)
     -----> Staticfile Buildpack version 1.6.13
     -----> Installing nginx
     Using nginx version 1.25.5
     -----> Installing nginx 1.25.5
     Download [https://buildpacks.cloudfoundry.org/dependencies/nginx-static/nginx-static_1.25.5_linux_x64_cflinuxfs4_e1fcd6a3.tgz]
     -----> Root folder /tmp/app
     -----> Copying project files into public
     -----> Configuring nginx
     Exit status 0
     Uploading droplet, build artifacts cache...
     Uploading droplet...
     Uploading build artifacts cache...
     Uploaded build artifacts cache (2.3M)
     Uploaded droplet (2.3M)
     Uploading complete

  Waiting for app test_app_A7sCNPOovD1MyNkgu4mrkA to start...

  Instances starting...
  Instances starting...
  Instances starting...

  name:              test_app_A7sCNPOovD1MyNkgu4mrkA
  requested state:   started
  routes:            test_app_A7sCNPOovD1MyNkgu4mrkA.cf.baba-yaga.env.wg-ard.ci.cloudfoundry.org
  last uploaded:     Thu 11 Jul 07:28:16 UTC 2024
  stack:             cflinuxfs4
  buildpacks:        
    name                   version   detect output   buildpack name
    staticfile_buildpack   1.6.13    staticfile      staticfile

  type:            web
  sidecars:        
  instances:       1/1
  memory usage:    1024M
  start command:   $HOME/boot.sh
       state     since                  cpu    memory     disk       logging        details
  #0   running   2024-07-11T07:28:25Z   0.0%   0B of 0B   0B of 0B   0B/s of 0B/s   
  Adding network policy from app test_app_A7sCNPOovD1MyNkgu4mrkA to app test_app_A7sCNPOovD1MyNkgu4mrkA in org acceptance-test-org-A7sCNPOovD1MyNkgu4mrkA / space acceptance-test-space-A7sCNPOovD1MyNkgu4mrkA as admin...
  OK

  STEP: running the BeforeBackup step for cf-nfsbroker @ 07/11/24 07:28:27.439
  STEP: checking the service name and plane name are provided @ 07/11/24 07:28:27.439
  STEP: creating an NFS service broker and service instance @ 07/11/24 07:28:27.439
  Setting API endpoint to https://api.cf.baba-yaga.env.wg-ard.ci.cloudfoundry.org...
  OK

  API endpoint:   https://api.cf.baba-yaga.env.wg-ard.ci.cloudfoundry.org
  API version:    3.169.0

  Not logged in. Use 'cf login' or 'cf login --sso' to log in.
  API endpoint: https://api.cf.baba-yaga.env.wg-ard.ci.cloudfoundry.org

  Authenticating...
  OK

  Select an org:
  1. acceptance-test-org-85viY2TapHNUxkzPv3eUFA
  2. acceptance-test-org-A7sCNPOovD1MyNkgu4mrkA
  3. acceptance-test-org-Jj7qDp5mLE0PN8bFqWNJlw
  4. system

  Org (enter to skip): API endpoint:   https://api.cf.baba-yaga.env.wg-ard.ci.cloudfoundry.org
  API version:    3.169.0
  user:           admin
  No org or space targeted, use 'cf target -o ORG -s SPACE'
  Creating org acceptance-test-org-RnUPP5gAFpC-Cl-0aCAfhg as admin...
  OK

  TIP: Use 'cf target -o "acceptance-test-org-RnUPP5gAFpC-Cl-0aCAfhg"' to target new org
  Creating space acceptance-test-space-RnUPP5gAFpC-Cl-0aCAfhg in org acceptance-test-org-RnUPP5gAFpC-Cl-0aCAfhg as admin...
  OK

  Assigning role SpaceManager to user admin in org acceptance-test-org-RnUPP5gAFpC-Cl-0aCAfhg / space acceptance-test-space-RnUPP5gAFpC-Cl-0aCAfhg as admin...
  OK

  Assigning role SpaceDeveloper to user admin in org acceptance-test-org-RnUPP5gAFpC-Cl-0aCAfhg / space acceptance-test-space-RnUPP5gAFpC-Cl-0aCAfhg as admin...
  OK

  TIP: Use 'cf target -o "acceptance-test-org-RnUPP5gAFpC-Cl-0aCAfhg" -s "acceptance-test-space-RnUPP5gAFpC-Cl-0aCAfhg"' to target new space
  API endpoint:   https://api.cf.baba-yaga.env.wg-ard.ci.cloudfoundry.org
  API version:    3.169.0
  user:           admin
  org:            acceptance-test-org-RnUPP5gAFpC-Cl-0aCAfhg
  space:          acceptance-test-space-RnUPP5gAFpC-Cl-0aCAfhg
  Enabling feature flag diego_docker as admin...
  OK

  Pushing app dratsApp to org acceptance-test-org-RnUPP5gAFpC-Cl-0aCAfhg / space acceptance-test-space-RnUPP5gAFpC-Cl-0aCAfhg as admin...

  name:              dratsApp
  requested state:   stopped
  routes:            dratsApp-timely-echidna-uj.cf.baba-yaga.env.wg-ard.ci.cloudfoundry.org
  last uploaded:     
  stack:             
  docker image:      

  type:           web
  sidecars:       
  instances:      0/1
  memory usage:   1024M
       state   since                  cpu    memory     disk       logging        details
  #0   down    2024-07-11T07:28:35Z   0.0%   0B of 0B   0B of 0B   0B/s of 0B/s   
  Enabling access to all plans of service offering nfs for org acceptance-test-org-RnUPP5gAFpC-Cl-0aCAfhg as admin...
  OK

  Creating service instance service-instance-RnUPP5gAFpC-Cl-0aCAfhg in org acceptance-test-org-RnUPP5gAFpC-Cl-0aCAfhg / space acceptance-test-space-RnUPP5gAFpC-Cl-0aCAfhg as admin...

  Service instance service-instance-RnUPP5gAFpC-Cl-0aCAfhg created.
  OK

  STEP: running the BeforeBackup step for cf-smbbroker @ 07/11/24 07:28:44.8
  STEP: checking the service name and plane name are provided @ 07/11/24 07:28:44.8
  STEP: creating an SMB service broker and service instance @ 07/11/24 07:28:44.8
  Setting API endpoint to https://api.cf.baba-yaga.env.wg-ard.ci.cloudfoundry.org...
  OK

  API endpoint:   https://api.cf.baba-yaga.env.wg-ard.ci.cloudfoundry.org
  API version:    3.169.0

  Not logged in. Use 'cf login' or 'cf login --sso' to log in.
  API endpoint: https://api.cf.baba-yaga.env.wg-ard.ci.cloudfoundry.org

  Authenticating...
  OK

  Select an org:
  1. acceptance-test-org-85viY2TapHNUxkzPv3eUFA
  2. acceptance-test-org-A7sCNPOovD1MyNkgu4mrkA
  3. acceptance-test-org-Jj7qDp5mLE0PN8bFqWNJlw
  4. acceptance-test-org-RnUPP5gAFpC-Cl-0aCAfhg
  5. system

  Org (enter to skip): API endpoint:   https://api.cf.baba-yaga.env.wg-ard.ci.cloudfoundry.org
  API version:    3.169.0
  user:           admin
  No org or space targeted, use 'cf target -o ORG -s SPACE'
  Creating org acceptance-test-org-GPIym_GoApw4RLLohpnq5w as admin...
  OK

  TIP: Use 'cf target -o "acceptance-test-org-GPIym_GoApw4RLLohpnq5w"' to target new org
  Creating space acceptance-test-space-GPIym_GoApw4RLLohpnq5w in org acceptance-test-org-GPIym_GoApw4RLLohpnq5w as admin...
  OK

  Assigning role SpaceManager to user admin in org acceptance-test-org-GPIym_GoApw4RLLohpnq5w / space acceptance-test-space-GPIym_GoApw4RLLohpnq5w as admin...
  OK

  Assigning role SpaceDeveloper to user admin in org acceptance-test-org-GPIym_GoApw4RLLohpnq5w / space acceptance-test-space-GPIym_GoApw4RLLohpnq5w as admin...
  OK

  TIP: Use 'cf target -o "acceptance-test-org-GPIym_GoApw4RLLohpnq5w" -s "acceptance-test-space-GPIym_GoApw4RLLohpnq5w"' to target new space
  API endpoint:   https://api.cf.baba-yaga.env.wg-ard.ci.cloudfoundry.org
  API version:    3.169.0
  user:           admin
  org:            acceptance-test-org-GPIym_GoApw4RLLohpnq5w
  space:          acceptance-test-space-GPIym_GoApw4RLLohpnq5w
  Enabling feature flag diego_docker as admin...
  OK

  Pushing app dratsApp to org acceptance-test-org-GPIym_GoApw4RLLohpnq5w / space acceptance-test-space-GPIym_GoApw4RLLohpnq5w as admin...

  name:              dratsApp
  requested state:   stopped
  routes:            dratsApp-boring-gazelle-qw.cf.baba-yaga.env.wg-ard.ci.cloudfoundry.org
  last uploaded:     
  stack:             
  docker image:      

  type:           web
  sidecars:       
  instances:      0/1
  memory usage:   1024M
       state   since                  cpu    memory     disk       logging        details
  #0   down    2024-07-11T07:28:49Z   0.0%   0B of 0B   0B of 0B   0B/s of 0B/s   
  Enabling access to all plans of service offering smb for org acceptance-test-org-GPIym_GoApw4RLLohpnq5w as admin...
  OK

  Creating service instance service-instance-GPIym_GoApw4RLLohpnq5w in org acceptance-test-org-GPIym_GoApw4RLLohpnq5w / space acceptance-test-space-GPIym_GoApw4RLLohpnq5w as admin...

  Service instance service-instance-GPIym_GoApw4RLLohpnq5w created.
  OK

  STEP: running the BeforeBackup step for cf-credhub @ 07/11/24 07:28:58.892
  Setting API endpoint to https://api.cf.baba-yaga.env.wg-ard.ci.cloudfoundry.org...
  OK

  API endpoint:   https://api.cf.baba-yaga.env.wg-ard.ci.cloudfoundry.org
  API version:    3.169.0

  Not logged in. Use 'cf login' or 'cf login --sso' to log in.
  API endpoint: https://api.cf.baba-yaga.env.wg-ard.ci.cloudfoundry.org

  Authenticating...
  OK

  Use 'cf target' to view or set your target org and space.
  Creating org acceptance-test-org-EFh0TbTJ3zUAjZdSBQZBqA as admin...
  OK

  TIP: Use 'cf target -o "acceptance-test-org-EFh0TbTJ3zUAjZdSBQZBqA"' to target new org
  Creating space acceptance-test-space-EFh0TbTJ3zUAjZdSBQZBqA in org acceptance-test-org-EFh0TbTJ3zUAjZdSBQZBqA as admin...
  OK

  Assigning role SpaceManager to user admin in org acceptance-test-org-EFh0TbTJ3zUAjZdSBQZBqA / space acceptance-test-space-EFh0TbTJ3zUAjZdSBQZBqA as admin...
  OK

  Assigning role SpaceDeveloper to user admin in org acceptance-test-org-EFh0TbTJ3zUAjZdSBQZBqA / space acceptance-test-space-EFh0TbTJ3zUAjZdSBQZBqA as admin...
  OK

  TIP: Use 'cf target -o "acceptance-test-org-EFh0TbTJ3zUAjZdSBQZBqA" -s "acceptance-test-space-EFh0TbTJ3zUAjZdSBQZBqA"' to target new space
  API endpoint:   https://api.cf.baba-yaga.env.wg-ard.ci.cloudfoundry.org
  API version:    3.169.0
  user:           admin
  org:            acceptance-test-org-EFh0TbTJ3zUAjZdSBQZBqA
  space:          acceptance-test-space-EFh0TbTJ3zUAjZdSBQZBqA
  Pushing app appEFh0TbTJ3zUAjZdSBQZBqA to org acceptance-test-org-EFh0TbTJ3zUAjZdSBQZBqA / space acceptance-test-space-EFh0TbTJ3zUAjZdSBQZBqA as admin...
  Applying manifest file /tmp/build/b7784222/src/github.com/cloudfoundry/disaster-recovery-acceptance-tests/fixtures/credhub-test-app/manifest.yml...

  Updating with these attributes...
    ---
    applications:
  + - name: appEFh0TbTJ3zUAjZdSBQZBqA
      path: /tmp/build/b7784222/src/github.com/cloudfoundry/disaster-recovery-acceptance-tests/fixtures/credhub-test-app
  +   default-route: true
  +   buildpacks:
  +   - go_buildpack
  +   env:
  +     GOPACKAGENAME: server
  Manifest applied
  Packaging files to upload...
  Uploading files...
 284.32 KiB / 284.32 KiB [==============================================================================================================================================================================================================================================================================================================================================================================================================================================================================] 100.00% 1s

  Waiting for API to complete processing files...

  name:              appEFh0TbTJ3zUAjZdSBQZBqA
  requested state:   stopped
  routes:            appEFh0TbTJ3zUAjZdSBQZBqA.cf.baba-yaga.env.wg-ard.ci.cloudfoundry.org
  last uploaded:     
  stack:             
  buildpacks:        

  type:           web
  sidecars:       
  instances:      0/1
  memory usage:   1024M
       state   since                  cpu    memory     disk       logging        details
  #0   down    2024-07-11T07:29:12Z   0.0%   0B of 0B   0B of 0B   0B/s of 0B/s   
  Starting app appEFh0TbTJ3zUAjZdSBQZBqA in org acceptance-test-org-EFh0TbTJ3zUAjZdSBQZBqA / space acceptance-test-space-EFh0TbTJ3zUAjZdSBQZBqA as admin...

  Staging app and tracing logs...
     Downloading go_buildpack...
     Downloaded go_buildpack
     Cell f8d91ecd-e4c2-4594-84e5-6099e14f053c creating container for instance 4a8e84d2-75bb-4dce-9319-bee0dbfa96d1
     Security group rules were updated
     Cell f8d91ecd-e4c2-4594-84e5-6099e14f053c successfully created container for instance 4a8e84d2-75bb-4dce-9319-bee0dbfa96d1
     Downloading app package...
     Downloaded app package (284.3K)
     -----> Go Buildpack version 1.10.19
     -----> Installing godep 80
     Download [https://buildpacks.cloudfoundry.org/dependencies/godep/godep_80_linux_x64_cflinuxfs4_20fea317.tgz]
     -----> Installing glide 0.13.3
     Download [https://buildpacks.cloudfoundry.org/dependencies/glide/glide_0.13.3_linux_x64_cflinuxfs4_be64c2ea.tgz]
     -----> Installing dep 0.5.4
     Download [https://buildpacks.cloudfoundry.org/dependencies/dep/dep_0.5.4_linux_x64_cflinuxfs4_a4d7f7ea.tgz]
     -----> Installing go 1.21.10
     Download [https://buildpacks.cloudfoundry.org/dependencies/go/go_1.21.10_linux_x64_cflinuxfs4_2cfb2d9f.tgz]
     **WARNING** Installing package '.' (default)
     -----> Running: go install -tags cloudfoundry -buildmode pie .
     Exit status 0
     Uploading droplet, build artifacts cache...
     Uploading droplet...
     Uploading build artifacts cache...
     Uploaded build artifacts cache (101.5M)
     Uploaded droplet (5.2M)
     Uploading complete
     Cell f8d91ecd-e4c2-4594-84e5-6099e14f053c stopping instance 4a8e84d2-75bb-4dce-9319-bee0dbfa96d1
     Cell f8d91ecd-e4c2-4594-84e5-6099e14f053c destroying container for instance 4a8e84d2-75bb-4dce-9319-bee0dbfa96d1

  Starting app appEFh0TbTJ3zUAjZdSBQZBqA in org acceptance-test-org-EFh0TbTJ3zUAjZdSBQZBqA / space acceptance-test-space-EFh0TbTJ3zUAjZdSBQZBqA as admin...

  Waiting for app to start...

  Instances starting...
  Instances starting...
  Instances starting...

  name:              appEFh0TbTJ3zUAjZdSBQZBqA
  requested state:   started
  routes:            appEFh0TbTJ3zUAjZdSBQZBqA.cf.baba-yaga.env.wg-ard.ci.cloudfoundry.org
  last uploaded:     Thu 11 Jul 07:30:30 UTC 2024
  stack:             cflinuxfs4
  buildpacks:        
    name           version   detect output   buildpack name
    go_buildpack   1.10.19   go              go

  type:           web
  sidecars:       
  instances:      1/1
  memory usage:   1024M
       state     since                  cpu    memory     disk       logging        details
  #0   running   2024-07-11T07:30:43Z   0.0%   0B of 1G   0B of 1G   0B/s of 0B/s   
  Showing health and status for app appEFh0TbTJ3zUAjZdSBQZBqA in org acceptance-test-org-EFh0TbTJ3zUAjZdSBQZBqA / space acceptance-test-space-EFh0TbTJ3zUAjZdSBQZBqA as admin...

  name:              appEFh0TbTJ3zUAjZdSBQZBqA
  requested state:   started
  routes:            appEFh0TbTJ3zUAjZdSBQZBqA.cf.baba-yaga.env.wg-ard.ci.cloudfoundry.org
  last uploaded:     Thu 11 Jul 07:30:30 UTC 2024
  stack:             cflinuxfs4
  buildpacks:        
    name           version   detect output   buildpack name
    go_buildpack   1.10.19   go              go

  type:           web
  sidecars:       
  instances:      1/1
  memory usage:   1024M
       state     since                  cpu    memory     disk       logging        details
  #0   running   2024-07-11T07:30:43Z   0.0%   0B of 1G   0B of 1G   0B/s of 0B/s   

  STEP: backing up cf @ 07/11/24 07:30:46.705
  [bbr] 2024/07/11 07:30:47 INFO - Looking for scripts
  [bbr] 2024/07/11 07:30:50 INFO - uaa/192c13f5-6395-492f-b10a-4bf78e818a5a/uaa/post-backup-unlock
  [bbr] 2024/07/11 07:30:50 INFO - uaa/192c13f5-6395-492f-b10a-4bf78e818a5a/uaa/post-restore-unlock
  [bbr] 2024/07/11 07:30:50 INFO - uaa/192c13f5-6395-492f-b10a-4bf78e818a5a/uaa/pre-backup-lock
  [bbr] 2024/07/11 07:30:50 INFO - uaa/192c13f5-6395-492f-b10a-4bf78e818a5a/uaa/pre-restore-lock
  [bbr] 2024/07/11 07:30:50 INFO - uaa/7f32628b-9001-4ed0-a829-81c1f614b864/uaa/post-backup-unlock
  [bbr] 2024/07/11 07:30:50 INFO - uaa/7f32628b-9001-4ed0-a829-81c1f614b864/uaa/post-restore-unlock
  [bbr] 2024/07/11 07:30:50 INFO - uaa/7f32628b-9001-4ed0-a829-81c1f614b864/uaa/pre-backup-lock
  [bbr] 2024/07/11 07:30:50 INFO - uaa/7f32628b-9001-4ed0-a829-81c1f614b864/uaa/pre-restore-lock
  [bbr] 2024/07/11 07:30:54 INFO - Detected order: credhub should be locked before uaa/uaa during backup
  [bbr] 2024/07/11 07:30:54 INFO - Detected order: credhub should be locked before uaa/uaa during restore
  [bbr] 2024/07/11 07:30:54 INFO - credhub/9208e608-a6de-4f42-a604-1ff658e2409f/credhub/metadata
  [bbr] 2024/07/11 07:30:54 INFO - credhub/9208e608-a6de-4f42-a604-1ff658e2409f/credhub/post-backup-unlock
  [bbr] 2024/07/11 07:30:54 INFO - credhub/9208e608-a6de-4f42-a604-1ff658e2409f/credhub/post-restore-unlock
  [bbr] 2024/07/11 07:30:54 INFO - credhub/9208e608-a6de-4f42-a604-1ff658e2409f/credhub/pre-backup-lock
  [bbr] 2024/07/11 07:30:54 INFO - credhub/9208e608-a6de-4f42-a604-1ff658e2409f/credhub/pre-restore-lock
  [bbr] 2024/07/11 07:30:54 INFO - Detected order: credhub should be locked before uaa/uaa during backup
  [bbr] 2024/07/11 07:30:54 INFO - Detected order: credhub should be locked before uaa/uaa during restore
  [bbr] 2024/07/11 07:30:54 INFO - credhub/a45f95e6-fd88-450f-ae09-f6f1f3285fb1/credhub/metadata
  [bbr] 2024/07/11 07:30:54 INFO - credhub/a45f95e6-fd88-450f-ae09-f6f1f3285fb1/credhub/post-backup-unlock
  [bbr] 2024/07/11 07:30:54 INFO - credhub/a45f95e6-fd88-450f-ae09-f6f1f3285fb1/credhub/post-restore-unlock
  [bbr] 2024/07/11 07:30:54 INFO - credhub/a45f95e6-fd88-450f-ae09-f6f1f3285fb1/credhub/pre-backup-lock
  [bbr] 2024/07/11 07:30:54 INFO - credhub/a45f95e6-fd88-450f-ae09-f6f1f3285fb1/credhub/pre-restore-lock
  [bbr] 2024/07/11 07:30:59 INFO - cc-worker/bd33ba26-ec79-41ab-9bbb-76ec17b557a6/cloud_controller_worker/post-backup-unlock
  [bbr] 2024/07/11 07:30:59 INFO - cc-worker/bd33ba26-ec79-41ab-9bbb-76ec17b557a6/cloud_controller_worker/post-restore-unlock
  [bbr] 2024/07/11 07:30:59 INFO - cc-worker/bd33ba26-ec79-41ab-9bbb-76ec17b557a6/cloud_controller_worker/pre-backup-lock
  [bbr] 2024/07/11 07:30:59 INFO - cc-worker/bd33ba26-ec79-41ab-9bbb-76ec17b557a6/cloud_controller_worker/pre-restore-lock
  [bbr] 2024/07/11 07:31:00 INFO - cc-worker/e9752676-4e3e-4b8b-8b72-647d91538620/cloud_controller_worker/post-backup-unlock
  [bbr] 2024/07/11 07:31:00 INFO - cc-worker/e9752676-4e3e-4b8b-8b72-647d91538620/cloud_controller_worker/post-restore-unlock
  [bbr] 2024/07/11 07:31:00 INFO - cc-worker/e9752676-4e3e-4b8b-8b72-647d91538620/cloud_controller_worker/pre-backup-lock
  [bbr] 2024/07/11 07:31:00 INFO - cc-worker/e9752676-4e3e-4b8b-8b72-647d91538620/cloud_controller_worker/pre-restore-lock
  [bbr] 2024/07/11 07:31:06 INFO - scheduler/04de45af-b64c-48fe-ba1d-83276320641d/cc_deployment_updater/post-backup-unlock
  [bbr] 2024/07/11 07:31:06 INFO - scheduler/04de45af-b64c-48fe-ba1d-83276320641d/cc_deployment_updater/post-restore-unlock
  [bbr] 2024/07/11 07:31:06 INFO - scheduler/04de45af-b64c-48fe-ba1d-83276320641d/cc_deployment_updater/pre-backup-lock
  [bbr] 2024/07/11 07:31:06 INFO - scheduler/04de45af-b64c-48fe-ba1d-83276320641d/cc_deployment_updater/pre-restore-lock
  [bbr] 2024/07/11 07:31:06 INFO - scheduler/04de45af-b64c-48fe-ba1d-83276320641d/cloud_controller_clock/post-backup-unlock
  [bbr] 2024/07/11 07:31:06 INFO - scheduler/04de45af-b64c-48fe-ba1d-83276320641d/cloud_controller_clock/post-restore-unlock
  [bbr] 2024/07/11 07:31:06 INFO - scheduler/04de45af-b64c-48fe-ba1d-83276320641d/cloud_controller_clock/pre-backup-lock
  [bbr] 2024/07/11 07:31:06 INFO - scheduler/04de45af-b64c-48fe-ba1d-83276320641d/cloud_controller_clock/pre-restore-lock
  [bbr] 2024/07/11 07:31:06 INFO - scheduler/04de45af-b64c-48fe-ba1d-83276320641d/tps/post-backup-unlock
  [bbr] 2024/07/11 07:31:06 INFO - scheduler/04de45af-b64c-48fe-ba1d-83276320641d/tps/post-restore-unlock
  [bbr] 2024/07/11 07:31:06 INFO - scheduler/04de45af-b64c-48fe-ba1d-83276320641d/tps/pre-backup-lock
  [bbr] 2024/07/11 07:31:06 INFO - scheduler/04de45af-b64c-48fe-ba1d-83276320641d/tps/pre-restore-lock
  [bbr] 2024/07/11 07:31:07 INFO - scheduler/4c5c1159-8ca7-4729-8977-a222e9041a4c/cc_deployment_updater/post-backup-unlock
  [bbr] 2024/07/11 07:31:07 INFO - scheduler/4c5c1159-8ca7-4729-8977-a222e9041a4c/cc_deployment_updater/post-restore-unlock
  [bbr] 2024/07/11 07:31:07 INFO - scheduler/4c5c1159-8ca7-4729-8977-a222e9041a4c/cc_deployment_updater/pre-backup-lock
  [bbr] 2024/07/11 07:31:07 INFO - scheduler/4c5c1159-8ca7-4729-8977-a222e9041a4c/cc_deployment_updater/pre-restore-lock
  [bbr] 2024/07/11 07:31:07 INFO - scheduler/4c5c1159-8ca7-4729-8977-a222e9041a4c/cloud_controller_clock/post-backup-unlock
  [bbr] 2024/07/11 07:31:07 INFO - scheduler/4c5c1159-8ca7-4729-8977-a222e9041a4c/cloud_controller_clock/post-restore-unlock
  [bbr] 2024/07/11 07:31:07 INFO - scheduler/4c5c1159-8ca7-4729-8977-a222e9041a4c/cloud_controller_clock/pre-backup-lock
  [bbr] 2024/07/11 07:31:07 INFO - scheduler/4c5c1159-8ca7-4729-8977-a222e9041a4c/cloud_controller_clock/pre-restore-lock
  [bbr] 2024/07/11 07:31:07 INFO - scheduler/4c5c1159-8ca7-4729-8977-a222e9041a4c/tps/post-backup-unlock
  [bbr] 2024/07/11 07:31:07 INFO - scheduler/4c5c1159-8ca7-4729-8977-a222e9041a4c/tps/post-restore-unlock
  [bbr] 2024/07/11 07:31:07 INFO - scheduler/4c5c1159-8ca7-4729-8977-a222e9041a4c/tps/pre-backup-lock
  [bbr] 2024/07/11 07:31:07 INFO - scheduler/4c5c1159-8ca7-4729-8977-a222e9041a4c/tps/pre-restore-lock
  [bbr] 2024/07/11 07:31:09 INFO - Detected order: bbr-smbbroker should be locked before uaa/uaa during backup
  [bbr] 2024/07/11 07:31:09 INFO - Detected order: bbr-smbbroker should be locked before credhub/credhub during backup
  [bbr] 2024/07/11 07:31:09 INFO - Detected order: bbr-smbbroker should be locked before capi/cloud_controller_ng during backup
  [bbr] 2024/07/11 07:31:09 INFO - Detected order: bbr-smbbroker should be locked before uaa/uaa during restore
  [bbr] 2024/07/11 07:31:09 INFO - Detected order: bbr-smbbroker should be locked before credhub/credhub during restore
  [bbr] 2024/07/11 07:31:09 INFO - Detected order: bbr-smbbroker should be locked before capi/cloud_controller_ng during restore
  [bbr] 2024/07/11 07:31:09 INFO - Detected order: nfsbroker-bbr-lock should be locked before uaa/uaa during backup
  [bbr] 2024/07/11 07:31:09 INFO - Detected order: nfsbroker-bbr-lock should be locked before credhub/credhub during backup
  [bbr] 2024/07/11 07:31:09 INFO - Detected order: nfsbroker-bbr-lock should be locked before capi/cloud_controller_ng during backup
  [bbr] 2024/07/11 07:31:09 INFO - Detected order: nfsbroker-bbr-lock should be locked before uaa/uaa during restore
  [bbr] 2024/07/11 07:31:09 INFO - Detected order: nfsbroker-bbr-lock should be locked before credhub/credhub during restore
  [bbr] 2024/07/11 07:31:09 INFO - Detected order: nfsbroker-bbr-lock should be locked before capi/cloud_controller_ng during restore
  [bbr] 2024/07/11 07:31:09 INFO - backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0/bbr-credhubdb/backup
  [bbr] 2024/07/11 07:31:09 INFO - backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0/bbr-credhubdb/restore
  [bbr] 2024/07/11 07:31:09 INFO - backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0/bbr-routingdb/backup
  [bbr] 2024/07/11 07:31:09 INFO - backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0/bbr-routingdb/restore
  [bbr] 2024/07/11 07:31:09 INFO - backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0/bbr-smbbroker/metadata
  [bbr] 2024/07/11 07:31:09 INFO - backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0/bbr-smbbroker/post-restore-unlock
  [bbr] 2024/07/11 07:31:09 INFO - backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0/bbr-uaadb/backup
  [bbr] 2024/07/11 07:31:09 INFO - backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0/bbr-uaadb/restore
  [bbr] 2024/07/11 07:31:09 INFO - backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0/nfsbroker-bbr-lock/metadata
  [bbr] 2024/07/11 07:31:09 INFO - backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0/nfsbroker-bbr-lock/post-restore-unlock
  [bbr] 2024/07/11 07:31:09 INFO - backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0/bbr-cfnetworkingdb/backup
  [bbr] 2024/07/11 07:31:09 INFO - backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0/bbr-cfnetworkingdb/restore
  [bbr] 2024/07/11 07:31:09 INFO - backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0/bbr-cloudcontrollerdb/backup
  [bbr] 2024/07/11 07:31:09 INFO - backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0/bbr-cloudcontrollerdb/restore
  [bbr] 2024/07/11 07:31:12 INFO - singleton-blobstore/76a1d847-8bad-45c5-8894-b46f80241a91/blobstore/backup
  [bbr] 2024/07/11 07:31:12 INFO - singleton-blobstore/76a1d847-8bad-45c5-8894-b46f80241a91/blobstore/restore
  [bbr] 2024/07/11 07:31:17 INFO - Detected order: routing-api should be locked before uaa/uaa during restore
  [bbr] 2024/07/11 07:31:17 INFO - api/9b479d2e-3447-46d0-bfec-1a4428690c3b/cloud_controller_ng/post-backup-unlock
  [bbr] 2024/07/11 07:31:17 INFO - api/9b479d2e-3447-46d0-bfec-1a4428690c3b/cloud_controller_ng/post-restore-unlock
  [bbr] 2024/07/11 07:31:17 INFO - api/9b479d2e-3447-46d0-bfec-1a4428690c3b/cloud_controller_ng/pre-backup-lock
  [bbr] 2024/07/11 07:31:17 INFO - api/9b479d2e-3447-46d0-bfec-1a4428690c3b/cloud_controller_ng/pre-restore-lock
  [bbr] 2024/07/11 07:31:17 INFO - api/9b479d2e-3447-46d0-bfec-1a4428690c3b/policy-server/post-backup-unlock
  [bbr] 2024/07/11 07:31:17 INFO - api/9b479d2e-3447-46d0-bfec-1a4428690c3b/policy-server/pre-backup-lock
  [bbr] 2024/07/11 07:31:17 INFO - api/9b479d2e-3447-46d0-bfec-1a4428690c3b/routing-api/metadata
  [bbr] 2024/07/11 07:31:17 INFO - api/9b479d2e-3447-46d0-bfec-1a4428690c3b/routing-api/post-backup-unlock
  [bbr] 2024/07/11 07:31:17 INFO - api/9b479d2e-3447-46d0-bfec-1a4428690c3b/routing-api/post-restore-unlock
  [bbr] 2024/07/11 07:31:17 INFO - api/9b479d2e-3447-46d0-bfec-1a4428690c3b/routing-api/pre-backup-lock
  [bbr] 2024/07/11 07:31:17 INFO - api/9b479d2e-3447-46d0-bfec-1a4428690c3b/routing-api/pre-restore-lock
  [bbr] 2024/07/11 07:31:18 INFO - Detected order: routing-api should be locked before uaa/uaa during restore
  [bbr] 2024/07/11 07:31:18 INFO - api/64d469c8-02e7-49b6-8033-2aa1e6f08568/cloud_controller_ng/post-backup-unlock
  [bbr] 2024/07/11 07:31:18 INFO - api/64d469c8-02e7-49b6-8033-2aa1e6f08568/cloud_controller_ng/post-restore-unlock
  [bbr] 2024/07/11 07:31:18 INFO - api/64d469c8-02e7-49b6-8033-2aa1e6f08568/cloud_controller_ng/pre-backup-lock
  [bbr] 2024/07/11 07:31:18 INFO - api/64d469c8-02e7-49b6-8033-2aa1e6f08568/cloud_controller_ng/pre-restore-lock
  [bbr] 2024/07/11 07:31:18 INFO - api/64d469c8-02e7-49b6-8033-2aa1e6f08568/policy-server/post-backup-unlock
  [bbr] 2024/07/11 07:31:18 INFO - api/64d469c8-02e7-49b6-8033-2aa1e6f08568/policy-server/pre-backup-lock
  [bbr] 2024/07/11 07:31:18 INFO - api/64d469c8-02e7-49b6-8033-2aa1e6f08568/routing-api/metadata
  [bbr] 2024/07/11 07:31:18 INFO - api/64d469c8-02e7-49b6-8033-2aa1e6f08568/routing-api/post-backup-unlock
  [bbr] 2024/07/11 07:31:18 INFO - api/64d469c8-02e7-49b6-8033-2aa1e6f08568/routing-api/post-restore-unlock
  [bbr] 2024/07/11 07:31:18 INFO - api/64d469c8-02e7-49b6-8033-2aa1e6f08568/routing-api/pre-backup-lock
  [bbr] 2024/07/11 07:31:18 INFO - api/64d469c8-02e7-49b6-8033-2aa1e6f08568/routing-api/pre-restore-lock
  [bbr] 2024/07/11 07:31:20 INFO - Running pre-checks for backup of cf...
  [bbr] 2024/07/11 07:31:27 INFO - Starting backup of cf...
  [bbr] 2024/07/11 07:31:27 INFO - Running pre-backup-lock scripts...
  [bbr] 2024/07/11 07:31:27 INFO - Locking tps on scheduler/04de45af-b64c-48fe-ba1d-83276320641d for backup...
  [bbr] 2024/07/11 07:31:27 INFO - Locking policy-server on api/9b479d2e-3447-46d0-bfec-1a4428690c3b for backup...
  [bbr] 2024/07/11 07:31:27 INFO - Locking cloud_controller_worker on cc-worker/bd33ba26-ec79-41ab-9bbb-76ec17b557a6 for backup...
  [bbr] 2024/07/11 07:31:27 INFO - Locking cloud_controller_clock on scheduler/04de45af-b64c-48fe-ba1d-83276320641d for backup...
  [bbr] 2024/07/11 07:31:27 INFO - Locking cloud_controller_worker on cc-worker/e9752676-4e3e-4b8b-8b72-647d91538620 for backup...
  [bbr] 2024/07/11 07:31:27 INFO - Locking cc_deployment_updater on scheduler/04de45af-b64c-48fe-ba1d-83276320641d for backup...
  [bbr] 2024/07/11 07:31:27 INFO - Locking cloud_controller_clock on scheduler/4c5c1159-8ca7-4729-8977-a222e9041a4c for backup...
  [bbr] 2024/07/11 07:31:27 INFO - Locking cc_deployment_updater on scheduler/4c5c1159-8ca7-4729-8977-a222e9041a4c for backup...
  [bbr] 2024/07/11 07:31:27 INFO - Locking tps on scheduler/4c5c1159-8ca7-4729-8977-a222e9041a4c for backup...
  [bbr] 2024/07/11 07:31:27 INFO - Locking routing-api on api/9b479d2e-3447-46d0-bfec-1a4428690c3b for backup...
  [bbr] 2024/07/11 07:31:27 INFO - Finished locking policy-server on api/9b479d2e-3447-46d0-bfec-1a4428690c3b for backup.
  [bbr] 2024/07/11 07:31:27 INFO - Locking policy-server on api/64d469c8-02e7-49b6-8033-2aa1e6f08568 for backup...
  [bbr] 2024/07/11 07:31:27 INFO - Finished locking routing-api on api/9b479d2e-3447-46d0-bfec-1a4428690c3b for backup.
  [bbr] 2024/07/11 07:31:27 INFO - Locking routing-api on api/64d469c8-02e7-49b6-8033-2aa1e6f08568 for backup...
  [bbr] 2024/07/11 07:31:27 INFO - Finished locking policy-server on api/64d469c8-02e7-49b6-8033-2aa1e6f08568 for backup.
  [bbr] 2024/07/11 07:31:27 INFO - Finished locking routing-api on api/64d469c8-02e7-49b6-8033-2aa1e6f08568 for backup.
  [bbr] 2024/07/11 07:31:27 INFO - Finished locking tps on scheduler/4c5c1159-8ca7-4729-8977-a222e9041a4c for backup.
  [bbr] 2024/07/11 07:31:27 INFO - Finished locking tps on scheduler/04de45af-b64c-48fe-ba1d-83276320641d for backup.
  [bbr] 2024/07/11 07:31:28 INFO - Finished locking cloud_controller_clock on scheduler/4c5c1159-8ca7-4729-8977-a222e9041a4c for backup.
  [bbr] 2024/07/11 07:31:28 INFO - Finished locking cc_deployment_updater on scheduler/4c5c1159-8ca7-4729-8977-a222e9041a4c for backup.
  [bbr] 2024/07/11 07:31:28 INFO - Finished locking cc_deployment_updater on scheduler/04de45af-b64c-48fe-ba1d-83276320641d for backup.
  [bbr] 2024/07/11 07:31:28 INFO - Finished locking cloud_controller_clock on scheduler/04de45af-b64c-48fe-ba1d-83276320641d for backup.
  [bbr] 2024/07/11 07:31:29 INFO - Finished locking cloud_controller_worker on cc-worker/e9752676-4e3e-4b8b-8b72-647d91538620 for backup.
  [bbr] 2024/07/11 07:31:30 INFO - Finished locking cloud_controller_worker on cc-worker/bd33ba26-ec79-41ab-9bbb-76ec17b557a6 for backup.
  [bbr] 2024/07/11 07:31:30 INFO - Locking cloud_controller_ng on api/64d469c8-02e7-49b6-8033-2aa1e6f08568 for backup...
  [bbr] 2024/07/11 07:31:30 INFO - Locking credhub on credhub/a45f95e6-fd88-450f-ae09-f6f1f3285fb1 for backup...
  [bbr] 2024/07/11 07:31:30 INFO - Locking credhub on credhub/9208e608-a6de-4f42-a604-1ff658e2409f for backup...
  [bbr] 2024/07/11 07:31:30 INFO - Locking cloud_controller_ng on api/9b479d2e-3447-46d0-bfec-1a4428690c3b for backup...
  [bbr] 2024/07/11 07:31:30 INFO - Finished locking credhub on credhub/9208e608-a6de-4f42-a604-1ff658e2409f for backup.
  [bbr] 2024/07/11 07:31:30 INFO - Finished locking credhub on credhub/a45f95e6-fd88-450f-ae09-f6f1f3285fb1 for backup.
  [bbr] 2024/07/11 07:31:42 INFO - Finished locking cloud_controller_ng on api/64d469c8-02e7-49b6-8033-2aa1e6f08568 for backup.
  [bbr] 2024/07/11 07:31:43 INFO - Finished locking cloud_controller_ng on api/9b479d2e-3447-46d0-bfec-1a4428690c3b for backup.
  [bbr] 2024/07/11 07:31:43 INFO - Locking uaa on uaa/7f32628b-9001-4ed0-a829-81c1f614b864 for backup...
  [bbr] 2024/07/11 07:31:43 INFO - Locking uaa on uaa/192c13f5-6395-492f-b10a-4bf78e818a5a for backup...
  [bbr] 2024/07/11 07:31:49 INFO - Finished locking uaa on uaa/192c13f5-6395-492f-b10a-4bf78e818a5a for backup.
  [bbr] 2024/07/11 07:31:50 INFO - Finished locking uaa on uaa/7f32628b-9001-4ed0-a829-81c1f614b864 for backup.
  [bbr] 2024/07/11 07:31:50 INFO - Finished running pre-backup-lock scripts.
  [bbr] 2024/07/11 07:31:50 INFO - Running backup scripts...
  [bbr] 2024/07/11 07:31:50 INFO - Backing up blobstore on singleton-blobstore/76a1d847-8bad-45c5-8894-b46f80241a91...
  [bbr] 2024/07/11 07:31:50 INFO - Backing up bbr-uaadb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0...
  [bbr] 2024/07/11 07:31:50 INFO - Backing up bbr-routingdb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0...
  [bbr] 2024/07/11 07:31:50 INFO - Backing up bbr-credhubdb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0...
  [bbr] 2024/07/11 07:31:50 INFO - Backing up bbr-cfnetworkingdb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0...
  [bbr] 2024/07/11 07:31:50 INFO - Backing up bbr-cloudcontrollerdb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0...
  [bbr] 2024/07/11 07:31:50 INFO - Finished backing up blobstore on singleton-blobstore/76a1d847-8bad-45c5-8894-b46f80241a91.
  [bbr] 2024/07/11 07:31:51 INFO - Finished backing up bbr-routingdb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0.
  [bbr] 2024/07/11 07:31:51 INFO - Finished backing up bbr-credhubdb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0.
  [bbr] 2024/07/11 07:31:51 INFO - Finished backing up bbr-cfnetworkingdb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0.
  [bbr] 2024/07/11 07:31:51 INFO - Finished backing up bbr-uaadb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0.
  [bbr] 2024/07/11 07:31:53 INFO - Finished backing up bbr-cloudcontrollerdb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0.
  [bbr] 2024/07/11 07:31:53 INFO - Finished running backup scripts.
  [bbr] 2024/07/11 07:31:53 INFO - Running post-backup-unlock scripts...
  [bbr] 2024/07/11 07:31:53 INFO - Unlocking uaa on uaa/7f32628b-9001-4ed0-a829-81c1f614b864...
  [bbr] 2024/07/11 07:31:53 INFO - Unlocking uaa on uaa/192c13f5-6395-492f-b10a-4bf78e818a5a...
  [bbr] 2024/07/11 07:31:59 INFO - Finished unlocking uaa on uaa/192c13f5-6395-492f-b10a-4bf78e818a5a.
  [bbr] 2024/07/11 07:31:59 INFO - Finished unlocking uaa on uaa/7f32628b-9001-4ed0-a829-81c1f614b864.
  [bbr] 2024/07/11 07:31:59 INFO - Unlocking cloud_controller_ng on api/64d469c8-02e7-49b6-8033-2aa1e6f08568...
  [bbr] 2024/07/11 07:31:59 INFO - Unlocking credhub on credhub/a45f95e6-fd88-450f-ae09-f6f1f3285fb1...
  [bbr] 2024/07/11 07:31:59 INFO - Unlocking credhub on credhub/9208e608-a6de-4f42-a604-1ff658e2409f...
  [bbr] 2024/07/11 07:31:59 INFO - Unlocking cloud_controller_ng on api/9b479d2e-3447-46d0-bfec-1a4428690c3b...
  [bbr] 2024/07/11 07:31:59 INFO - Finished unlocking credhub on credhub/9208e608-a6de-4f42-a604-1ff658e2409f.
  [bbr] 2024/07/11 07:31:59 INFO - Finished unlocking credhub on credhub/a45f95e6-fd88-450f-ae09-f6f1f3285fb1.
  [bbr] 2024/07/11 07:32:12 INFO - Finished unlocking cloud_controller_ng on api/64d469c8-02e7-49b6-8033-2aa1e6f08568.
  [bbr] 2024/07/11 07:32:12 INFO - Finished unlocking cloud_controller_ng on api/9b479d2e-3447-46d0-bfec-1a4428690c3b.
  [bbr] 2024/07/11 07:32:12 INFO - Unlocking policy-server on api/9b479d2e-3447-46d0-bfec-1a4428690c3b...
  [bbr] 2024/07/11 07:32:12 INFO - Unlocking cloud_controller_worker on cc-worker/bd33ba26-ec79-41ab-9bbb-76ec17b557a6...
  [bbr] 2024/07/11 07:32:12 INFO - Unlocking cc_deployment_updater on scheduler/04de45af-b64c-48fe-ba1d-83276320641d...
  [bbr] 2024/07/11 07:32:12 INFO - Unlocking cc_deployment_updater on scheduler/4c5c1159-8ca7-4729-8977-a222e9041a4c...
  [bbr] 2024/07/11 07:32:12 INFO - Unlocking tps on scheduler/04de45af-b64c-48fe-ba1d-83276320641d...
  [bbr] 2024/07/11 07:32:12 INFO - Unlocking cloud_controller_clock on scheduler/04de45af-b64c-48fe-ba1d-83276320641d...
  [bbr] 2024/07/11 07:32:12 INFO - Unlocking tps on scheduler/4c5c1159-8ca7-4729-8977-a222e9041a4c...
  [bbr] 2024/07/11 07:32:12 INFO - Unlocking cloud_controller_clock on scheduler/4c5c1159-8ca7-4729-8977-a222e9041a4c...
  [bbr] 2024/07/11 07:32:12 INFO - Unlocking routing-api on api/9b479d2e-3447-46d0-bfec-1a4428690c3b...
  [bbr] 2024/07/11 07:32:12 INFO - Unlocking cloud_controller_worker on cc-worker/e9752676-4e3e-4b8b-8b72-647d91538620...
  [bbr] 2024/07/11 07:32:13 INFO - Finished unlocking cloud_controller_worker on cc-worker/bd33ba26-ec79-41ab-9bbb-76ec17b557a6.
  [bbr] 2024/07/11 07:32:13 INFO - Unlocking policy-server on api/64d469c8-02e7-49b6-8033-2aa1e6f08568...
  [bbr] 2024/07/11 07:32:13 INFO - Finished unlocking cloud_controller_worker on cc-worker/e9752676-4e3e-4b8b-8b72-647d91538620.
  [bbr] 2024/07/11 07:32:13 INFO - Unlocking routing-api on api/64d469c8-02e7-49b6-8033-2aa1e6f08568...
  [bbr] 2024/07/11 07:32:13 INFO - Finished unlocking routing-api on api/9b479d2e-3447-46d0-bfec-1a4428690c3b.
  [bbr] 2024/07/11 07:32:13 INFO - Finished unlocking cloud_controller_clock on scheduler/04de45af-b64c-48fe-ba1d-83276320641d.
  [bbr] 2024/07/11 07:32:13 INFO - Finished unlocking tps on scheduler/4c5c1159-8ca7-4729-8977-a222e9041a4c.
  [bbr] 2024/07/11 07:32:13 INFO - Finished unlocking cc_deployment_updater on scheduler/4c5c1159-8ca7-4729-8977-a222e9041a4c.
  [bbr] 2024/07/11 07:32:13 INFO - Finished unlocking cc_deployment_updater on scheduler/04de45af-b64c-48fe-ba1d-83276320641d.
  [bbr] 2024/07/11 07:32:13 INFO - Finished unlocking tps on scheduler/04de45af-b64c-48fe-ba1d-83276320641d.
  [bbr] 2024/07/11 07:32:13 INFO - Finished unlocking cloud_controller_clock on scheduler/4c5c1159-8ca7-4729-8977-a222e9041a4c.
  [bbr] 2024/07/11 07:32:13 INFO - Finished unlocking routing-api on api/64d469c8-02e7-49b6-8033-2aa1e6f08568.
  [bbr] 2024/07/11 07:32:14 INFO - Finished unlocking policy-server on api/9b479d2e-3447-46d0-bfec-1a4428690c3b.
  [bbr] 2024/07/11 07:32:14 INFO - Finished unlocking policy-server on api/64d469c8-02e7-49b6-8033-2aa1e6f08568.
  [bbr] 2024/07/11 07:32:14 INFO - Finished running post-backup-unlock scripts.
  [bbr] 2024/07/11 07:32:15 INFO - Copying backup -- 98M uncompressed -- for job blobstore on singleton-blobstore/76a1d847-8bad-45c5-8894-b46f80241a91...
  [bbr] 2024/07/11 07:32:15 INFO - Copying backup -- 1.2M uncompressed -- for job bbr-cfnetworkingdb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0...
  [bbr] 2024/07/11 07:32:15 INFO - Copying backup -- 428K uncompressed -- for job bbr-cloudcontrollerdb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0...
  [bbr] 2024/07/11 07:32:15 INFO - Copying backup for job blobstore on singleton-blobstore/76a1d847-8bad-45c5-8894-b46f80241a91 -- 5% complete
  [bbr] 2024/07/11 07:32:15 INFO - Copying backup for job blobstore on singleton-blobstore/76a1d847-8bad-45c5-8894-b46f80241a91 -- 10% complete
  [bbr] 2024/07/11 07:32:15 INFO - Copying backup -- 60K uncompressed -- for job bbr-uaadb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0...
  [bbr] 2024/07/11 07:32:15 INFO - Copying backup for job blobstore on singleton-blobstore/76a1d847-8bad-45c5-8894-b46f80241a91 -- 15% complete
  [bbr] 2024/07/11 07:32:15 INFO - Copying backup -- 32K uncompressed -- for job bbr-credhubdb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0...
  [bbr] 2024/07/11 07:32:15 INFO - Copying backup for job blobstore on singleton-blobstore/76a1d847-8bad-45c5-8894-b46f80241a91 -- 20% complete
  [bbr] 2024/07/11 07:32:15 INFO - Copying backup -- 12K uncompressed -- for job bbr-routingdb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0...
  [bbr] 2024/07/11 07:32:15 INFO - Copying backup for job blobstore on singleton-blobstore/76a1d847-8bad-45c5-8894-b46f80241a91 -- 25% complete
  [bbr] 2024/07/11 07:32:16 INFO - Copying backup for job blobstore on singleton-blobstore/76a1d847-8bad-45c5-8894-b46f80241a91 -- 30% complete
  [bbr] 2024/07/11 07:32:16 INFO - Copying backup for job bbr-cfnetworkingdb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0 -- 5% complete
  [bbr] 2024/07/11 07:32:16 INFO - Copying backup for job bbr-cfnetworkingdb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0 -- 11% complete
  [bbr] 2024/07/11 07:32:16 INFO - Copying backup for job bbr-cfnetworkingdb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0 -- 18% complete
  [bbr] 2024/07/11 07:32:16 INFO - Copying backup for job bbr-cfnetworkingdb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0 -- 23% complete
  [bbr] 2024/07/11 07:32:16 INFO - Copying backup for job bbr-cfnetworkingdb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0 -- 28% complete
  [bbr] 2024/07/11 07:32:16 INFO - Copying backup for job bbr-cfnetworkingdb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0 -- 33% complete
  [bbr] 2024/07/11 07:32:16 INFO - Copying backup for job bbr-cfnetworkingdb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0 -- 38% complete
  [bbr] 2024/07/11 07:32:16 INFO - Copying backup for job bbr-cfnetworkingdb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0 -- 43% complete
  [bbr] 2024/07/11 07:32:16 INFO - Copying backup for job bbr-cfnetworkingdb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0 -- 49% complete
  [bbr] 2024/07/11 07:32:16 INFO - Copying backup for job bbr-cfnetworkingdb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0 -- 54% complete
  [bbr] 2024/07/11 07:32:16 INFO - Copying backup for job bbr-cfnetworkingdb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0 -- 60% complete
  [bbr] 2024/07/11 07:32:16 INFO - Copying backup for job bbr-cfnetworkingdb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0 -- 65% complete
  [bbr] 2024/07/11 07:32:16 INFO - Copying backup for job bbr-cfnetworkingdb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0 -- 70% complete
  [bbr] 2024/07/11 07:32:16 INFO - Copying backup for job bbr-cfnetworkingdb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0 -- 75% complete
  [bbr] 2024/07/11 07:32:16 INFO - Copying backup for job bbr-cfnetworkingdb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0 -- 81% complete
  [bbr] 2024/07/11 07:32:16 INFO - Copying backup for job bbr-cfnetworkingdb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0 -- 86% complete
  [bbr] 2024/07/11 07:32:16 INFO - Copying backup for job bbr-cfnetworkingdb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0 -- 91% complete
  [bbr] 2024/07/11 07:32:16 INFO - Copying backup for job bbr-cfnetworkingdb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0 -- 96% complete
  [bbr] 2024/07/11 07:32:16 INFO - Finished copying backup -- for job bbr-cfnetworkingdb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0...
  [bbr] 2024/07/11 07:32:16 INFO - Starting validity checks -- for job bbr-cfnetworkingdb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0...
  [bbr] 2024/07/11 07:32:16 INFO - Copying backup for job bbr-cloudcontrollerdb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0 -- 7% complete
  [bbr] 2024/07/11 07:32:16 INFO - Copying backup for job bbr-cloudcontrollerdb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0 -- 14% complete
  [bbr] 2024/07/11 07:32:16 INFO - Copying backup for job bbr-cloudcontrollerdb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0 -- 21% complete
  [bbr] 2024/07/11 07:32:16 INFO - Copying backup for job bbr-cloudcontrollerdb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0 -- 28% complete
  [bbr] 2024/07/11 07:32:16 INFO - Copying backup for job bbr-cloudcontrollerdb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0 -- 36% complete
  [bbr] 2024/07/11 07:32:16 INFO - Copying backup for job bbr-cloudcontrollerdb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0 -- 47% complete
  [bbr] 2024/07/11 07:32:16 INFO - Copying backup for job bbr-cloudcontrollerdb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0 -- 54% complete
  [bbr] 2024/07/11 07:32:16 INFO - Copying backup for job bbr-cloudcontrollerdb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0 -- 62% complete
  [bbr] 2024/07/11 07:32:16 INFO - Copying backup for job bbr-cloudcontrollerdb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0 -- 69% complete
  [bbr] 2024/07/11 07:32:16 INFO - Copying backup for job bbr-cloudcontrollerdb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0 -- 77% complete
  [bbr] 2024/07/11 07:32:16 INFO - Copying backup for job bbr-cloudcontrollerdb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0 -- 84% complete
  [bbr] 2024/07/11 07:32:16 INFO - Copying backup for job bbr-cloudcontrollerdb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0 -- 92% complete
  [bbr] 2024/07/11 07:32:16 INFO - Copying backup for job bbr-cloudcontrollerdb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0 -- 99% complete
  [bbr] 2024/07/11 07:32:16 INFO - Copying backup for job bbr-cloudcontrollerdb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0 -- 100% complete
  [bbr] 2024/07/11 07:32:16 INFO - Finished copying backup -- for job bbr-cloudcontrollerdb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0...
  [bbr] 2024/07/11 07:32:16 INFO - Starting validity checks -- for job bbr-cloudcontrollerdb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0...
  [bbr] 2024/07/11 07:32:16 INFO - Copying backup for job blobstore on singleton-blobstore/76a1d847-8bad-45c5-8894-b46f80241a91 -- 35% complete
  [bbr] 2024/07/11 07:32:16 INFO - Copying backup for job blobstore on singleton-blobstore/76a1d847-8bad-45c5-8894-b46f80241a91 -- 40% complete
  [bbr] 2024/07/11 07:32:16 INFO - Copying backup for job blobstore on singleton-blobstore/76a1d847-8bad-45c5-8894-b46f80241a91 -- 45% complete
  [bbr] 2024/07/11 07:32:16 INFO - Copying backup for job bbr-uaadb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0 -- 16% complete
  [bbr] 2024/07/11 07:32:16 INFO - Copying backup for job bbr-uaadb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0 -- 70% complete
  [bbr] 2024/07/11 07:32:16 INFO - Copying backup for job bbr-uaadb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0 -- 100% complete
  [bbr] 2024/07/11 07:32:16 INFO - Finished copying backup -- for job bbr-uaadb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0...
  [bbr] 2024/07/11 07:32:16 INFO - Starting validity checks -- for job bbr-uaadb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0...
  [bbr] 2024/07/11 07:32:16 INFO - Copying backup for job bbr-credhubdb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0 -- 93% complete
  [bbr] 2024/07/11 07:32:16 INFO - Finished copying backup -- for job bbr-credhubdb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0...
  [bbr] 2024/07/11 07:32:16 INFO - Starting validity checks -- for job bbr-credhubdb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0...
  [bbr] 2024/07/11 07:32:16 INFO - Copying backup for job blobstore on singleton-blobstore/76a1d847-8bad-45c5-8894-b46f80241a91 -- 50% complete
  [bbr] 2024/07/11 07:32:16 INFO - Copying backup for job bbr-routingdb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0 -- 83% complete
  [bbr] 2024/07/11 07:32:16 INFO - Finished copying backup -- for job bbr-routingdb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0...
  [bbr] 2024/07/11 07:32:16 INFO - Starting validity checks -- for job bbr-routingdb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0...
  [bbr] 2024/07/11 07:32:16 INFO - Copying backup for job blobstore on singleton-blobstore/76a1d847-8bad-45c5-8894-b46f80241a91 -- 55% complete
  [bbr] 2024/07/11 07:32:16 INFO - Copying backup for job blobstore on singleton-blobstore/76a1d847-8bad-45c5-8894-b46f80241a91 -- 60% complete
  [bbr] 2024/07/11 07:32:16 INFO - Copying backup for job blobstore on singleton-blobstore/76a1d847-8bad-45c5-8894-b46f80241a91 -- 65% complete
  [bbr] 2024/07/11 07:32:16 INFO - Copying backup for job blobstore on singleton-blobstore/76a1d847-8bad-45c5-8894-b46f80241a91 -- 70% complete
  [bbr] 2024/07/11 07:32:16 INFO - Copying backup for job blobstore on singleton-blobstore/76a1d847-8bad-45c5-8894-b46f80241a91 -- 75% complete
  [bbr] 2024/07/11 07:32:16 INFO - Copying backup for job blobstore on singleton-blobstore/76a1d847-8bad-45c5-8894-b46f80241a91 -- 80% complete
  [bbr] 2024/07/11 07:32:16 INFO - Copying backup for job blobstore on singleton-blobstore/76a1d847-8bad-45c5-8894-b46f80241a91 -- 85% complete
  [bbr] 2024/07/11 07:32:16 INFO - Copying backup for job blobstore on singleton-blobstore/76a1d847-8bad-45c5-8894-b46f80241a91 -- 90% complete
  [bbr] 2024/07/11 07:32:16 INFO - Copying backup for job blobstore on singleton-blobstore/76a1d847-8bad-45c5-8894-b46f80241a91 -- 95% complete
  [bbr] 2024/07/11 07:32:16 INFO - Finished copying backup -- for job blobstore on singleton-blobstore/76a1d847-8bad-45c5-8894-b46f80241a91...
  [bbr] 2024/07/11 07:32:16 INFO - Starting validity checks -- for job blobstore on singleton-blobstore/76a1d847-8bad-45c5-8894-b46f80241a91...
  [bbr] 2024/07/11 07:32:16 INFO - Finished validity checks -- for job bbr-cfnetworkingdb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0...
  [bbr] 2024/07/11 07:32:17 INFO - Finished validity checks -- for job bbr-cloudcontrollerdb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0...
  [bbr] 2024/07/11 07:32:17 INFO - Finished validity checks -- for job bbr-credhubdb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0...
  [bbr] 2024/07/11 07:32:17 INFO - Finished validity checks -- for job bbr-uaadb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0...
  [bbr] 2024/07/11 07:32:17 INFO - Finished validity checks -- for job bbr-routingdb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0...
  [bbr] 2024/07/11 07:32:18 INFO - Finished validity checks -- for job blobstore on singleton-blobstore/76a1d847-8bad-45c5-8894-b46f80241a91...
  [bbr] 2024/07/11 07:32:18 INFO - Backup created of cf on 2024-07-11 07:32:18.320061673 +0000 UTC m=+91.601179433

  Trying to connect to api url: https://api.cf.baba-yaga.env.wg-ard.ci.cloudfoundry.org
  STEP: running the AfterBackup step for cf-routing @ 07/11/24 07:32:23.581
  STEP: Getting CF OAuth Token @ 07/11/24 07:32:23.581
  STEP: Adding an entry in the router group table @ 07/11/24 07:32:23.602
  STEP: running the AfterBackup step for app-uptime @ 07/11/24 07:32:23.631
  STEP: stopping checking the app @ 07/11/24 07:32:23.631
2024/07/11 07:32:23 writing to stopCheckingAppAlive...
2024/07/11 07:32:23 writing to stopCheckingAPIGoesDown...
2024/07/11 07:32:23 reading from valueAPIWasDown...
  STEP: running the AfterBackup step for cf-app @ 07/11/24 07:32:23.631
  Deleting app test_app_Jj7qDp5mLE0PN8bFqWNJlw in org acceptance-test-org-Jj7qDp5mLE0PN8bFqWNJlw / space acceptance-test-space-Jj7qDp5mLE0PN8bFqWNJlw as admin...
  OK

  Starting app stopped_test_app_Jj7qDp5mLE0PN8bFqWNJlw in org acceptance-test-org-Jj7qDp5mLE0PN8bFqWNJlw / space acceptance-test-space-Jj7qDp5mLE0PN8bFqWNJlw as admin...

  Waiting for app to start...

  Instances starting...
  Instances starting...
  Instances starting...

  name:              stopped_test_app_Jj7qDp5mLE0PN8bFqWNJlw
  requested state:   started
  routes:            stopped_test_app_Jj7qDp5mLE0PN8bFqWNJlw.cf.baba-yaga.env.wg-ard.ci.cloudfoundry.org
  last uploaded:     Thu 11 Jul 07:27:28 UTC 2024
  stack:             cflinuxfs4
  buildpacks:        
    name                   version   detect output   buildpack name
    staticfile_buildpack   1.6.13    staticfile      staticfile

  type:           web
  sidecars:       
  instances:      1/1
  memory usage:   1024M
       state     since                  cpu    memory     disk       logging        details
  #0   running   2024-07-11T07:32:32Z   0.0%   0B of 1G   0B of 1G   0B/s of 0B/s   
  Stopping app running_test_app_Jj7qDp5mLE0PN8bFqWNJlw in org acceptance-test-org-Jj7qDp5mLE0PN8bFqWNJlw / space acceptance-test-space-Jj7qDp5mLE0PN8bFqWNJlw as admin...
  OK

  STEP: running the AfterBackup step for cf-uaa @ 07/11/24 07:32:34.276
  STEP: we delete the user and verify @ 07/11/24 07:32:34.276
  Setting API endpoint to https://api.cf.baba-yaga.env.wg-ard.ci.cloudfoundry.org...
  OK

  API endpoint:   https://api.cf.baba-yaga.env.wg-ard.ci.cloudfoundry.org
  API version:    3.169.0

  Not logged in. Use 'cf login' or 'cf login --sso' to log in.
  API endpoint: https://api.cf.baba-yaga.env.wg-ard.ci.cloudfoundry.org

  Authenticating...
  OK

  Use 'cf target' to view or set your target org and space.
  Deleting user uaa-test-user-DpbrpcUGcTO9GICuROHrYA as admin...
  OK

  Logging out admin...
  OK

  Setting API endpoint to https://api.cf.baba-yaga.env.wg-ard.ci.cloudfoundry.org...
  OK

  API endpoint:   https://api.cf.baba-yaga.env.wg-ard.ci.cloudfoundry.org
  API version:    3.169.0

  Not logged in. Use 'cf login' or 'cf login --sso' to log in.
  STEP: user has been deleted. authentication should fail @ 07/11/24 07:32:38.515
  API endpoint: https://api.cf.baba-yaga.env.wg-ard.ci.cloudfoundry.org

  Authenticating...
  Error Code: 401
  Raw Response: {"error":"invalid_client","error_description":"Bad credentials"}
  FAILED
  STEP: running the AfterBackup step for cf-networking @ 07/11/24 07:32:38.588
  Removing network policy from app test_app_A7sCNPOovD1MyNkgu4mrkA to app test_app_A7sCNPOovD1MyNkgu4mrkA in org acceptance-test-org-A7sCNPOovD1MyNkgu4mrkA / space acceptance-test-space-A7sCNPOovD1MyNkgu4mrkA as admin...
  OK

  STEP: running the AfterBackup step for cf-nfsbroker @ 07/11/24 07:32:38.853
  STEP: deleting the NFS service instance after backup @ 07/11/24 07:32:38.853
  Deleting service instance service-instance-RnUPP5gAFpC-Cl-0aCAfhg in org acceptance-test-org-RnUPP5gAFpC-Cl-0aCAfhg / space acceptance-test-space-RnUPP5gAFpC-Cl-0aCAfhg as admin...

  Service instance service-instance-RnUPP5gAFpC-Cl-0aCAfhg deleted.
  OK

  STEP: running the AfterBackup step for cf-smbbroker @ 07/11/24 07:32:45.051
  STEP: deleting the SMB service instance after backup @ 07/11/24 07:32:45.051
  Deleting service instance service-instance-GPIym_GoApw4RLLohpnq5w in org acceptance-test-org-GPIym_GoApw4RLLohpnq5w / space acceptance-test-space-GPIym_GoApw4RLLohpnq5w as admin...

  Service instance service-instance-GPIym_GoApw4RLLohpnq5w deleted.
  OK

  STEP: running the AfterBackup step for cf-credhub @ 07/11/24 07:32:51.231
  STEP: restoring to cf @ 07/11/24 07:32:51.443
  [bbr] 2024/07/11 07:32:51 INFO - Starting restore of cf...
  [bbr] 2024/07/11 07:32:51 INFO - Validating backup artifact for cf...
  [bbr] 2024/07/11 07:32:52 INFO - Looking for scripts
  [bbr] 2024/07/11 07:32:59 INFO - uaa/192c13f5-6395-492f-b10a-4bf78e818a5a/uaa/post-backup-unlock
  [bbr] 2024/07/11 07:32:59 INFO - uaa/192c13f5-6395-492f-b10a-4bf78e818a5a/uaa/post-restore-unlock
  [bbr] 2024/07/11 07:32:59 INFO - uaa/192c13f5-6395-492f-b10a-4bf78e818a5a/uaa/pre-backup-lock
  [bbr] 2024/07/11 07:32:59 INFO - uaa/192c13f5-6395-492f-b10a-4bf78e818a5a/uaa/pre-restore-lock
  [bbr] 2024/07/11 07:33:00 INFO - uaa/7f32628b-9001-4ed0-a829-81c1f614b864/uaa/post-backup-unlock
  [bbr] 2024/07/11 07:33:00 INFO - uaa/7f32628b-9001-4ed0-a829-81c1f614b864/uaa/post-restore-unlock
  [bbr] 2024/07/11 07:33:00 INFO - uaa/7f32628b-9001-4ed0-a829-81c1f614b864/uaa/pre-backup-lock
  [bbr] 2024/07/11 07:33:00 INFO - uaa/7f32628b-9001-4ed0-a829-81c1f614b864/uaa/pre-restore-lock
  [bbr] 2024/07/11 07:33:04 INFO - Detected order: credhub should be locked before uaa/uaa during backup
  [bbr] 2024/07/11 07:33:04 INFO - Detected order: credhub should be locked before uaa/uaa during restore
  [bbr] 2024/07/11 07:33:04 INFO - credhub/9208e608-a6de-4f42-a604-1ff658e2409f/credhub/metadata
  [bbr] 2024/07/11 07:33:04 INFO - credhub/9208e608-a6de-4f42-a604-1ff658e2409f/credhub/post-backup-unlock
  [bbr] 2024/07/11 07:33:04 INFO - credhub/9208e608-a6de-4f42-a604-1ff658e2409f/credhub/post-restore-unlock
  [bbr] 2024/07/11 07:33:04 INFO - credhub/9208e608-a6de-4f42-a604-1ff658e2409f/credhub/pre-backup-lock
  [bbr] 2024/07/11 07:33:04 INFO - credhub/9208e608-a6de-4f42-a604-1ff658e2409f/credhub/pre-restore-lock
  [bbr] 2024/07/11 07:33:05 INFO - Detected order: credhub should be locked before uaa/uaa during backup
  [bbr] 2024/07/11 07:33:05 INFO - Detected order: credhub should be locked before uaa/uaa during restore
  [bbr] 2024/07/11 07:33:05 INFO - credhub/a45f95e6-fd88-450f-ae09-f6f1f3285fb1/credhub/metadata
  [bbr] 2024/07/11 07:33:05 INFO - credhub/a45f95e6-fd88-450f-ae09-f6f1f3285fb1/credhub/post-backup-unlock
  [bbr] 2024/07/11 07:33:05 INFO - credhub/a45f95e6-fd88-450f-ae09-f6f1f3285fb1/credhub/post-restore-unlock
  [bbr] 2024/07/11 07:33:05 INFO - credhub/a45f95e6-fd88-450f-ae09-f6f1f3285fb1/credhub/pre-backup-lock
  [bbr] 2024/07/11 07:33:05 INFO - credhub/a45f95e6-fd88-450f-ae09-f6f1f3285fb1/credhub/pre-restore-lock
  [bbr] 2024/07/11 07:33:09 INFO - Detected order: bbr-smbbroker should be locked before uaa/uaa during backup
  [bbr] 2024/07/11 07:33:09 INFO - Detected order: bbr-smbbroker should be locked before credhub/credhub during backup
  [bbr] 2024/07/11 07:33:09 INFO - Detected order: bbr-smbbroker should be locked before capi/cloud_controller_ng during backup
  [bbr] 2024/07/11 07:33:09 INFO - Detected order: bbr-smbbroker should be locked before uaa/uaa during restore
  [bbr] 2024/07/11 07:33:09 INFO - Detected order: bbr-smbbroker should be locked before credhub/credhub during restore
  [bbr] 2024/07/11 07:33:09 INFO - Detected order: bbr-smbbroker should be locked before capi/cloud_controller_ng during restore
  [bbr] 2024/07/11 07:33:09 INFO - Detected order: nfsbroker-bbr-lock should be locked before uaa/uaa during backup
  [bbr] 2024/07/11 07:33:09 INFO - Detected order: nfsbroker-bbr-lock should be locked before credhub/credhub during backup
  [bbr] 2024/07/11 07:33:09 INFO - Detected order: nfsbroker-bbr-lock should be locked before capi/cloud_controller_ng during backup
  [bbr] 2024/07/11 07:33:09 INFO - Detected order: nfsbroker-bbr-lock should be locked before uaa/uaa during restore
  [bbr] 2024/07/11 07:33:09 INFO - Detected order: nfsbroker-bbr-lock should be locked before credhub/credhub during restore
  [bbr] 2024/07/11 07:33:09 INFO - Detected order: nfsbroker-bbr-lock should be locked before capi/cloud_controller_ng during restore
  [bbr] 2024/07/11 07:33:09 INFO - backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0/bbr-credhubdb/backup
  [bbr] 2024/07/11 07:33:09 INFO - backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0/bbr-credhubdb/restore
  [bbr] 2024/07/11 07:33:09 INFO - backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0/bbr-routingdb/backup
  [bbr] 2024/07/11 07:33:09 INFO - backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0/bbr-routingdb/restore
  [bbr] 2024/07/11 07:33:09 INFO - backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0/bbr-smbbroker/metadata
  [bbr] 2024/07/11 07:33:09 INFO - backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0/bbr-smbbroker/post-restore-unlock
  [bbr] 2024/07/11 07:33:09 INFO - backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0/bbr-uaadb/backup
  [bbr] 2024/07/11 07:33:09 INFO - backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0/bbr-uaadb/restore
  [bbr] 2024/07/11 07:33:09 INFO - backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0/nfsbroker-bbr-lock/metadata
  [bbr] 2024/07/11 07:33:09 INFO - backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0/nfsbroker-bbr-lock/post-restore-unlock
  [bbr] 2024/07/11 07:33:09 INFO - backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0/bbr-cfnetworkingdb/backup
  [bbr] 2024/07/11 07:33:09 INFO - backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0/bbr-cfnetworkingdb/restore
  [bbr] 2024/07/11 07:33:09 INFO - backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0/bbr-cloudcontrollerdb/backup
  [bbr] 2024/07/11 07:33:09 INFO - backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0/bbr-cloudcontrollerdb/restore
  [bbr] 2024/07/11 07:33:13 INFO - scheduler/04de45af-b64c-48fe-ba1d-83276320641d/cc_deployment_updater/post-backup-unlock
  [bbr] 2024/07/11 07:33:13 INFO - scheduler/04de45af-b64c-48fe-ba1d-83276320641d/cc_deployment_updater/post-restore-unlock
  [bbr] 2024/07/11 07:33:13 INFO - scheduler/04de45af-b64c-48fe-ba1d-83276320641d/cc_deployment_updater/pre-backup-lock
  [bbr] 2024/07/11 07:33:13 INFO - scheduler/04de45af-b64c-48fe-ba1d-83276320641d/cc_deployment_updater/pre-restore-lock
  [bbr] 2024/07/11 07:33:13 INFO - scheduler/04de45af-b64c-48fe-ba1d-83276320641d/cloud_controller_clock/post-backup-unlock
  [bbr] 2024/07/11 07:33:13 INFO - scheduler/04de45af-b64c-48fe-ba1d-83276320641d/cloud_controller_clock/post-restore-unlock
  [bbr] 2024/07/11 07:33:13 INFO - scheduler/04de45af-b64c-48fe-ba1d-83276320641d/cloud_controller_clock/pre-backup-lock
  [bbr] 2024/07/11 07:33:13 INFO - scheduler/04de45af-b64c-48fe-ba1d-83276320641d/cloud_controller_clock/pre-restore-lock
  [bbr] 2024/07/11 07:33:13 INFO - scheduler/04de45af-b64c-48fe-ba1d-83276320641d/tps/post-backup-unlock
  [bbr] 2024/07/11 07:33:13 INFO - scheduler/04de45af-b64c-48fe-ba1d-83276320641d/tps/post-restore-unlock
  [bbr] 2024/07/11 07:33:13 INFO - scheduler/04de45af-b64c-48fe-ba1d-83276320641d/tps/pre-backup-lock
  [bbr] 2024/07/11 07:33:13 INFO - scheduler/04de45af-b64c-48fe-ba1d-83276320641d/tps/pre-restore-lock
  [bbr] 2024/07/11 07:33:14 INFO - scheduler/4c5c1159-8ca7-4729-8977-a222e9041a4c/tps/post-backup-unlock
  [bbr] 2024/07/11 07:33:14 INFO - scheduler/4c5c1159-8ca7-4729-8977-a222e9041a4c/tps/post-restore-unlock
  [bbr] 2024/07/11 07:33:14 INFO - scheduler/4c5c1159-8ca7-4729-8977-a222e9041a4c/tps/pre-backup-lock
  [bbr] 2024/07/11 07:33:14 INFO - scheduler/4c5c1159-8ca7-4729-8977-a222e9041a4c/tps/pre-restore-lock
  [bbr] 2024/07/11 07:33:14 INFO - scheduler/4c5c1159-8ca7-4729-8977-a222e9041a4c/cc_deployment_updater/post-backup-unlock
  [bbr] 2024/07/11 07:33:14 INFO - scheduler/4c5c1159-8ca7-4729-8977-a222e9041a4c/cc_deployment_updater/post-restore-unlock
  [bbr] 2024/07/11 07:33:14 INFO - scheduler/4c5c1159-8ca7-4729-8977-a222e9041a4c/cc_deployment_updater/pre-backup-lock
  [bbr] 2024/07/11 07:33:14 INFO - scheduler/4c5c1159-8ca7-4729-8977-a222e9041a4c/cc_deployment_updater/pre-restore-lock
  [bbr] 2024/07/11 07:33:14 INFO - scheduler/4c5c1159-8ca7-4729-8977-a222e9041a4c/cloud_controller_clock/post-backup-unlock
  [bbr] 2024/07/11 07:33:14 INFO - scheduler/4c5c1159-8ca7-4729-8977-a222e9041a4c/cloud_controller_clock/post-restore-unlock
  [bbr] 2024/07/11 07:33:14 INFO - scheduler/4c5c1159-8ca7-4729-8977-a222e9041a4c/cloud_controller_clock/pre-backup-lock
  [bbr] 2024/07/11 07:33:14 INFO - scheduler/4c5c1159-8ca7-4729-8977-a222e9041a4c/cloud_controller_clock/pre-restore-lock
  [bbr] 2024/07/11 07:33:17 INFO - cc-worker/bd33ba26-ec79-41ab-9bbb-76ec17b557a6/cloud_controller_worker/post-backup-unlock
  [bbr] 2024/07/11 07:33:17 INFO - cc-worker/bd33ba26-ec79-41ab-9bbb-76ec17b557a6/cloud_controller_worker/post-restore-unlock
  [bbr] 2024/07/11 07:33:17 INFO - cc-worker/bd33ba26-ec79-41ab-9bbb-76ec17b557a6/cloud_controller_worker/pre-backup-lock
  [bbr] 2024/07/11 07:33:17 INFO - cc-worker/bd33ba26-ec79-41ab-9bbb-76ec17b557a6/cloud_controller_worker/pre-restore-lock
  [bbr] 2024/07/11 07:33:18 INFO - cc-worker/e9752676-4e3e-4b8b-8b72-647d91538620/cloud_controller_worker/post-backup-unlock
  [bbr] 2024/07/11 07:33:18 INFO - cc-worker/e9752676-4e3e-4b8b-8b72-647d91538620/cloud_controller_worker/post-restore-unlock
  [bbr] 2024/07/11 07:33:18 INFO - cc-worker/e9752676-4e3e-4b8b-8b72-647d91538620/cloud_controller_worker/pre-backup-lock
  [bbr] 2024/07/11 07:33:18 INFO - cc-worker/e9752676-4e3e-4b8b-8b72-647d91538620/cloud_controller_worker/pre-restore-lock
  [bbr] 2024/07/11 07:33:21 INFO - Detected order: routing-api should be locked before uaa/uaa during restore
  [bbr] 2024/07/11 07:33:21 INFO - api/9b479d2e-3447-46d0-bfec-1a4428690c3b/cloud_controller_ng/post-backup-unlock
  [bbr] 2024/07/11 07:33:21 INFO - api/9b479d2e-3447-46d0-bfec-1a4428690c3b/cloud_controller_ng/post-restore-unlock
  [bbr] 2024/07/11 07:33:21 INFO - api/9b479d2e-3447-46d0-bfec-1a4428690c3b/cloud_controller_ng/pre-backup-lock
  [bbr] 2024/07/11 07:33:21 INFO - api/9b479d2e-3447-46d0-bfec-1a4428690c3b/cloud_controller_ng/pre-restore-lock
  [bbr] 2024/07/11 07:33:21 INFO - api/9b479d2e-3447-46d0-bfec-1a4428690c3b/policy-server/post-backup-unlock
  [bbr] 2024/07/11 07:33:21 INFO - api/9b479d2e-3447-46d0-bfec-1a4428690c3b/policy-server/pre-backup-lock
  [bbr] 2024/07/11 07:33:21 INFO - api/9b479d2e-3447-46d0-bfec-1a4428690c3b/routing-api/metadata
  [bbr] 2024/07/11 07:33:21 INFO - api/9b479d2e-3447-46d0-bfec-1a4428690c3b/routing-api/post-backup-unlock
  [bbr] 2024/07/11 07:33:21 INFO - api/9b479d2e-3447-46d0-bfec-1a4428690c3b/routing-api/post-restore-unlock
  [bbr] 2024/07/11 07:33:21 INFO - api/9b479d2e-3447-46d0-bfec-1a4428690c3b/routing-api/pre-backup-lock
  [bbr] 2024/07/11 07:33:21 INFO - api/9b479d2e-3447-46d0-bfec-1a4428690c3b/routing-api/pre-restore-lock
  [bbr] 2024/07/11 07:33:22 INFO - Detected order: routing-api should be locked before uaa/uaa during restore
  [bbr] 2024/07/11 07:33:22 INFO - api/64d469c8-02e7-49b6-8033-2aa1e6f08568/cloud_controller_ng/post-backup-unlock
  [bbr] 2024/07/11 07:33:22 INFO - api/64d469c8-02e7-49b6-8033-2aa1e6f08568/cloud_controller_ng/post-restore-unlock
  [bbr] 2024/07/11 07:33:22 INFO - api/64d469c8-02e7-49b6-8033-2aa1e6f08568/cloud_controller_ng/pre-backup-lock
  [bbr] 2024/07/11 07:33:22 INFO - api/64d469c8-02e7-49b6-8033-2aa1e6f08568/cloud_controller_ng/pre-restore-lock
  [bbr] 2024/07/11 07:33:22 INFO - api/64d469c8-02e7-49b6-8033-2aa1e6f08568/policy-server/post-backup-unlock
  [bbr] 2024/07/11 07:33:22 INFO - api/64d469c8-02e7-49b6-8033-2aa1e6f08568/policy-server/pre-backup-lock
  [bbr] 2024/07/11 07:33:22 INFO - api/64d469c8-02e7-49b6-8033-2aa1e6f08568/routing-api/metadata
  [bbr] 2024/07/11 07:33:22 INFO - api/64d469c8-02e7-49b6-8033-2aa1e6f08568/routing-api/post-backup-unlock
  [bbr] 2024/07/11 07:33:22 INFO - api/64d469c8-02e7-49b6-8033-2aa1e6f08568/routing-api/post-restore-unlock
  [bbr] 2024/07/11 07:33:22 INFO - api/64d469c8-02e7-49b6-8033-2aa1e6f08568/routing-api/pre-backup-lock
  [bbr] 2024/07/11 07:33:22 INFO - api/64d469c8-02e7-49b6-8033-2aa1e6f08568/routing-api/pre-restore-lock
  [bbr] 2024/07/11 07:33:24 INFO - singleton-blobstore/76a1d847-8bad-45c5-8894-b46f80241a91/blobstore/backup
  [bbr] 2024/07/11 07:33:24 INFO - singleton-blobstore/76a1d847-8bad-45c5-8894-b46f80241a91/blobstore/restore
  [bbr] 2024/07/11 07:33:31 INFO - Copying backup -- 1.2M uncompressed -- for job bbr-cfnetworkingdb on backup-restore/0...
  [bbr] 2024/07/11 07:33:31 INFO - Copying backup -- 98M uncompressed -- for job blobstore on singleton-blobstore/0...
  [bbr] 2024/07/11 07:33:31 INFO - Copying backup -- 432K uncompressed -- for job bbr-cloudcontrollerdb on backup-restore/0...
  [bbr] 2024/07/11 07:33:31 INFO - Copying backup -- 12K uncompressed -- for job bbr-routingdb on backup-restore/0...
  [bbr] 2024/07/11 07:33:31 INFO - Copying backup -- 32K uncompressed -- for job bbr-credhubdb on backup-restore/0...
  [bbr] 2024/07/11 07:33:31 INFO - Copying backup -- 60K uncompressed -- for job bbr-uaadb on backup-restore/0...
  [bbr] 2024/07/11 07:33:32 INFO - Copying backup for job blobstore on singleton-blobstore/76a1d847-8bad-45c5-8894-b46f80241a91 -- 5% complete
  [bbr] 2024/07/11 07:33:32 INFO - Copying backup for job blobstore on singleton-blobstore/76a1d847-8bad-45c5-8894-b46f80241a91 -- 10% complete
  [bbr] 2024/07/11 07:33:32 INFO - Copying backup for job bbr-cloudcontrollerdb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0 -- 7% complete
  [bbr] 2024/07/11 07:33:32 INFO - Copying backup for job bbr-cloudcontrollerdb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0 -- 14% complete
  [bbr] 2024/07/11 07:33:32 INFO - Copying backup for job bbr-cloudcontrollerdb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0 -- 22% complete
  [bbr] 2024/07/11 07:33:32 INFO - Copying backup for job bbr-cloudcontrollerdb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0 -- 29% complete
  [bbr] 2024/07/11 07:33:32 INFO - Copying backup for job bbr-cloudcontrollerdb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0 -- 37% complete
  [bbr] 2024/07/11 07:33:32 INFO - Copying backup for job bbr-cloudcontrollerdb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0 -- 44% complete
  [bbr] 2024/07/11 07:33:32 INFO - Copying backup for job bbr-cloudcontrollerdb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0 -- 51% complete
  [bbr] 2024/07/11 07:33:32 INFO - Copying backup for job bbr-cloudcontrollerdb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0 -- 59% complete
  [bbr] 2024/07/11 07:33:32 INFO - Copying backup for job bbr-cloudcontrollerdb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0 -- 66% complete
  [bbr] 2024/07/11 07:33:32 INFO - Copying backup for job bbr-cloudcontrollerdb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0 -- 74% complete
  [bbr] 2024/07/11 07:33:32 INFO - Copying backup for job bbr-cloudcontrollerdb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0 -- 81% complete
  [bbr] 2024/07/11 07:33:32 INFO - Copying backup for job bbr-cloudcontrollerdb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0 -- 88% complete
  [bbr] 2024/07/11 07:33:32 INFO - Copying backup for job bbr-cloudcontrollerdb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0 -- 96% complete
  [bbr] 2024/07/11 07:33:32 INFO - Copying backup for job blobstore on singleton-blobstore/76a1d847-8bad-45c5-8894-b46f80241a91 -- 15% complete
  [bbr] 2024/07/11 07:33:32 INFO - Copying backup for job bbr-uaadb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0 -- 53% complete
  [bbr] 2024/07/11 07:33:32 INFO - Copying backup for job bbr-uaadb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0 -- 100% complete
  [bbr] 2024/07/11 07:33:32 INFO - Copying backup for job blobstore on singleton-blobstore/76a1d847-8bad-45c5-8894-b46f80241a91 -- 20% complete
  [bbr] 2024/07/11 07:33:32 INFO - Copying backup for job bbr-routingdb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0 -- 83% complete
  [bbr] 2024/07/11 07:33:32 INFO - Copying backup for job blobstore on singleton-blobstore/76a1d847-8bad-45c5-8894-b46f80241a91 -- 25% complete
  [bbr] 2024/07/11 07:33:32 INFO - Copying backup for job blobstore on singleton-blobstore/76a1d847-8bad-45c5-8894-b46f80241a91 -- 30% complete
  [bbr] 2024/07/11 07:33:32 INFO - Copying backup for job blobstore on singleton-blobstore/76a1d847-8bad-45c5-8894-b46f80241a91 -- 35% complete
  [bbr] 2024/07/11 07:33:32 INFO - Copying backup for job bbr-credhubdb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0 -- 93% complete
  [bbr] 2024/07/11 07:33:32 INFO - Copying backup for job blobstore on singleton-blobstore/76a1d847-8bad-45c5-8894-b46f80241a91 -- 40% complete
  [bbr] 2024/07/11 07:33:32 INFO - Copying backup for job bbr-cfnetworkingdb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0 -- 5% complete
  [bbr] 2024/07/11 07:33:32 INFO - Copying backup for job bbr-cfnetworkingdb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0 -- 10% complete
  [bbr] 2024/07/11 07:33:32 INFO - Copying backup for job bbr-cfnetworkingdb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0 -- 15% complete
  [bbr] 2024/07/11 07:33:32 INFO - Copying backup for job bbr-cfnetworkingdb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0 -- 20% complete
  [bbr] 2024/07/11 07:33:32 INFO - Copying backup for job bbr-cfnetworkingdb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0 -- 26% complete
  [bbr] 2024/07/11 07:33:32 INFO - Copying backup for job bbr-cfnetworkingdb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0 -- 31% complete
  [bbr] 2024/07/11 07:33:32 INFO - Copying backup for job bbr-cfnetworkingdb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0 -- 36% complete
  [bbr] 2024/07/11 07:33:32 INFO - Copying backup for job bbr-cfnetworkingdb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0 -- 41% complete
  [bbr] 2024/07/11 07:33:32 INFO - Copying backup for job bbr-cfnetworkingdb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0 -- 47% complete
  [bbr] 2024/07/11 07:33:32 INFO - Copying backup for job bbr-cfnetworkingdb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0 -- 52% complete
  [bbr] 2024/07/11 07:33:32 INFO - Copying backup for job bbr-cfnetworkingdb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0 -- 57% complete
  [bbr] 2024/07/11 07:33:32 INFO - Copying backup for job bbr-cfnetworkingdb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0 -- 62% complete
  [bbr] 2024/07/11 07:33:32 INFO - Copying backup for job bbr-cfnetworkingdb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0 -- 68% complete
  [bbr] 2024/07/11 07:33:32 INFO - Copying backup for job bbr-cfnetworkingdb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0 -- 73% complete
  [bbr] 2024/07/11 07:33:32 INFO - Copying backup for job bbr-cfnetworkingdb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0 -- 78% complete
  [bbr] 2024/07/11 07:33:32 INFO - Copying backup for job bbr-cfnetworkingdb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0 -- 83% complete
  [bbr] 2024/07/11 07:33:32 INFO - Copying backup for job bbr-cfnetworkingdb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0 -- 89% complete
  [bbr] 2024/07/11 07:33:32 INFO - Copying backup for job bbr-cfnetworkingdb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0 -- 94% complete
  [bbr] 2024/07/11 07:33:32 INFO - Copying backup for job bbr-cfnetworkingdb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0 -- 99% complete
  [bbr] 2024/07/11 07:33:32 INFO - Copying backup for job blobstore on singleton-blobstore/76a1d847-8bad-45c5-8894-b46f80241a91 -- 45% complete
  [bbr] 2024/07/11 07:33:32 INFO - Copying backup for job blobstore on singleton-blobstore/76a1d847-8bad-45c5-8894-b46f80241a91 -- 50% complete
  [bbr] 2024/07/11 07:33:32 INFO - Copying backup for job blobstore on singleton-blobstore/76a1d847-8bad-45c5-8894-b46f80241a91 -- 55% complete
  [bbr] 2024/07/11 07:33:32 INFO - Copying backup for job blobstore on singleton-blobstore/76a1d847-8bad-45c5-8894-b46f80241a91 -- 60% complete
  [bbr] 2024/07/11 07:33:32 INFO - Copying backup for job blobstore on singleton-blobstore/76a1d847-8bad-45c5-8894-b46f80241a91 -- 65% complete
  [bbr] 2024/07/11 07:33:32 INFO - Copying backup for job blobstore on singleton-blobstore/76a1d847-8bad-45c5-8894-b46f80241a91 -- 70% complete
  [bbr] 2024/07/11 07:33:32 INFO - Copying backup for job blobstore on singleton-blobstore/76a1d847-8bad-45c5-8894-b46f80241a91 -- 75% complete
  [bbr] 2024/07/11 07:33:32 INFO - Finished copying backup for job bbr-uaadb on backup-restore/0.
  [bbr] 2024/07/11 07:33:32 INFO - Copying backup for job blobstore on singleton-blobstore/76a1d847-8bad-45c5-8894-b46f80241a91 -- 80% complete
  [bbr] 2024/07/11 07:33:32 INFO - Finished copying backup for job bbr-cloudcontrollerdb on backup-restore/0.
  [bbr] 2024/07/11 07:33:32 INFO - Finished copying backup for job bbr-routingdb on backup-restore/0.
  [bbr] 2024/07/11 07:33:32 INFO - Copying backup for job blobstore on singleton-blobstore/76a1d847-8bad-45c5-8894-b46f80241a91 -- 85% complete
  [bbr] 2024/07/11 07:33:32 INFO - Copying backup for job blobstore on singleton-blobstore/76a1d847-8bad-45c5-8894-b46f80241a91 -- 90% complete
  [bbr] 2024/07/11 07:33:33 INFO - Copying backup for job blobstore on singleton-blobstore/76a1d847-8bad-45c5-8894-b46f80241a91 -- 95% complete
  [bbr] 2024/07/11 07:33:33 INFO - Copying backup for job blobstore on singleton-blobstore/76a1d847-8bad-45c5-8894-b46f80241a91 -- 100% complete
  [bbr] 2024/07/11 07:33:33 INFO - Finished copying backup for job bbr-credhubdb on backup-restore/0.
  [bbr] 2024/07/11 07:33:33 INFO - Finished copying backup for job bbr-cfnetworkingdb on backup-restore/0.
  [bbr] 2024/07/11 07:33:34 INFO - Finished copying backup for job blobstore on singleton-blobstore/0.
  [bbr] 2024/07/11 07:33:34 INFO - Running pre-restore-lock scripts...
  [bbr] 2024/07/11 07:33:34 INFO - Locking cc_deployment_updater on scheduler/04de45af-b64c-48fe-ba1d-83276320641d for restore...
  [bbr] 2024/07/11 07:33:35 INFO - Finished locking cc_deployment_updater on scheduler/04de45af-b64c-48fe-ba1d-83276320641d for restore.
  [bbr] 2024/07/11 07:33:35 INFO - Locking cloud_controller_clock on scheduler/04de45af-b64c-48fe-ba1d-83276320641d for restore...
  [bbr] 2024/07/11 07:33:37 INFO - Finished locking cloud_controller_clock on scheduler/04de45af-b64c-48fe-ba1d-83276320641d for restore.
  [bbr] 2024/07/11 07:33:37 INFO - Locking tps on scheduler/04de45af-b64c-48fe-ba1d-83276320641d for restore...
  [bbr] 2024/07/11 07:33:37 INFO - Finished locking tps on scheduler/04de45af-b64c-48fe-ba1d-83276320641d for restore.
  [bbr] 2024/07/11 07:33:37 INFO - Locking tps on scheduler/4c5c1159-8ca7-4729-8977-a222e9041a4c for restore...
  [bbr] 2024/07/11 07:33:38 INFO - Finished locking tps on scheduler/4c5c1159-8ca7-4729-8977-a222e9041a4c for restore.
  [bbr] 2024/07/11 07:33:38 INFO - Locking cc_deployment_updater on scheduler/4c5c1159-8ca7-4729-8977-a222e9041a4c for restore...
  [bbr] 2024/07/11 07:33:39 INFO - Finished locking cc_deployment_updater on scheduler/4c5c1159-8ca7-4729-8977-a222e9041a4c for restore.
  [bbr] 2024/07/11 07:33:39 INFO - Locking cloud_controller_clock on scheduler/4c5c1159-8ca7-4729-8977-a222e9041a4c for restore...
  [bbr] 2024/07/11 07:33:41 INFO - Finished locking cloud_controller_clock on scheduler/4c5c1159-8ca7-4729-8977-a222e9041a4c for restore.
  [bbr] 2024/07/11 07:33:41 INFO - Locking cloud_controller_worker on cc-worker/bd33ba26-ec79-41ab-9bbb-76ec17b557a6 for restore...
  [bbr] 2024/07/11 07:33:42 INFO - Finished locking cloud_controller_worker on cc-worker/bd33ba26-ec79-41ab-9bbb-76ec17b557a6 for restore.
  [bbr] 2024/07/11 07:33:42 INFO - Locking cloud_controller_worker on cc-worker/e9752676-4e3e-4b8b-8b72-647d91538620 for restore...
  [bbr] 2024/07/11 07:33:46 INFO - Finished locking cloud_controller_worker on cc-worker/e9752676-4e3e-4b8b-8b72-647d91538620 for restore.
  [bbr] 2024/07/11 07:33:46 INFO - Locking routing-api on api/9b479d2e-3447-46d0-bfec-1a4428690c3b for restore...
  [bbr] 2024/07/11 07:34:01 INFO - Finished locking routing-api on api/9b479d2e-3447-46d0-bfec-1a4428690c3b for restore.
  [bbr] 2024/07/11 07:34:01 INFO - Locking routing-api on api/64d469c8-02e7-49b6-8033-2aa1e6f08568 for restore...
  [bbr] 2024/07/11 07:34:17 INFO - Finished locking routing-api on api/64d469c8-02e7-49b6-8033-2aa1e6f08568 for restore.
  [bbr] 2024/07/11 07:34:17 INFO - Locking credhub on credhub/9208e608-a6de-4f42-a604-1ff658e2409f for restore...
  [bbr] 2024/07/11 07:34:17 INFO - Finished locking credhub on credhub/9208e608-a6de-4f42-a604-1ff658e2409f for restore.
  [bbr] 2024/07/11 07:34:17 INFO - Locking credhub on credhub/a45f95e6-fd88-450f-ae09-f6f1f3285fb1 for restore...
  [bbr] 2024/07/11 07:34:18 INFO - Finished locking credhub on credhub/a45f95e6-fd88-450f-ae09-f6f1f3285fb1 for restore.
  [bbr] 2024/07/11 07:34:18 INFO - Locking cloud_controller_ng on api/9b479d2e-3447-46d0-bfec-1a4428690c3b for restore...
  [bbr] 2024/07/11 07:34:26 INFO - Finished locking cloud_controller_ng on api/9b479d2e-3447-46d0-bfec-1a4428690c3b for restore.
  [bbr] 2024/07/11 07:34:26 INFO - Locking cloud_controller_ng on api/64d469c8-02e7-49b6-8033-2aa1e6f08568 for restore...
  [bbr] 2024/07/11 07:34:37 INFO - Finished locking cloud_controller_ng on api/64d469c8-02e7-49b6-8033-2aa1e6f08568 for restore.
  [bbr] 2024/07/11 07:34:37 INFO - Locking uaa on uaa/192c13f5-6395-492f-b10a-4bf78e818a5a for restore...
  [bbr] 2024/07/11 07:34:40 INFO - Finished locking uaa on uaa/192c13f5-6395-492f-b10a-4bf78e818a5a for restore.
  [bbr] 2024/07/11 07:34:40 INFO - Locking uaa on uaa/7f32628b-9001-4ed0-a829-81c1f614b864 for restore...
  [bbr] 2024/07/11 07:34:44 INFO - Finished locking uaa on uaa/7f32628b-9001-4ed0-a829-81c1f614b864 for restore.
  [bbr] 2024/07/11 07:34:44 INFO - Finished running pre-restore-lock scripts.
  [bbr] 2024/07/11 07:34:44 INFO - Running restore scripts...
  [bbr] 2024/07/11 07:34:44 INFO - Restoring bbr-credhubdb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0...
  [bbr] 2024/07/11 07:34:45 INFO - Finished restoring bbr-credhubdb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0.
  [bbr] 2024/07/11 07:34:45 INFO - Restoring bbr-routingdb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0...
  [bbr] 2024/07/11 07:34:46 INFO - Finished restoring bbr-routingdb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0.
  [bbr] 2024/07/11 07:34:46 INFO - Restoring bbr-uaadb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0...
  [bbr] 2024/07/11 07:34:47 INFO - Finished restoring bbr-uaadb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0.
  [bbr] 2024/07/11 07:34:47 INFO - Restoring bbr-cfnetworkingdb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0...
  [bbr] 2024/07/11 07:34:50 INFO - Finished restoring bbr-cfnetworkingdb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0.
  [bbr] 2024/07/11 07:34:50 INFO - Restoring bbr-cloudcontrollerdb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0...
  [bbr] 2024/07/11 07:35:03 INFO - Finished restoring bbr-cloudcontrollerdb on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0.
  [bbr] 2024/07/11 07:35:03 INFO - Restoring blobstore on singleton-blobstore/76a1d847-8bad-45c5-8894-b46f80241a91...
  [bbr] 2024/07/11 07:35:03 INFO - Finished restoring blobstore on singleton-blobstore/76a1d847-8bad-45c5-8894-b46f80241a91.
  [bbr] 2024/07/11 07:35:03 INFO - Finished running restore scripts.
  [bbr] 2024/07/11 07:35:03 INFO - Completed restore of cf
  [bbr] 2024/07/11 07:35:03 INFO - Running post-restore-unlock scripts...
  [bbr] 2024/07/11 07:35:03 INFO - Unlocking uaa on uaa/192c13f5-6395-492f-b10a-4bf78e818a5a...
  [bbr] 2024/07/11 07:35:40 INFO - Finished unlocking uaa on uaa/192c13f5-6395-492f-b10a-4bf78e818a5a.
  [bbr] 2024/07/11 07:35:40 INFO - Unlocking uaa on uaa/7f32628b-9001-4ed0-a829-81c1f614b864...
  [bbr] 2024/07/11 07:36:28 INFO - Finished unlocking uaa on uaa/7f32628b-9001-4ed0-a829-81c1f614b864.
  [bbr] 2024/07/11 07:36:28 INFO - Unlocking credhub on credhub/9208e608-a6de-4f42-a604-1ff658e2409f...
  [bbr] 2024/07/11 07:36:55 INFO - Finished unlocking credhub on credhub/9208e608-a6de-4f42-a604-1ff658e2409f.
  [bbr] 2024/07/11 07:36:55 INFO - Unlocking credhub on credhub/a45f95e6-fd88-450f-ae09-f6f1f3285fb1...
  [bbr] 2024/07/11 07:37:24 INFO - Finished unlocking credhub on credhub/a45f95e6-fd88-450f-ae09-f6f1f3285fb1.
  [bbr] 2024/07/11 07:37:24 INFO - Unlocking cloud_controller_ng on api/9b479d2e-3447-46d0-bfec-1a4428690c3b...
  [bbr] 2024/07/11 07:38:00 INFO - Finished unlocking cloud_controller_ng on api/9b479d2e-3447-46d0-bfec-1a4428690c3b.
  [bbr] 2024/07/11 07:38:00 INFO - Unlocking cloud_controller_ng on api/64d469c8-02e7-49b6-8033-2aa1e6f08568...
  [bbr] 2024/07/11 07:38:39 INFO - Finished unlocking cloud_controller_ng on api/64d469c8-02e7-49b6-8033-2aa1e6f08568.
  [bbr] 2024/07/11 07:38:39 INFO - Unlocking bbr-smbbroker on backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0...
  [TIMEDOUT] in [It] - /tmp/build/b7784222/src/github.com/cloudfoundry/disaster-recovery-acceptance-tests/runner/runner.go:52 @ 07/11/24 08:25:50.249
  STEP: running bbr restore-cleanup @ 07/11/24 08:25:50.25
  [bbr] 2024/07/11 08:25:50 INFO - Looking for scripts
  [bbr] 2024/07/11 08:26:01 INFO - Detected order: credhub should be locked before uaa/uaa during backup
  [bbr] 2024/07/11 08:26:01 INFO - Detected order: credhub should be locked before uaa/uaa during restore
  [bbr] 2024/07/11 08:26:01 INFO - credhub/9208e608-a6de-4f42-a604-1ff658e2409f/credhub/metadata
  [bbr] 2024/07/11 08:26:01 INFO - credhub/9208e608-a6de-4f42-a604-1ff658e2409f/credhub/post-backup-unlock
  [bbr] 2024/07/11 08:26:01 INFO - credhub/9208e608-a6de-4f42-a604-1ff658e2409f/credhub/post-restore-unlock
  [bbr] 2024/07/11 08:26:01 INFO - credhub/9208e608-a6de-4f42-a604-1ff658e2409f/credhub/pre-backup-lock
  [bbr] 2024/07/11 08:26:01 INFO - credhub/9208e608-a6de-4f42-a604-1ff658e2409f/credhub/pre-restore-lock
  [bbr] 2024/07/11 08:26:02 INFO - Detected order: credhub should be locked before uaa/uaa during backup
  [bbr] 2024/07/11 08:26:02 INFO - Detected order: credhub should be locked before uaa/uaa during restore
  [bbr] 2024/07/11 08:26:02 INFO - credhub/a45f95e6-fd88-450f-ae09-f6f1f3285fb1/credhub/metadata
  [bbr] 2024/07/11 08:26:02 INFO - credhub/a45f95e6-fd88-450f-ae09-f6f1f3285fb1/credhub/post-backup-unlock
  [bbr] 2024/07/11 08:26:02 INFO - credhub/a45f95e6-fd88-450f-ae09-f6f1f3285fb1/credhub/post-restore-unlock
  [bbr] 2024/07/11 08:26:02 INFO - credhub/a45f95e6-fd88-450f-ae09-f6f1f3285fb1/credhub/pre-backup-lock
  [bbr] 2024/07/11 08:26:02 INFO - credhub/a45f95e6-fd88-450f-ae09-f6f1f3285fb1/credhub/pre-restore-lock
  [bbr] 2024/07/11 08:26:06 INFO - uaa/192c13f5-6395-492f-b10a-4bf78e818a5a/uaa/post-backup-unlock
  [bbr] 2024/07/11 08:26:06 INFO - uaa/192c13f5-6395-492f-b10a-4bf78e818a5a/uaa/post-restore-unlock
  [bbr] 2024/07/11 08:26:06 INFO - uaa/192c13f5-6395-492f-b10a-4bf78e818a5a/uaa/pre-backup-lock
  [bbr] 2024/07/11 08:26:06 INFO - uaa/192c13f5-6395-492f-b10a-4bf78e818a5a/uaa/pre-restore-lock
  [bbr] 2024/07/11 08:26:07 INFO - uaa/7f32628b-9001-4ed0-a829-81c1f614b864/uaa/post-backup-unlock
  [bbr] 2024/07/11 08:26:07 INFO - uaa/7f32628b-9001-4ed0-a829-81c1f614b864/uaa/post-restore-unlock
  [bbr] 2024/07/11 08:26:07 INFO - uaa/7f32628b-9001-4ed0-a829-81c1f614b864/uaa/pre-backup-lock
  [bbr] 2024/07/11 08:26:07 INFO - uaa/7f32628b-9001-4ed0-a829-81c1f614b864/uaa/pre-restore-lock
  [bbr] 2024/07/11 08:26:08 INFO - cc-worker/bd33ba26-ec79-41ab-9bbb-76ec17b557a6/cloud_controller_worker/post-backup-unlock
  [bbr] 2024/07/11 08:26:08 INFO - cc-worker/bd33ba26-ec79-41ab-9bbb-76ec17b557a6/cloud_controller_worker/post-restore-unlock
  [bbr] 2024/07/11 08:26:08 INFO - cc-worker/bd33ba26-ec79-41ab-9bbb-76ec17b557a6/cloud_controller_worker/pre-backup-lock
  [bbr] 2024/07/11 08:26:08 INFO - cc-worker/bd33ba26-ec79-41ab-9bbb-76ec17b557a6/cloud_controller_worker/pre-restore-lock
  [bbr] 2024/07/11 08:26:09 INFO - cc-worker/e9752676-4e3e-4b8b-8b72-647d91538620/cloud_controller_worker/post-backup-unlock
  [bbr] 2024/07/11 08:26:09 INFO - cc-worker/e9752676-4e3e-4b8b-8b72-647d91538620/cloud_controller_worker/post-restore-unlock
  [bbr] 2024/07/11 08:26:09 INFO - cc-worker/e9752676-4e3e-4b8b-8b72-647d91538620/cloud_controller_worker/pre-backup-lock
  [bbr] 2024/07/11 08:26:09 INFO - cc-worker/e9752676-4e3e-4b8b-8b72-647d91538620/cloud_controller_worker/pre-restore-lock
  [bbr] 2024/07/11 08:26:11 INFO - Detected order: routing-api should be locked before uaa/uaa during restore
  [bbr] 2024/07/11 08:26:11 INFO - api/9b479d2e-3447-46d0-bfec-1a4428690c3b/cloud_controller_ng/post-backup-unlock
  [bbr] 2024/07/11 08:26:11 INFO - api/9b479d2e-3447-46d0-bfec-1a4428690c3b/cloud_controller_ng/post-restore-unlock
  [bbr] 2024/07/11 08:26:11 INFO - api/9b479d2e-3447-46d0-bfec-1a4428690c3b/cloud_controller_ng/pre-backup-lock
  [bbr] 2024/07/11 08:26:11 INFO - api/9b479d2e-3447-46d0-bfec-1a4428690c3b/cloud_controller_ng/pre-restore-lock
  [bbr] 2024/07/11 08:26:11 INFO - api/9b479d2e-3447-46d0-bfec-1a4428690c3b/policy-server/post-backup-unlock
  [bbr] 2024/07/11 08:26:11 INFO - api/9b479d2e-3447-46d0-bfec-1a4428690c3b/policy-server/pre-backup-lock
  [bbr] 2024/07/11 08:26:11 INFO - api/9b479d2e-3447-46d0-bfec-1a4428690c3b/routing-api/metadata
  [bbr] 2024/07/11 08:26:11 INFO - api/9b479d2e-3447-46d0-bfec-1a4428690c3b/routing-api/post-backup-unlock
  [bbr] 2024/07/11 08:26:11 INFO - api/9b479d2e-3447-46d0-bfec-1a4428690c3b/routing-api/post-restore-unlock
  [bbr] 2024/07/11 08:26:11 INFO - api/9b479d2e-3447-46d0-bfec-1a4428690c3b/routing-api/pre-backup-lock
  [bbr] 2024/07/11 08:26:11 INFO - api/9b479d2e-3447-46d0-bfec-1a4428690c3b/routing-api/pre-restore-lock
  [bbr] 2024/07/11 08:26:11 INFO - Detected order: routing-api should be locked before uaa/uaa during restore
  [bbr] 2024/07/11 08:26:11 INFO - api/64d469c8-02e7-49b6-8033-2aa1e6f08568/cloud_controller_ng/post-backup-unlock
  [bbr] 2024/07/11 08:26:11 INFO - api/64d469c8-02e7-49b6-8033-2aa1e6f08568/cloud_controller_ng/post-restore-unlock
  [bbr] 2024/07/11 08:26:11 INFO - api/64d469c8-02e7-49b6-8033-2aa1e6f08568/cloud_controller_ng/pre-backup-lock
  [bbr] 2024/07/11 08:26:11 INFO - api/64d469c8-02e7-49b6-8033-2aa1e6f08568/cloud_controller_ng/pre-restore-lock
  [bbr] 2024/07/11 08:26:11 INFO - api/64d469c8-02e7-49b6-8033-2aa1e6f08568/policy-server/post-backup-unlock
  [bbr] 2024/07/11 08:26:11 INFO - api/64d469c8-02e7-49b6-8033-2aa1e6f08568/policy-server/pre-backup-lock
  [bbr] 2024/07/11 08:26:11 INFO - api/64d469c8-02e7-49b6-8033-2aa1e6f08568/routing-api/metadata
  [bbr] 2024/07/11 08:26:11 INFO - api/64d469c8-02e7-49b6-8033-2aa1e6f08568/routing-api/post-backup-unlock
  [bbr] 2024/07/11 08:26:11 INFO - api/64d469c8-02e7-49b6-8033-2aa1e6f08568/routing-api/post-restore-unlock
  [bbr] 2024/07/11 08:26:11 INFO - api/64d469c8-02e7-49b6-8033-2aa1e6f08568/routing-api/pre-backup-lock
  [bbr] 2024/07/11 08:26:11 INFO - api/64d469c8-02e7-49b6-8033-2aa1e6f08568/routing-api/pre-restore-lock
  [bbr] 2024/07/11 08:26:13 INFO - singleton-blobstore/76a1d847-8bad-45c5-8894-b46f80241a91/blobstore/backup
  [bbr] 2024/07/11 08:26:13 INFO - singleton-blobstore/76a1d847-8bad-45c5-8894-b46f80241a91/blobstore/restore
  [bbr] 2024/07/11 08:26:15 INFO - Detected order: bbr-smbbroker should be locked before uaa/uaa during backup
  [bbr] 2024/07/11 08:26:15 INFO - Detected order: bbr-smbbroker should be locked before credhub/credhub during backup
  [bbr] 2024/07/11 08:26:15 INFO - Detected order: bbr-smbbroker should be locked before capi/cloud_controller_ng during backup
  [bbr] 2024/07/11 08:26:15 INFO - Detected order: bbr-smbbroker should be locked before uaa/uaa during restore
  [bbr] 2024/07/11 08:26:15 INFO - Detected order: bbr-smbbroker should be locked before credhub/credhub during restore
  [bbr] 2024/07/11 08:26:15 INFO - Detected order: bbr-smbbroker should be locked before capi/cloud_controller_ng during restore
  [bbr] 2024/07/11 08:26:15 INFO - Detected order: nfsbroker-bbr-lock should be locked before uaa/uaa during backup
  [bbr] 2024/07/11 08:26:15 INFO - Detected order: nfsbroker-bbr-lock should be locked before credhub/credhub during backup
  [bbr] 2024/07/11 08:26:15 INFO - Detected order: nfsbroker-bbr-lock should be locked before capi/cloud_controller_ng during backup
  [bbr] 2024/07/11 08:26:15 INFO - Detected order: nfsbroker-bbr-lock should be locked before uaa/uaa during restore
  [bbr] 2024/07/11 08:26:15 INFO - Detected order: nfsbroker-bbr-lock should be locked before credhub/credhub during restore
  [bbr] 2024/07/11 08:26:15 INFO - Detected order: nfsbroker-bbr-lock should be locked before capi/cloud_controller_ng during restore
  [bbr] 2024/07/11 08:26:15 INFO - backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0/bbr-cfnetworkingdb/backup
  [bbr] 2024/07/11 08:26:15 INFO - backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0/bbr-cfnetworkingdb/restore
  [bbr] 2024/07/11 08:26:15 INFO - backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0/bbr-cloudcontrollerdb/backup
  [bbr] 2024/07/11 08:26:15 INFO - backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0/bbr-cloudcontrollerdb/restore
  [bbr] 2024/07/11 08:26:15 INFO - backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0/bbr-credhubdb/backup
  [bbr] 2024/07/11 08:26:15 INFO - backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0/bbr-credhubdb/restore
  [bbr] 2024/07/11 08:26:15 INFO - backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0/bbr-routingdb/backup
  [bbr] 2024/07/11 08:26:15 INFO - backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0/bbr-routingdb/restore
  [bbr] 2024/07/11 08:26:15 INFO - backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0/bbr-smbbroker/metadata
  [bbr] 2024/07/11 08:26:15 INFO - backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0/bbr-smbbroker/post-restore-unlock
  [bbr] 2024/07/11 08:26:15 INFO - backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0/bbr-uaadb/backup
  [bbr] 2024/07/11 08:26:15 INFO - backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0/bbr-uaadb/restore
  [bbr] 2024/07/11 08:26:15 INFO - backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0/nfsbroker-bbr-lock/metadata
  [bbr] 2024/07/11 08:26:15 INFO - backup-restore/cb4da01b-4972-4034-b345-5d51fffc8cd0/nfsbroker-bbr-lock/post-restore-unlock
  [bbr] 2024/07/11 08:26:18 INFO - scheduler/04de45af-b64c-48fe-ba1d-83276320641d/cc_deployment_updater/post-backup-unlock
  [bbr] 2024/07/11 08:26:18 INFO - scheduler/04de45af-b64c-48fe-ba1d-83276320641d/cc_deployment_updater/post-restore-unlock
  [bbr] 2024/07/11 08:26:18 INFO - scheduler/04de45af-b64c-48fe-ba1d-83276320641d/cc_deployment_updater/pre-backup-lock
  [bbr] 2024/07/11 08:26:18 INFO - scheduler/04de45af-b64c-48fe-ba1d-83276320641d/cc_deployment_updater/pre-restore-lock
  [bbr] 2024/07/11 08:26:18 INFO - scheduler/04de45af-b64c-48fe-ba1d-83276320641d/cloud_controller_clock/post-backup-unlock
  [bbr] 2024/07/11 08:26:18 INFO - scheduler/04de45af-b64c-48fe-ba1d-83276320641d/cloud_controller_clock/post-restore-unlock
  [bbr] 2024/07/11 08:26:18 INFO - scheduler/04de45af-b64c-48fe-ba1d-83276320641d/cloud_controller_clock/pre-backup-lock
  [bbr] 2024/07/11 08:26:18 INFO - scheduler/04de45af-b64c-48fe-ba1d-83276320641d/cloud_controller_clock/pre-restore-lock
  [bbr] 2024/07/11 08:26:18 INFO - scheduler/04de45af-b64c-48fe-ba1d-83276320641d/tps/post-backup-unlock
  [bbr] 2024/07/11 08:26:18 INFO - scheduler/04de45af-b64c-48fe-ba1d-83276320641d/tps/post-restore-unlock
  [bbr] 2024/07/11 08:26:18 INFO - scheduler/04de45af-b64c-48fe-ba1d-83276320641d/tps/pre-backup-lock
  [bbr] 2024/07/11 08:26:18 INFO - scheduler/04de45af-b64c-48fe-ba1d-83276320641d/tps/pre-restore-lock
  [bbr] 2024/07/11 08:26:19 INFO - scheduler/4c5c1159-8ca7-4729-8977-a222e9041a4c/cc_deployment_updater/post-backup-unlock
  [bbr] 2024/07/11 08:26:19 INFO - scheduler/4c5c1159-8ca7-4729-8977-a222e9041a4c/cc_deployment_updater/post-restore-unlock
  [bbr] 2024/07/11 08:26:19 INFO - scheduler/4c5c1159-8ca7-4729-8977-a222e9041a4c/cc_deployment_updater/pre-backup-lock
  [bbr] 2024/07/11 08:26:19 INFO - scheduler/4c5c1159-8ca7-4729-8977-a222e9041a4c/cc_deployment_updater/pre-restore-lock
  [bbr] 2024/07/11 08:26:19 INFO - scheduler/4c5c1159-8ca7-4729-8977-a222e9041a4c/cloud_controller_clock/post-backup-unlock
  [bbr] 2024/07/11 08:26:19 INFO - scheduler/4c5c1159-8ca7-4729-8977-a222e9041a4c/cloud_controller_clock/post-restore-unlock
  [bbr] 2024/07/11 08:26:19 INFO - scheduler/4c5c1159-8ca7-4729-8977-a222e9041a4c/cloud_controller_clock/pre-backup-lock
  [bbr] 2024/07/11 08:26:19 INFO - scheduler/4c5c1159-8ca7-4729-8977-a222e9041a4c/cloud_controller_clock/pre-restore-lock
  [bbr] 2024/07/11 08:26:19 INFO - scheduler/4c5c1159-8ca7-4729-8977-a222e9041a4c/tps/post-backup-unlock
  [bbr] 2024/07/11 08:26:19 INFO - scheduler/4c5c1159-8ca7-4729-8977-a222e9041a4c/tps/post-restore-unlock
  [bbr] 2024/07/11 08:26:19 INFO - scheduler/4c5c1159-8ca7-4729-8977-a222e9041a4c/tps/pre-backup-lock
  [bbr] 2024/07/11 08:26:19 INFO - scheduler/4c5c1159-8ca7-4729-8977-a222e9041a4c/tps/pre-restore-lock
  [TIMEDOUT] in [AfterEach] - /tmp/build/b7784222/src/github.com/cloudfoundry/disaster-recovery-acceptance-tests/runner/runner.go:174 @ 07/11/24 08:26:20.25
  STEP: cleaning up the artifact @ 07/11/24 08:26:20.251
  STEP: running the Cleanup step for cf-routing @ 07/11/24 08:26:20.262
  STEP: running the Cleanup step for app-uptime @ 07/11/24 08:26:20.317
  STEP: cleaning up orgs, spaces and apps @ 07/11/24 08:26:20.317
  Deleting org acceptance-test-org-85viY2TapHNUxkzPv3eUFA as admin...
  [bbr] 2024/07/11 08:26:21 INFO - Running post-restore-unlock scripts...
  [bbr] 2024/07/11 08:26:21 INFO - Unlocking uaa on uaa/192c13f5-6395-492f-b10a-4bf78e818a5a...
  [bbr] 2024/07/11 08:26:21 INFO - Finished unlocking uaa on uaa/192c13f5-6395-492f-b10a-4bf78e818a5a.
  [bbr] 2024/07/11 08:26:21 INFO - Unlocking uaa on uaa/7f32628b-9001-4ed0-a829-81c1f614b864...
  [bbr] 2024/07/11 08:26:22 INFO - Finished unlocking uaa on uaa/7f32628b-9001-4ed0-a829-81c1f614b864.
  [bbr] 2024/07/11 08:26:22 INFO - Unlocking credhub on credhub/9208e608-a6de-4f42-a604-1ff658e2409f...
  [TIMEDOUT] in [AfterEach] - /tmp/build/b7784222/src/github.com/cloudfoundry/disaster-recovery-acceptance-tests/runner/runner.go:204 @ 07/11/24 08:26:50.318
  STEP: running the Cleanup step for cf-app @ 07/11/24 08:26:50.319
  STEP: cleaning up orgs and spaces @ 07/11/24 08:26:50.319
  Deleting org acceptance-test-org-Jj7qDp5mLE0PN8bFqWNJlw as admin...
  [bbr] 2024/07/11 08:26:54 INFO - Finished unlocking credhub on credhub/9208e608-a6de-4f42-a604-1ff658e2409f.
  [bbr] 2024/07/11 08:26:54 INFO - Unlocking credhub on credhub/a45f95e6-fd88-450f-ae09-f6f1f3285fb1...
  [TIMEDOUT] in [AfterEach] - /tmp/build/b7784222/src/github.com/cloudfoundry/disaster-recovery-acceptance-tests/runner/runner.go:204 @ 07/11/24 08:27:20.319
  STEP: running the Cleanup step for cf-uaa @ 07/11/24 08:27:20.321
  STEP: We delete the user @ 07/11/24 08:27:20.321
  Setting API endpoint to https://api.cf.baba-yaga.env.wg-ard.ci.cloudfoundry.org...
  OK

  API endpoint:   https://api.cf.baba-yaga.env.wg-ard.ci.cloudfoundry.org
  API version:    3.169.0

  Not logged in. Use 'cf login' or 'cf login --sso' to log in.
  API endpoint: https://api.cf.baba-yaga.env.wg-ard.ci.cloudfoundry.org

  Authenticating...
  OK

  Use 'cf target' to view or set your target org and space.
  Deleting user uaa-test-user-DpbrpcUGcTO9GICuROHrYA as admin...
  [bbr] 2024/07/11 08:27:26 INFO - Finished unlocking credhub on credhub/a45f95e6-fd88-450f-ae09-f6f1f3285fb1.
  [bbr] 2024/07/11 08:27:26 INFO - Unlocking cloud_controller_ng on api/9b479d2e-3447-46d0-bfec-1a4428690c3b...
  [TIMEDOUT] in [AfterEach] - /tmp/build/b7784222/src/github.com/cloudfoundry/disaster-recovery-acceptance-tests/runner/runner.go:204 @ 07/11/24 08:27:50.322
  STEP: running the Cleanup step for cf-networking @ 07/11/24 08:27:50.323
  STEP: cleaning up orgs and spaces @ 07/11/24 08:27:50.323
  Deleting org acceptance-test-org-A7sCNPOovD1MyNkgu4mrkA as admin...
  [bbr] 2024/07/11 08:27:56 INFO - Finished unlocking cloud_controller_ng on api/9b479d2e-3447-46d0-bfec-1a4428690c3b.
  [bbr] 2024/07/11 08:27:56 INFO - Unlocking cloud_controller_ng on api/64d469c8-02e7-49b6-8033-2aa1e6f08568...
  [TIMEDOUT] in [AfterEach] - /tmp/build/b7784222/src/github.com/cloudfoundry/disaster-recovery-acceptance-tests/runner/runner.go:204 @ 07/11/24 08:28:20.323
  STEP: running the Cleanup step for cf-nfsbroker @ 07/11/24 08:28:20.325
  STEP: nfs cleanup @ 07/11/24 08:28:20.325
  Deleting org acceptance-test-org-RnUPP5gAFpC-Cl-0aCAfhg as admin...
  [bbr] 2024/07/11 08:28:27 INFO - Finished unlocking cloud_controller_ng on api/64d469c8-02e7-49b6-8033-2aa1e6f08568.
  [bbr] 2024/07/11 08:28:27 INFO - Unlocking cloud_controller_worker on cc-worker/bd33ba26-ec79-41ab-9bbb-76ec17b557a6...
  [bbr] 2024/07/11 08:28:27 INFO - Finished unlocking cloud_controller_worker on cc-worker/bd33ba26-ec79-41ab-9bbb-76ec17b557a6.
  [bbr] 2024/07/11 08:28:27 INFO - Unlocking cloud_controller_worker on cc-worker/e9752676-4e3e-4b8b-8b72-647d91538620...
  [bbr] 2024/07/11 08:28:28 INFO - Finished unlocking cloud_controller_worker on cc-worker/e9752676-4e3e-4b8b-8b72-647d91538620.
  [bbr] 2024/07/11 08:28:28 INFO - Unlocking routing-api on api/9b479d2e-3447-46d0-bfec-1a4428690c3b...
  OK

  TIP: No org or space targeted, use 'cf target -o ORG -s SPACE' to target an org and space.
  OK

  TIP: No org or space targeted, use 'cf target -o ORG -s SPACE' to target an org and space.
  OK

  TIP: No org or space targeted, use 'cf target -o ORG -s SPACE' to target an org and space.
  OK

  OK

  TIP: No org or space targeted, use 'cf target -o ORG -s SPACE' to target an org and space.
  STEP: running the Cleanup step for cf-smbbroker @ 07/11/24 08:28:38.904
  STEP: smb cleanup @ 07/11/24 08:28:38.904
  Deleting org acceptance-test-org-GPIym_GoApw4RLLohpnq5w as admin...
  OK

  TIP: No org or space targeted, use 'cf target -o ORG -s SPACE' to target an org and space.
  STEP: running the Cleanup step for cf-credhub @ 07/11/24 08:28:42.197
  Deleting org acceptance-test-org-EFh0TbTJ3zUAjZdSBQZBqA as admin...
  OK

  TIP: No org or space targeted, use 'cf target -o ORG -s SPACE' to target an org and space.
  STEP: cleaning up the test context @ 07/11/24 08:28:48.774
  STEP: remove workspace directory @ 07/11/24 08:28:48.774
• [TIMEDOUT] [3767.760 seconds]
backing up Cloud Foundry [It] backs up and restores a CF
/tmp/build/b7784222/src/github.com/cloudfoundry/disaster-recovery-acceptance-tests/runner/runner.go:52

  [TIMEDOUT] A suite timeout occurred
  In [It] at: /tmp/build/b7784222/src/github.com/cloudfoundry/disaster-recovery-acceptance-tests/runner/runner.go:52 @ 07/11/24 08:25:50.249

  Full Stack Trace

  This is the Progress Report generated when the suite timeout occurred:
    backing up Cloud Foundry backs up and restores a CF (Spec Runtime: 59m49.211s)
      /tmp/build/b7784222/src/github.com/cloudfoundry/disaster-recovery-acceptance-tests/runner/runner.go:52
      In [It] (Node Runtime: 59m47.807s)
        /tmp/build/b7784222/src/github.com/cloudfoundry/disaster-recovery-acceptance-tests/runner/runner.go:52
        At [By Step] restoring to cf (Step Runtime: 52m58.806s)
          /tmp/build/b7784222/src/github.com/cloudfoundry/disaster-recovery-acceptance-tests/runner/runner.go:119

        Spec Goroutine
        goroutine 22 [select]
          github.com/onsi/gomega/internal.(*AsyncAssertion).match(0xc0000c22a0, {0x91ec98, 0xc00033c210}, 0x1, {0xc0002aa600, 0x1, 0x1})
            /tmp/build/b7784222/pkg/mod/github.com/onsi/gomega@v1.33.1/internal/async_assertion.go:538
          github.com/onsi/gomega/internal.(*AsyncAssertion).Should(0xc0000c22a0, {0x91ec98, 0xc00033c210}, {0xc0002aa600, 0x1, 0x1})
            /tmp/build/b7784222/pkg/mod/github.com/onsi/gomega@v1.33.1/internal/async_assertion.go:145
        > github.com/cloudfoundry/disaster-recovery-acceptance-tests/runner.runCommandWithStream({0x867374, 0x16}, {0x91c4e0, 0xc0000808a0}, {0x91c4e0, 0xc0000808a0}, {0xc0005021a0, 0x187}, {0x0, 0x0, ...})
            /tmp/build/b7784222/src/github.com/cloudfoundry/disaster-recovery-acceptance-tests/runner/command_line_helpers.go:65
              | 
              |     Expect(err).ToNot(HaveOccurred())
              >     Eventually(session).Should(gexec.Exit(), "Command timed out: "+commandDescription)
              |     return session
              | }
        > github.com/cloudfoundry/disaster-recovery-acceptance-tests/runner.RunCommandSuccessfullyWithFailureMessage({0x867374, 0x16}, {0xc0005021a0, 0x187}, {0x0, 0x0, 0x0})
            /tmp/build/b7784222/src/github.com/cloudfoundry/disaster-recovery-acceptance-tests/runner/command_line_helpers.go:31
              | 
              | func RunCommandSuccessfullyWithFailureMessage(commandDescription, cmd string, args ...string) *gexec.Session {
              >     session := runCommandWithStream(commandDescription, GinkgoWriter, GinkgoWriter, cmd, args...)
              |     Expect(session).To(gexec.Exit(0), "Command errored: "+commandDescription)
              |     return session
        > github.com/cloudfoundry/disaster-recovery-acceptance-tests/runner.RunDisasterRecoveryAcceptanceTests.func2()
            /tmp/build/b7784222/src/github.com/cloudfoundry/disaster-recovery-acceptance-tests/runner/runner.go:120
              | restoreRunning = true
              | By("restoring to " + config.CloudFoundryConfig.Name)
              > RunCommandSuccessfullyWithFailureMessage(
              |     "bbr deployment restore",
              |     fmt.Sprintf(
          github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func3({0x0?, 0x0?})
            /tmp/build/b7784222/pkg/mod/github.com/onsi/ginkgo/v2@v2.19.0/internal/node.go:472
          github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
            /tmp/build/b7784222/pkg/mod/github.com/onsi/ginkgo/v2@v2.19.0/internal/suite.go:894
          github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 6
            /tmp/build/b7784222/pkg/mod/github.com/onsi/ginkgo/v2@v2.19.0/internal/suite.go:881

  There were additional failures detected.  To view them in detail run ginkgo -vv
------------------------------

Summarizing 1 Failure:
  [TIMEDOUT] backing up Cloud Foundry [It] backs up and restores a CF
  /tmp/build/b7784222/src/github.com/cloudfoundry/disaster-recovery-acceptance-tests/runner/runner.go:52

Ran 1 of 1 Specs in 3767.761 seconds
FAIL! - Suite Timeout Elapsed -- 0 Passed | 1 Failed | 0 Pending | 0 Skipped
--- FAIL: TestPcfBackupAndRestoreAcceptanceTests (3767.76s)
FAIL

Ginkgo ran 1 suite in 1h2m58.56067817s

Test Suite Failed
exit status 1
jochenehret commented 3 months ago

Test has failed again: https://concourse.wg-ard.ci.cloudfoundry.org/teams/main/pipelines/cf-deployment/jobs/bbr-run-drats/builds/1448

I can give a few observations. If the test is green, it takes around 20 minutes to complete. If it fails as above, it runs into the global 60 minutes timeout. It seems that the test gets stuck at the post-restore unlock "bbr-smbbroker" script:

  [bbr] 2024/08/21 14:53:40 INFO - Unlocking bbr-smbbroker on backup-restore/53c514ae-0782-4928-a42e-26090511cb4b...
  [TIMEDOUT] in [It] - /tmp/build/b7784222/src/github.com/cloudfoundry/disaster-recovery-acceptance-tests/runner/runner.go:52 @ 08/21/24 15:40:28.903
  STEP: running bbr restore-cleanup @ 08/21/24 15:40:28.908
  [bbr] 2024/08/21 15:40:29 INFO - Looking for scripts
  [bbr] 2024/08/21 15:40:32 INFO - cc-worker/3c5c1c65-ce0a-49d7-8f94-a5ea9345d456/cloud_controller_worker/post-backup-unlock

I've checked the "backup-restore" VM of the test "cf" deployment. The script produces no logs:

# ll /var/vcap/sys/log/bbr-smbbroker/
total 8
drwxrwx---  2 root vcap 4096 Aug 21 14:22 ./
drwxr-x--- 20 root vcap 4096 Aug 21 14:22 ../

I could execute the script manually and it succeeded after a short time:

/var/vcap/jobs/bbr-smbbroker# ./bin/bbr/post-restore-unlock
+ PATH=/var/vcap/packages/cf-cli-8-linux/bin:/var/vcap/bosh/bin:/var/vcap/jobs/bpm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/var/vcap/bosh/bin
+ API_ENDPOINT=https://api.cf.baba-yaga.env.wg-ard.ci.cloudfoundry.org
+ ORG=system
+ SPACE=smb-broker-space
+ APP_NAME=smbbroker
+ cf_auth_and_target
++ date +%Y-%m-%dT%H:%M:%S.%NZ
+ echo '2024-08-22T07:01:59.088103921Z: Authenticate and target...'
2024-08-22T07:01:59.088103921Z: Authenticate and target...
+ cf api https://api.cf.baba-yaga.env.wg-ard.ci.cloudfoundry.org --skip-ssl-validation
Setting API endpoint to https://api.cf.baba-yaga.env.wg-ard.ci.cloudfoundry.org...
OK
(...)

So as a first debug step, should we make sure that the "unlock" scripts write their logs to /var/vcap/sys/log?

jochenehret commented 3 months ago

I still don't have a full understanding of the restore process, but here are some more observations: This is the post-restore-unlock sequence in a successful restore procedure:

  [bbr] 2024/08/27 17:09:45 INFO - Completed restore of cf
  [bbr] 2024/08/27 17:09:45 INFO - Running post-restore-unlock scripts...
  [bbr] 2024/08/27 17:09:45 INFO - Unlocking uaa on uaa/fc557801-fbd9-4afa-852c-f283b901c18a...
  [bbr] 2024/08/27 17:10:32 INFO - Finished unlocking uaa on uaa/fc557801-fbd9-4afa-852c-f283b901c18a.
  [bbr] 2024/08/27 17:10:32 INFO - Unlocking uaa on uaa/3a055696-2ee0-41f4-ba3a-c2ee8563dad9...
  [bbr] 2024/08/27 17:11:20 INFO - Finished unlocking uaa on uaa/3a055696-2ee0-41f4-ba3a-c2ee8563dad9.
  [bbr] 2024/08/27 17:11:20 INFO - Unlocking cloud_controller_ng on api/fe489e35-ff83-4289-a637-0037581e1fa8...
  [bbr] 2024/08/27 17:11:56 INFO - Finished unlocking cloud_controller_ng on api/fe489e35-ff83-4289-a637-0037581e1fa8.
  [bbr] 2024/08/27 17:11:56 INFO - Unlocking cloud_controller_ng on api/ba02f73a-ee67-41da-9368-f8955f18f15f...
  [bbr] 2024/08/27 17:12:34 INFO - Finished unlocking cloud_controller_ng on api/ba02f73a-ee67-41da-9368-f8955f18f15f.
  [bbr] 2024/08/27 17:12:34 INFO - Unlocking credhub on credhub/00617f00-144f-48ae-a625-e10ec11f8274...
  [bbr] 2024/08/27 17:13:07 INFO - Finished unlocking credhub on credhub/00617f00-144f-48ae-a625-e10ec11f8274.
  [bbr] 2024/08/27 17:13:07 INFO - Unlocking credhub on credhub/1ca53a62-ea0f-4abb-964d-dd375cfd9639...
  [bbr] 2024/08/27 17:13:35 INFO - Finished unlocking credhub on credhub/1ca53a62-ea0f-4abb-964d-dd375cfd9639.
  [bbr] 2024/08/27 17:13:35 INFO - Unlocking cloud_controller_worker on cc-worker/7ba19cd5-7e56-4c5a-bd8a-4424c63efff9...
  [bbr] 2024/08/27 17:13:35 INFO - Finished unlocking cloud_controller_worker on cc-worker/7ba19cd5-7e56-4c5a-bd8a-4424c63efff9.
  [bbr] 2024/08/27 17:13:35 INFO - Unlocking cloud_controller_worker on cc-worker/aef5b90c-3688-47de-ab13-0f0c4147ea8c...
  [bbr] 2024/08/27 17:13:36 INFO - Finished unlocking cloud_controller_worker on cc-worker/aef5b90c-3688-47de-ab13-0f0c4147ea8c.
  [bbr] 2024/08/27 17:13:36 INFO - Unlocking routing-api on api/fe489e35-ff83-4289-a637-0037581e1fa8...
  [bbr] 2024/08/27 17:14:06 INFO - Finished unlocking routing-api on api/fe489e35-ff83-4289-a637-0037581e1fa8.
  [bbr] 2024/08/27 17:14:06 INFO - Unlocking routing-api on api/ba02f73a-ee67-41da-9368-f8955f18f15f...
  [bbr] 2024/08/27 17:14:36 INFO - Finished unlocking routing-api on api/ba02f73a-ee67-41da-9368-f8955f18f15f.
  [bbr] 2024/08/27 17:14:36 INFO - Unlocking tps on scheduler/52c38003-f17e-43cb-b406-daded6a085b2...
  [bbr] 2024/08/27 17:14:37 INFO - Finished unlocking tps on scheduler/52c38003-f17e-43cb-b406-daded6a085b2.
  [bbr] 2024/08/27 17:14:37 INFO - Unlocking cc_deployment_updater on scheduler/52c38003-f17e-43cb-b406-daded6a085b2...
  [bbr] 2024/08/27 17:14:37 INFO - Finished unlocking cc_deployment_updater on scheduler/52c38003-f17e-43cb-b406-daded6a085b2.
  [bbr] 2024/08/27 17:14:37 INFO - Unlocking cloud_controller_clock on scheduler/52c38003-f17e-43cb-b406-daded6a085b2...
  [bbr] 2024/08/27 17:14:37 INFO - Finished unlocking cloud_controller_clock on scheduler/52c38003-f17e-43cb-b406-daded6a085b2.
  [bbr] 2024/08/27 17:14:37 INFO - Unlocking cc_deployment_updater on scheduler/fa821d84-b357-4617-a215-1c8b4865ac87...
  [bbr] 2024/08/27 17:14:37 INFO - Finished unlocking cc_deployment_updater on scheduler/fa821d84-b357-4617-a215-1c8b4865ac87.
  [bbr] 2024/08/27 17:14:37 INFO - Unlocking cloud_controller_clock on scheduler/fa821d84-b357-4617-a215-1c8b4865ac87...
  [bbr] 2024/08/27 17:14:38 INFO - Finished unlocking cloud_controller_clock on scheduler/fa821d84-b357-4617-a215-1c8b4865ac87.
  [bbr] 2024/08/27 17:14:38 INFO - Unlocking tps on scheduler/fa821d84-b357-4617-a215-1c8b4865ac87...
  [bbr] 2024/08/27 17:14:38 INFO - Finished unlocking tps on scheduler/fa821d84-b357-4617-a215-1c8b4865ac87.
  [bbr] 2024/08/27 17:14:38 INFO - Unlocking bbr-smbbroker on backup-restore/9489e7eb-de5c-4db1-ad29-68a71334ddf5...
  [bbr] 2024/08/27 17:15:04 INFO - Finished unlocking bbr-smbbroker on backup-restore/9489e7eb-de5c-4db1-ad29-68a71334ddf5.
  [bbr] 2024/08/27 17:15:04 INFO - Unlocking nfsbroker-bbr-lock on backup-restore/9489e7eb-de5c-4db1-ad29-68a71334ddf5...
  [bbr] 2024/08/27 17:15:37 INFO - Finished unlocking nfsbroker-bbr-lock on backup-restore/9489e7eb-de5c-4db1-ad29-68a71334ddf5.
  [bbr] 2024/08/27 17:15:37 INFO - Finished running post-restore-unlock scripts.

Note that the jobs on the "api", "cc-worker" and "scheduler" VMs are unlocked first. The "bbr-smbbroker" and "nfsbroker-bbr-lock" jobs are unlocked last. Those perform (to my knowledge) a "cf push" for the broker apps which succeeds.

This is the post-restore-unlock sequence in a failed restore procedure:

  [bbr] 2024/08/27 18:45:14 INFO - Completed restore of cf
  [bbr] 2024/08/27 18:45:14 INFO - Running post-restore-unlock scripts...
  [bbr] 2024/08/27 18:45:14 INFO - Unlocking uaa on uaa/3147e571-e4a3-4e3b-a47a-70c7dbf01cf9...
  [bbr] 2024/08/27 18:46:01 INFO - Finished unlocking uaa on uaa/3147e571-e4a3-4e3b-a47a-70c7dbf01cf9.
  [bbr] 2024/08/27 18:46:01 INFO - Unlocking uaa on uaa/cca553d2-cdc0-47fb-b471-56374da3649c...
  [bbr] 2024/08/27 18:46:47 INFO - Finished unlocking uaa on uaa/cca553d2-cdc0-47fb-b471-56374da3649c.
  [bbr] 2024/08/27 18:46:47 INFO - Unlocking cloud_controller_ng on api/2af9f60b-6c7c-4429-a435-995c8e0d8e2e...
  [bbr] 2024/08/27 18:47:24 INFO - Finished unlocking cloud_controller_ng on api/2af9f60b-6c7c-4429-a435-995c8e0d8e2e.
  [bbr] 2024/08/27 18:47:24 INFO - Unlocking cloud_controller_ng on api/8cbb8175-fe00-421f-874e-f1db1e4446a1...
  [bbr] 2024/08/27 18:48:02 INFO - Finished unlocking cloud_controller_ng on api/8cbb8175-fe00-421f-874e-f1db1e4446a1.
  [bbr] 2024/08/27 18:48:02 INFO - Unlocking credhub on credhub/3931a29e-494b-4730-a3f8-9c48ad3b6fd8...
  [bbr] 2024/08/27 18:48:31 INFO - Finished unlocking credhub on credhub/3931a29e-494b-4730-a3f8-9c48ad3b6fd8.
  [bbr] 2024/08/27 18:48:31 INFO - Unlocking credhub on credhub/5234be8f-8e93-4356-85d2-3fbd9acb5ce2...
  [bbr] 2024/08/27 18:48:59 INFO - Finished unlocking credhub on credhub/5234be8f-8e93-4356-85d2-3fbd9acb5ce2.
  [bbr] 2024/08/27 18:48:59 INFO - Unlocking nfsbroker-bbr-lock on backup-restore/bda52edc-0792-4aa3-ae72-c364a5b432a9...

Here the "nfsbroker-bbr-lock" is being unlocked before all cloud controller jobs are unlocked. So the "cf push" for the nfs broker app cannot be executed and the unlock script hangs until the test timeout is reached. By restarting the cloud controller jobs on "api", "cc-worker" and "scheduler" manually, I managed to make the "nfsbroker-bbr-lock" script continue.

Looks like some dependencies for the "nfsbroker" and "smbbroker" unlocking scripts are not set correctly?