Failed to restage management ssb-solrcloud

[FuhuXia]

Running routing restage commands on all space apps. Seeing error for management:ssb-solrcloud

cf target -o gsa-datagov -s management ; cf restage --strategy rolling ssb-solrcloud

Error in cf logs ssb-solrcloud --recent:

...
ERR {"timestamp":"1715115467.353290558","source":"cloud-service-broker","message":"cloud-service-broker.Database Setup","log_level":2,"data":{"error":"DB_HOST, DB_USERNAME and DB_PASSWORD are required environment variables"}}
OUT Exit status 1
...

It is fine for management-staging ssb-solrcloud

[btylerburton]

are DB_HOST, DB_USERNAME and DB_PASSWORD available as env vars in the application?

[FuhuXia]

Yes they are defined in VCAP_SERVICES cf env. In management-staging the same error was also observed, but the action has retry logic built in, and it was successful after retry. management-prod never get successful result with retry. I can run a few more times to see if it gives any good result. Or I can unbind and rebind the mysql db to see if it make any difference.

[FuhuXia]

The last restage attemp was successful.

[linux ~]# cf restage --strategy rolling ssb-solrcloud

...

name:              ssb-solrcloud
requested state:   started
routes:            ssb-solrcloud-gsa-datagov-management.app.cloud.gov
last uploaded:     Thu 09 May 11:30:23 EDT 2024
stack:             cflinuxfs4
buildpacks:        
    name               version   detect output   buildpack name
    binary_buildpack   1.1.11    binary          binary

type:           web
sidecars:       
instances:      1/1
memory usage:   1024M
     state     since                  cpu    memory        disk           details
#0   running   2024-05-09T15:33:35Z   0.0%   22.9M of 1G   823.5M of 2G   

[btylerburton]

Anything different in the steps you took before or the process output?

[FuhuXia]

Nope. It was the last try (after 5-7 attempts in total of the same command) before I move on to the next unbind-rebind way. I guess it tells us how glitchy the whole thing is.