Detect early and recover better from invalid git config (e.g. service instance branch with custom model rebased against model)

[gberche-orange]

As a coab operator

• in order to avoid CC orphan mitigation process to loop on asking to delete service instance
• in order to be notifed early if git config is invalid (in our case, the coab smoke test was red)
• I need an invalid git config to prevent coab from starting.

See https://github.com/orange-cloudfoundry/paas-templates/issues/844#issuecomment-670432493 for details of such occurence.

In particular, the coab templates instance branches were incorrectly rebased against develop instead of a custom model branch.

Ways to detect in advance failure to push to templates instance branch:

• clone and push a dummy commit to the templates instance branch, e.g. /coab-git-health-checks/health-check-timestamp.txt.

Alternatives:

• focus instead on recovery of paas-templates failure to avoid having the COA pipeline become red for all service instances
  • [ ] order the git operations to have secrets operations done after paas-templates. Initial reason in com.orange.oss.cloudfoundry.broker.opsautomation.ondemandbroker.sample.BoshBrokerApplication#processorChain() is not relevant anymore following git pooling and relative propagation delay in COA processing of secrets (10s) and templates (120-300s) after the detection of the git commit (between 1 and 60 seconds)
    • [ ] Protect against side effect with COA behavior using load and concurrency tests using noop
      • Cases of concurrent service instance:
      • coab and osb-cmdb smoke tests with distinct service offerings
      • user pipelines ?
  • git revert the secret commit upon failure of the templates commit

        // git push will trigger 1st for paas-templates and then 2nd for paas-secrets,
        // reducing occurences of fail-fast consistency check failures
        // see related https://github.com/orange-cloudfoundry/cf-ops-automation/issues/201
        processors.add(secretsGitProcessor);
        processors.add(templateGitProcessor);
        processors.add(boshProcessor);

[gberche-orange]

Possible design done with @JCL38-ORANGE :

• Perform a git clone on both secrets and template repos at spring bean initialization
  • [ ] In GitProcessor, be provided in the constructor, the necessary Context with associated git keys to perform the clone in preConstruct and the cleanup in preDelete.
    • This context is currently created in com.orange.oss.cloudfoundry.broker.opsautomation.ondemandbroker.sample.BoshBrokerApplication#paasTemplateContextFilter() inner class DefaultBrokerProcessor()

      private void registerPaasTemplatesBranches(Context ctx) {
          ctx.contextKeys.put(TEMPLATES_REPOSITORY_ALIAS_NAME + GitProcessorContext.checkOutRemoteBranch.toString(), pipelineProperties.getCheckOutRemoteBranch()); //"develop"
          ctx.contextKeys.put(TEMPLATES_REPOSITORY_ALIAS_NAME + GitProcessorContext.createBranchIfMissing.toString(), pipelineProperties.getCreateBranchIfMissing()); //"feature-coadepls-cassandra-serviceinstances"
      }

    • [ ] refactor DefaultBrokerProcessor to move up registerPaasTemplatesBranches() as method of BoshBrokerApplication shared with secretsGitProcessor() and templateGitProcessor()
    • update broken tests
    • [ ] test this feature: new GitProcessorSpringBeanTest that uses a SpringContext to invoke preconstruct and postdelete
      • clones_on_preconstruct:
      • given a clone issue: simulated with a SimpleGitManager mock throwing a realistic exception
      • when spring context is created
      • then the spring context loading fails with a message explaining the clone failure
      • cleans_up_on_postdelete
      • given a successfull clone issue: simulated with a SimpleGitManager doing noop
      • given a successfull cleanup: simulated with a SimpleGitManager doing noop
      • when spring context is created and deleted
      • then SimpleGitManager clone and cleanup methods were called exactly once.