inf-terraform-aws - preserve errored.tfstate file in case of write failure in make deploy

[tbugfinder]

In case terraform cannot upload the state file to the configured bucket it writes it to a file errored.tfstate. It is important to preserve this file so that manual error handling can be done (e.g. upload file manually).

In addition it might be beneficial to check before that permissions and infrastructure setup is properly configured for running the terraform deploy command.

https://github.com/opendevstack/ods-quickstarters/blob/35ca70fa8dd327e227fc66676fa4eaf3eb05faa5/inf-terraform-aws/Jenkinsfile.template#L227-L233

FYI: @nichtraunzer

[tbugfinder]

Error message:

aws_cloudformation_stack.cft-s3: Still creating... [30s elapsed]
aws_cloudformation_stack.cft-s3: Creation complete after 37s [id=arn:aws:cloudformation:<snip>
Error: Failed to save state

Error saving state: failed to upload state: AccessDenied: Access Denied
    status code: 403, request id: <snip>
Error: Failed to persist state to backend

The error shown above has prevented Terraform from writing the updated state
to the configured backend. To allow for recovery, the state has been written
to the file "errored.tfstate" in the current working di

[nichtraunzer]

In case terraform cannot upload the state file to the configured bucket it writes it to a file errored.tfstate. It is important to preserve this file so that manual error handling can be done (e.g. upload file manually).

• can u describe what u mean by "preserve" ? (how long, where should it be preserved, ...)

I assume this error occurs because the policy for the shared S3 bucket has not been updated for the given account. I also assume this error will not occur when we switch to dedicated S3 buckets to store the TF state for each account -am I right ?

[tbugfinder]

The error was raised as the bucket policy wasn't set properly. I cannot tell all error conditions which could come up nevertheless if terraform errors out and leaves a file errored.tfstate on the ephemeral jenkins agent the this file should be uploaded within the pipeline run as an artifact.

[tbugfinder]

https://www.jenkins.io/doc/pipeline/steps/core/

==>

archiveArtifacts artifacts: 'errored.tfstate',
                   allowEmptyArchive: true

[nichtraunzer]

I would like to see this feature into the new InfrastructureService provided by the Jenkins Shared Library #740 - draft should be available by early next week.