Implement terraform state locking in S3 backends - Githubissues

ministryofjustice / modernisation-platform

A place for the core work of the Modernisation Platform • This repository is defined and managed in Terraform

https://user-guide.modernisation-platform.service.justice.gov.uk

MIT License

680 stars 289 forks source link

Implement terraform state locking in S3 backends #2467

Closed dms1981 closed 2 years ago

dms1981 commented 2 years ago

User Story

As a Modernisation Platform Engineer I need to implement backend locking of the terraform statefile So that only one agent can modify the statefile at any given time

User Type(s)

Value

Without state file locking, it's possible for two jobs to access the statefile at the same time, leading to the creation of terraform statefile versions which are not in agreement with each other. In the case of a new account creation, this can lead to a state where terraform jobs cannot successfully run due to missing information.

Also, given our approach to work where many hands can be busy at one time, state file locking is an ideal solution to prevent any problems with clashing workflows.

Questions / Assumptions / Hypothesis

Proposal

We should implement a DynamoDB table and the necessary configuration elements specified by Hashicorp in the S3 backend type documentation

Environments

[x] apex
[x] ccms-ebs
[x] cooker
[x] core-logging
[x] core-security
[x] core-vpc
[x] data-and-insights-wepi
[x] delius-iaps
[x] digital-prison-reporting
[x] equip
[x] example
[x] maatdb
[x] main.tf
[x] mlra
[x] nomis
[x] oasys
[x] performance-hub
[x] ppud
[x] refer-monitor
[x] sprinkler
[x] testing
[x] threat-and-vulnerability-mgmt
[x] xhibit-portal

Definition of done

[x] State file locking has been implemented across our backend.tf files
[x] another team member has reviewed

Reference

How to write good user stories Backend type: S3

SteveLinden commented 2 years ago

Initial PR put in place for review

SteveLinden commented 2 years ago

I think this is now in place for the core-vpc code and needs to be tested. The above has been edited to list all the environments that have a backend.tf which probably need to be amended.

SteveLinden commented 2 years ago

core-vpc and core-security completed (individual PRs) (e.g. #2552) Others will be done in one PR

SteveLinden commented 2 years ago

Changes made for the above but I am now going through them to see which do not work. bichard7 is the first culprit, and I will remove them from the list above and try to take them out of the PR

SteveLinden commented 2 years ago

Non-working items to remove from PR

bichard7 - Error refreshing state: InvalidObjectState: The operation is not valid for the object's storage class analytical-platform-data - Error refreshing state: InvalidObjectState: The operation is not valid for the object's storage class analytical-platform-management - Error refreshing state: InvalidObjectState: The operation is not valid for the object's storage
Bootstrap/delegate-access - not included Bootstrap/secure-baselines - not included Core-network-services - S3 issues Core-sandbox - S3 issues core-shared-services - S3 issues Mi-platform - S3 issues Remote-supervision - storage class issues

Sprinkler has been left in place but it indicates it will destroy 9 items

SteveLinden commented 2 years ago

plan on core-sandbox indicates it will destroy 10 items so this is being left out. Added core-network-services, core-shared-services and mi-platform

SteveLinden commented 2 years ago

Has been applied