search

hashicorp / terraform

Terraform enables you to safely and predictably create, change, and improve infrastructure. It is a source-available tool that codifies APIs into declarative configuration files that can be shared amongst team members, treated as code, edited, reviewed, and versioned.
https://www.terraform.io/
Other
42.6k stars 9.54k forks source link

Dynamo Db Lock Table TTLs for locks #21671

Open asaf-erlich opened 5 years ago

asaf-erlich commented 5 years ago

Current Terraform Version

Terraform v0.11.13

Use-cases

Terraform creates aws resources. Sometimes the job which executes terraform crashes (in our particular use case a Jenkins job might crash during Jenkins master restart, which is a bad architecture design of Jenkins but that's besides the point, the terraform process can crash randomly either way). From that point on the lock in dynamo db will be held forever or until manual action is taken to release the lock.

We have a cleanup job that runs hours later (executes terraform destroy for all the modules) that would normally clean up these aws resources, but it will fail because the lock is held forever (unless manual action is taken).

Attempted Solutions

At the moment every time this fails someone has to manually go into dyanamo db in aws using either the console or cli and delete the locked resource lockID item. Then when the job to clean up resource is executed it works.

Proposal

Dynamo db tables already offer a way to add a row / value with a TTL: https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/time-to-live-ttl-how-to.html

It would be very nice if this could be just something that is configured. My suggestion is just to add a value for ttl inside the terraform backend configuration, like dynamodb_lock_ttl. From an implementation point of view if the value is not set for an existing table and it needs to be enabled this could be problematic. But terraform could just throw a warning that it only supports ttl for newly created tables.

References

Potentially this is related: https://github.com/hashicorp/terraform/issues/15429

Let me know if you need more information. Thank you, -Asaf

djrodgerspryor commented 3 years ago

Would you be open to a contribution to add this feature?

asaf-erlich commented 3 years ago

@djrodgerspryor I would be open to it in perhaps a few months. My wife and I have a 2 month old and I just don't have time available outside of work to do something like this right now. But I think it's an awesome feature and someone should really do it. I'll try to keep this in mind when I feel like I have the time.

dawez commented 2 years ago

@asaf-erlich do you have any progess on this?

asaf-erlich commented 2 years ago

No I apologize. The pandemic + having a baby in the last year is making the feasibility of my working on this extremely unlikely.

billcchung commented 1 year ago

@asaf-erlich we're having this issue as well, and I'm happy to contribute to this if it's not yet started by you.

asaf-erlich commented 1 year ago

Hello @billcchung, I never made any progress on this issue. Feel free to make any contribution if you have the bandwidth to do so.

billcchung commented 1 year ago

hi @asaf-erlich I've opened a PR https://github.com/hashicorp/terraform/pull/32287 for this, would you mind taking a look?

asaf-erlich commented 1 year ago

Hello @billcchung , I can try to take a look either this week or next week and do my best to provide feedback. I would like to point out that I'm not a maintainer of terraform nor have I ever written code for this code base. Just saw that 2 other people already responded on the PR so hopefully that is not an issue. Either way I appreciate you submitting this PR. Thank you.

raphaelbittan commented 9 months ago

Has this ever been resolved ?

asaf-erlich commented 9 months ago

Unfortunately the draft pr was never finished.

raphaelbittan commented 9 months ago

Thanks ☺️