opentelekomcloud_compute_instance_v2 stop_before_destroy custom shutdown timeout

[econsult-devops]

Hello,

Our services run on ECS instances, and we need them to shut down gracefully to prevent disruptions within their clusters. This is currently accomplished by using the _terraformdata resource, which runs a shutdown script on the server remotely. We do this just before initiating server destruction.

We want to reduce SSH access to our servers and keep our Terraform projects and modules streamlined. That's why, we're looking for an alternative to remote scripts to handle graceful shutdowns before destruction.

In the opentelekomcloud_compute_instance_v2 resource, there is an optional stop_before_destroy parameter, which stops the server before destroying it. The shutdown does not provide enough time for our services to complete their shutdown process. Our service has a 10-minute shutdown timeout, but the server is often destroyed prematurely, disrupting the cluster synchronization between members.

Request:

Would it be possible to add a configurable delay option to the stop_before_destroy parameter, allowing us to set the required timeout for graceful shutdowns before the server is fully destroyed?

[anton-sidelnikov]

Hi @econsult-devops The stop_before_destroy relies on the server’s status and is considered complete when the status is SHUTOFF. Your request sounds more like a new feature for the resource that would allow running scripts before the server is deleted, with the deletion occurring only after these scripts have completed successfully. You could ask customer support for this feature.

Another option, is to put shutdown script to machines like:

# /etc/systemd/system/graceful-shutdown.service
# This service will execute your script during the shutdown process.
[Unit]
Description=Graceful shutdown service
DefaultDependencies=no
Before=shutdown.target reboot.target halt.target

[Service]
Type=oneshot
ExecStart=/opt/shutdown-script.sh
RemainAfterExit=true

[Install]
WantedBy=halt.target reboot.target shutdown.target

it possible with user_data or custom image

[econsult-devops]

Hi @anton-sidelnikov, thanks for your response.

It seems there was a misunderstanding. We don’t need a script to run before server deletion. Instead, we need the instance to avoid destruction until the server has fully shut down.

Our service is already configured for a graceful shutdown, and systemd has a sufficient timeout to prevent forced termination during shutdown. However, the instance still seems to be destroyed before the service stops and the instance powers off completely.

I opened this issue based on a note in the opentelekomcloud_compute_instance_v2 documentation:

stop_before_destroy - (Optional) Whether to try stopping the instance gracefully before destroying it, thus giving the guest OS daemons a chance to stop correctly. If the instance doesn’t stop within a timeout, it will be destroyed anyway.

The last sentence suggests there’s a timeout before the instance is forcefully destroyed. Could you confirm if this behavior is documented correctly, and if so, is there a way to increase the timeout period?

[anton-sidelnikov]

Hi @econsult-devops, yes i checked the code and provide update in #2703, which would be available in next release 14.11.2024.

[econsult-devops]

That's great, thanks a lot!