Handle `SIGTERM` messages to shut down gracefully #129

Open MeltyBot opened 3 years ago

Migrated from GitLab: https://gitlab.com/meltano/sdk/-/issues/130

Originally created by @aaronsteers on 2021-05-17 18:16:11

Summary

As brought up in this slack conversation, we have an opportunity to handle SIGTERM messages gracefully.

Proposed benefits

This could improve SDK-based taps' functionality when being told to shut down gracefully.

Proposal details

For any sorted incremental streams: we can flush the current stream and cancel syncing any other records.
For non-sorted streams and/or streams which do not support incremental refresh: we may choose to continue syncing until the stream sync completes (or until SIGKILL forces shutdown).
Avoid starting any other streams once the running streams are handled, as specified above.

Best reasons not to build

No downsides, no reason not to implement.

View 1 previous comment from the original issue on GitLab

This has been marked as stale because it is unassigned, and has not had recent activity. It will be closed after 21 days if no further activity occurs. If this should never go stale, please add the evergreen label, or request that it be added.

Still relevant

This has been marked as stale because it is unassigned, and has not had recent activity. It will be closed after 21 days if no further activity occurs. If this should never go stale, please add the evergreen label, or request that it be added.

Handling SIGTERM gracefully could allow SDK users to use AWS spot instances and similar without the complexity that comes from attempting to handle hibernation. A long-running pipeline could run whenever there is cheap compute available, gracefully pause when there is no cheap compute available, and then resume where it left off once it becomes available again.

I'm not sure the compute cost of SDK pipelines is a notable concern for anyone, but it'd be nice in any case. Helps with unexpected shutdowns in general.

meltano / sdk