Add ability to pause/unpause temporal workflows for specific connections

[lmossman]

What

When investigating OC issues, especially those that are affecting all connections, it would be useful to be able to pause and unpause the temporal workflows for all or for specific connections. This can help greatly in cutting down the noise in logs, isolating the issue to a limited set of connections, and testing if issues are related to scale or not.

How

Add logic to the beginning of the connection manager temporal workflow that checks something to decide whether it should continue running the workflow. If not, the temporal workflow should await, with an unblock condition that is fired when the connection is unpaused.

This something could take a few forms:

• A status/column on the connection record in the db
• A LaunchDarkly flag that can be toggled per connection
• Something else

The main thing to keep in mind here is that this flag should be visible and easily configurable, unlike the old "quarantined" status that connections would be put into, where it was nearly impossible to understand which connections were quarantined, and very cumbersome to unquarantine them.

Ideally, when a connection is "paused" in this way it would be made visible in the Airbyte UI, e.g. displaying a message such as This connection has been paused by an Airbyte administrator. This may require a new connection status to be added to the database.

Goal

• Potential requirements :
  • Pause/unpause all connections
  • Pause/unpause single connection
  • Pause/unpause connections for specific source/destination

Start out with a spike into LaunchDarkly to see if it satisfies some or all of these requirements

• Outcome of this issue is ticket to implement the LaunchDarkly integration or to write a tech spec that defines the frontend and backend work needed here

[lmossman]

• Could use Retool to make API requests to perform these actions
• Something like an "admin panel" would be useful here to remove the need to perform db queries during an OC issue

[davinchia]

This is less relevant with the introduction of 'self-healing' workflows.