changefeedccl: make it easier to rotate credentials using external connections

[amruss]

Today to rotate credentials using external connections you need to:

• Pause the job
• drop the external connection
• recreate the external connection with the same name
• unpause the job

We should (1) let you alter an external connection and (2) allow you to add a secondary key that will be tried once the first one fails (so you don't have to add the new key after it stops be useful

Jira issue: CRDB-27764

[miretskiy]

I don't think we necessarily need to have support for secondary key. The operator, when they rotate the key, must ensure that there is an overlap between the old and the new key lifetime -- that's pretty standard. So, the operator alters external connection to set the new key which is already active; and the old key should remain active for the next X hours.

[data-matt]

@miretskiy , the issue is that you would need to build some sort of automation script to do that and you introduce a concern that for a period (no matter how small) nothing will be emitted.

[miretskiy]

I'm not sure I understand how automation script comes into play.
The customer wishes to use short lived, explicit credentials. Explicit part is important -- the customer explicitly specifies the token to use when issuing requests. The customer is responsible for providing new token before the current one expires. We have a mechanism -- via oath -- that automatically rotates those keys. The customer can choose that. Or the customer somehow must alter external connection to specify the new key prior to the expiration of the current key.

The concern that nothing is emitted: that's the question for changefeed -- it's more of an optimization than a requirement for a key rotation -- meaning it might be nice for changefeed to notice that external connection changed, and reload that connection w/out having to restart the whole changefeed.

[data-matt]

I agree, either way the customer need to provide the new credentials, however if it was as seamless as the only operation to execute is something like:

ALTER CHANGEFEED {job_id} SET sink = 's3://{BUCKET NAME}?AWS_ACCESS_KEY_ID={ACCESS_KEY_ID}&AWS_SECRET_ACCESS_KEY={SECRET_ACCESS_KEY}';

That would remove some operational overhead.

[blathers-crl[bot]]

cc @cockroachdb/cdc