cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.2k stars 3.82k forks source link

externalconn: improper validation uri params for cloudstorage sinks #88928

Open jayshrivastava opened 2 years ago

jayshrivastava commented 2 years ago

This statement causes an error because of a validation failure:

CREATE EXTERNAL CONNECTION test AS 's3://bucket/path?AWS_SECRET_ACCESS_KEY={AWS_SECRET_ACCESS_KEY}&AWS_ACCESS_KEY_ID={AWS_ACCESS_KEY_ID}&partition_format=daily'

failed to create s3 external connection: unknown S3 query parameters: partition_format

This statement does not

CREATE CHANGEFEED FOR foo INTO 's3://bucket/path?AWS_SECRET_ACCESS_KEY={AWS_SECRET_ACCESS_KEY}&AWS_ACCESS_KEY_ID={AWS_ACCESS_KEY_ID}&partition_format=daily'

These two should have the same behavior. I believe changefeeds have the correct behavior in this case, so external connections should be updated to match.

Similarly, the parameters file_size and topic_prefix fail validation when used with external connections. This is for all sinks (s3, azure, gcp).

Jira issue: CRDB-20041

Epic CRDB-25684

blathers-crl[bot] commented 2 years ago

cc @cockroachdb/cdc

amruss commented 2 years ago

Going to put this in the "follow up work we need to do in 23.1 for named sinks" epic