Introduces two different pieces of backend functionality in support of a UI for updating storage mappings. The first is a directive for updating the actual storage mappings, including validating the new mapping by ensuring we have appropriate permissions to the bucket. The second is a postgres function that creates a publication of all the existing specs under a given prefix.
Only top-level tenant prefixes are supported at this time. It shouldn't be too hard to support adding new prefixes in the future, but it seemed wise to avoid the complexity for now.
The directive is meant to validate our access to the users storage buckets. It only supports GCS and S3 at this time, but it shouldn't be too hard to add support for azure and even custom storage endpoints. The checks are done by putting a file, listing it, and deleting it. This is somewhat more crude than checking our users permissions, but it works for any cloud provider and ensures that we're only requiring a minimal set of permissions.
The republish_prefix function is needed in order to apply the newly updated storage mappings. Stitching this together with the directive is being left as a UI responsibility for now.
Workflow steps:
The basic steps are (assume --profile local if running locally):
`flowctl raw rpc --function exchange_directive_token --body '{"bearer_token": ""}'
update applied_directives set user_claims = '{"addStore": {"provider": "S3", "bucket": "estuary-test"}, "catalogPrefix": "phil/"}' where id = '<id-from-rpc>'
Observe applied directive status and storage mappings
If directive was successful, flowctl raw rpc --function republish_prefix --body '{"prefix": "phil/"}' to publish everything under the tenant
I ran through these steps and tested a few different buckets and scenarios:
add GCS bucket with and without a prefix
add an S3 bucket with and without a prefix
add an S3 bucket with an explicit region
add a mapping that's already in the historical mappings, and ensure that it's not duplicated (should only be at index 0)
Some sad-path scenarios:
catalog prefix in user claims is one that the user doens't admin results in invalidClaims
catalog prefix that is not a top-level tenant prefix results in invalidClaims
republish_prefix called with prefix that user doesn't have read access to (results in publication of empty draft)
Description:
Introduces two different pieces of backend functionality in support of a UI for updating storage mappings. The first is a directive for updating the actual storage mappings, including validating the new mapping by ensuring we have appropriate permissions to the bucket. The second is a postgres function that creates a publication of all the existing specs under a given prefix.
Only top-level tenant prefixes are supported at this time. It shouldn't be too hard to support adding new prefixes in the future, but it seemed wise to avoid the complexity for now.
The directive is meant to validate our access to the users storage buckets. It only supports GCS and S3 at this time, but it shouldn't be too hard to add support for azure and even custom storage endpoints. The checks are done by putting a file, listing it, and deleting it. This is somewhat more crude than checking our users permissions, but it works for any cloud provider and ensures that we're only requiring a minimal set of permissions.
The
republish_prefixfunction is needed in order to apply the newly updated storage mappings. Stitching this together with the directive is being left as a UI responsibility for now.Workflow steps:
The basic steps are (assume
--profile localif running locally):update applied_directives set user_claims = '{"addStore": {"provider": "S3", "bucket": "estuary-test"}, "catalogPrefix": "phil/"}' where id = '<id-from-rpc>'flowctl raw rpc --function republish_prefix --body '{"prefix": "phil/"}'to publish everything under the tenantI ran through these steps and tested a few different buckets and scenarios:
Some sad-path scenarios:
invalidClaimsinvalidClaimsDocumentation links affected:
None yet, will document once we have UI done.
Notes for reviewers: n/a
This change is