Feature: Consider adding an option to use stage with out `AWS KEY`.

[yufan022]

Summary refer: https://github.com/datafuselabs/opendal/issues/1088

When users deploy their own Databend, it is better to execute COPY INTO/STAGE without AWS KEY.

now:

create stage aksk url = 's3://xxx/' credentials=(aws_key_id='xxx' aws_secret_key='xxx');
COPY INTO testblock
  FROM @aksk
  pattern ='.*[.]raw'
  file_format = (type = 'ndjson');

target:

create stage aksk url = 's3://xx/xx/';
COPY INTO testblock
  FROM @aksk
  pattern ='.*[.]raw'
  file_format = (type = 'ndjson');

https://github.com/datafuselabs/opendal/issues/1088#issuecomment-1358903695 As @Xuanwo mentioned, The Databend-Cloud and the user-deployed environment have different considerations, it is better to add an option.

[Xuanwo]

Try using CREATE STAGE my_s3_stage url='s3://load/files/' connection=(role_arn='xxxxxxxxx');

[BohuTANG]

Databend supports at: https://github.com/datafuselabs/databend/blob/a592bc836b7cdf2f740296ec349429a65be533b3/src/query/sql/src/planner/binder/location.rs#L173-L185

[yufan022]

👍let me try

[yufan022]

Currently, we use it like this:

# FUSE bucket databend-query.toml
[storage]
type = "s3"
allow_insecure = true

[storage.s3]
bucket = "<your-bucket-name>"
endpoint_url = "<your-endpoint>"
# without aksk
# access_key_id = "<your-key-id>"
# secret_access_key = "<your-account-key>"

---
# copy into stage
CREATE STAGE sharding_$INDEX url = 's3://xx/$INDEX/';
COPY INTO log.sharding_$INDEX from @sharding_$INDEX pattern='.*[.]csv$' FILE_FORMAT = (TYPE = 'CSV' FIELD_DELIMITER = '\t' RECORD_DELIMITER = '\n') PURGE=true

depends on https://github.com/Xuanwo/reqsign/blob/79f74b5b80a1d8503e62750459848fa4222514c9/src/aws/credential.rs#L267-L338