brokerload加载腾讯云cosn对象时,按照文档设置后提示BE access S3 file failed, SdkResponseCode=403, SdkErrorType=15, SdkErrorMessage=No response body.

[lin0703]

cosn的永久密钥已经在pythonsdk中验证过,表现正常 brokerload语句如下 LOAD LABEL dwa_log_app_sum_di_2024_04_08 ( DATA INFILE("cosn://xxxx/*") INTO TABLExxxxx FORMAT AS parquet (xxxxxxx,user_id_index,pv) COLUMNS FROM PATH AS (p_day) set ( user_id_bm=to_bitmap(user_id_index) ) ) WITH BROKER ( "fs.cosn.userinfo.secretId" = "xxxx", "fs.cosn.userinfo.secretKey" = "xxxx", "fs.cosn.bucket.endpoint_suffix" = "cos.ap-guangzhou.myqcloud.com" ); 执行这段语句后, show load order by createtime desc limit 1的errormsg返回type:LOAD_RUN_FAIL; msg:IOError: BE access S3 file failed, SdkResponseCode=403, SdkErrorType=15, SdkErrorMessage=No response body.

我用同样的语句去加载s3的文件,一切正常 LOAD LABEL dwa_log_app_sum_di_2024_04_09 ( DATA INFILE("s3a://bigdata-oneentity-dw-eu-central-1/prod/hudi/dws/dws_log_app_sum_di*/p_day=2024-04-08/*.parquet") INTO TABLEdwa_log_app_sum_di FORMAT AS parquet (app_event_id,app_id,product_id,mobile_device_name,os_version,network_status,app_version,app_channel,app_event_params,is_batch_channel,is_anonymous_channel,platform,active_country,active_province,active_city,data_center,user_id_index,pv) COLUMNS FROM PATH AS (p_day) set ( user_id_bm=to_bitmap(user_id_index) ) ) WITH BROKER ( "fs.s3a.access.key" = "", "fs.s3a.secret.key" = "", "fs.s3a.endpoint" = "s3.eu-central-1.amazonaws.com" );

谁能提供指导

[lin0703]

需要将hadoop-cos-{hadoop.version}-{version}.jar cos_api-bundle-{version}.jar (从https://cloud.tencent.com/document/product/436/6884 https://github.com/tencentyun/hadoop-cos/releases获取) 放到fe和be的lib下。

再将集群的hdfs-site.xml和core-site.xml放到fe和be的conf下

[lin0703]

在be的sr/conf/core-site.xml里添加fs.cosn.userinfo.secretId和fs.cosn.userinfo.secretKey属性

[github-actions[bot]]

We have marked this issue as stale because it has been inactive for 6 months. If this issue is still relevant, removing the stale label or adding a comment will keep it active. Otherwise, we'll close it in 10 days to keep the issue queue tidy. Thank you for your contribution to StarRocks!