Open AlexanderChiuluvB opened 2 years ago
great , any contribution is welcome please look to ./test/ folder and try to figure out how to write tests
Alibaba OSS have something like Azurite
or minio
for test locally?
grea , any contribution is welcome please look to ./test/ folder and try to figure out how to write tests
Alibaba OSS have something like Azurite or minio for test locally?
OSS is S3 compatiable, minio can serve as a gateway for OSS
@AlexanderChiuluvB any news? Could I help with anything?
sorry I have been a little bit busy lately I will start push forward it may be 1 wk later
@AlexanderChiuluvB @Slach any update, we can test it recently
@john8628 is Alibaba OSS is S3 compatible, all should works with S3 protocol
@Slach thanks for your reply ,i will try it ;
Can you provide an example of Alibaba OSS by S3 configuration, thank you.
remote_storage: s3
s3:
access_key: "oss key"
secret_key: "oss secret_key"
bucket: "oss bucket"
endpoint: "oss endpoint"
region: "oss region"
assume_role_arn: ""
force_path_style: false
path: dd/backup/pro
disable_ssl: false
compression_level: 1
compression_format: tar
sse: ""
disable_cert_verification: false
storage_class: STANDARD
concurrency: 1
part_size: 0
debug: false
@john8628 Hi, I use Oss as my remote storage, the backup data can be uploaded to Oss successfully, but clickhouse-backup list remote shows "The specified key does not exist". Do you have encountered this problem? Thanks.
Yes, it often happens when your path or your endpoint is not right;
@john8628 How did you solve this problem?
check your path ; ak/sk and endpoint is inner network?
check your path ; ak/sk and endpoint is inner network?
Could you give me some hints? This is my configuration.
s3:
access_key: "xxxxx"
secret_key: "xxxx"
bucket: "test-clickhouse-backup"
endpoint: "https://test-clickhouse-backup.oss-cn-beijing-internal.aliyuncs.com/"
region: "oss-cn-beijing"
acl: private
assume_role_arn: ""
force_path_style: false
path: "core3"
disable_ssl: false
compression_level: 1
compression_format: tar
sse: ""
sse_kms_key_id: ""
sse_customer_algorithm: ""
sse_customer_key: ""
sse_customer_key_md5: ""
sse_kms_encryption_context: ""
disable_cert_verification: false
use_custom_storage_class: false
storage_class: STANDARD
custom_storage_class_map: {}
concurrency: 3
part_size: 0
max_parts_count: 5000
allow_multipart_download: false
object_labels: {}
debug: false
gcs:
Now, the data can be uploaded to oss successfully, but the command clickhouse-backup list remote shows "The specified key does not exist". The oss Web UI is like this.
oh , endpoint is not right , following the https://www.alibabacloud.com/help/en/object-storage-service/latest/regions-and-endpoints ; and if your abosolute path is oss://{bucket}{specific path} then your path should be specific path;your path should be test-clickhouse-backup/core3/
oh , endpoint is not right , following the https://www.alibabacloud.com/help/en/object-storage-service/latest/regions-and-endpoints ; and if your abosolute path is oss://{bucket}{specific path} then your path should be specific path;your path should be test-clickhouse-backup/core3/
Actually, I have tested before with the endpoint oss-cn-beijing-internal.aliyuncs.com, and it shows the error "HostID: oss-cn-beijing-internal.aliyuncs.com, api error SecondLevelDomainForbidden: Please use virtual hosted style to access."
It means that the bucket name must be added to the head of endpoints.
@and1990 try
s3:
bucket: test-clickhouse-backup
force_path_style: true
path: core3
endpoint: https://oss-cn-beijing-internal.aliyuncs.com
use old version ,it can help you
- name: S3_ENDPOINT
value: oss-cn-hangzhou-internal.aliyuncs.com
- name: S3_FORCE_PATH_STYLE
value: "false"
I want to support the tool upload data to OSS Cloud. Can I be assigned with this task?