Closed dxygit1 closed 6 months ago
provide more context
which container you setup env
section?
When adding this parameter during deployment, the following error occurs
- name: CLICKHOUSE_PASSWORD
value: abc123
- name: CLICKHOUSE_USE_EMBEDDED_BACKUP_RESTORE
value: "true"
- name: LOG_LEVEL
value: debug
- name: ALLOW_EMPTY_BACKUPS
value: "true"
- name: API_LISTEN
value: 0.0.0.0:7171
- name: API_CREATE_INTEGRATION_TABLES
value: "true"
- name: BACKUPS_TO_KEEP_REMOTE
value: "3"
- name: REMOTE_STORAGE
value: s3
- name: S3_ACL
value: private
- name: S3_ENDPOINT
value: https://oss-cn-hangzhou.aliyuncs.com
- name: S3_BUCKET
value: acos-service
- name: S3_PATH
value: backup
- name: S3_ACCESS_KEY
value: xxxx
- name: S3_SECRET_KEY
value: xxx
- name: S3_FORCE_PATH_STYLE
value: "true"
- name: S3_DISABLE_SSL
value: "true"
- name: S3_DEBUG
value: "true"
kubectl logs pod error
error clickhouse `timeout: 5m`, not enough for `use_embedded_backup_restore: true` logger=server.Run
apply true
instead of ture
Can you take a look at my configuration
- name: CLICKHOUSE_PASSWORD
value: abc123
- name: CLICKHOUSE_USE_EMBEDDED_BACKUP_RESTORE
value: "true"
- name: LOG_LEVEL
value: debug
- name: ALLOW_EMPTY_BACKUPS
value: "true"
- name: API_LISTEN
value: 0.0.0.0:7171
- name: API_CREATE_INTEGRATION_TABLES
value: "true"
- name: BACKUPS_TO_KEEP_REMOTE
value: "3"
- name: REMOTE_STORAGE
value: s3
- name: S3_ACL
value: private
- name: S3_ENDPOINT
value: https://oss-cn-hangzhou.aliyuncs.com
- name: S3_BUCKET
value: acos-service
- name: S3_PATH
value: backup
- name: S3_ACCESS_KEY
value: xxxx
- name: S3_SECRET_KEY
value: xxx
- name: S3_FORCE_PATH_STYLE
value: "true"
- name: S3_DISABLE_SSL
value: "true"
- name: S3_DEBUG
value: "true"
The image was added later for demonstration purposes. Changing it to true will result in an error.
Changing it to true will result in an error.
@dxygit1 Do you mean error which described in https://github.com/Altinity/clickhouse-backup/issues/877 ?
The error is this. It has nothing to do with either.
error clickhouse `timeout: 5m`, not enough for `use_embedded_backup_restore: true` logger=server.Run
just add in your configuration
- name: CLICKHOUSE_TIMEOUT
value: 4h
cause BACKUP / RESTORE SQL command could take a lot of time during execution
use_backup_embedded_restore: true
require define separate backup disk
look example in https://github.com/Altinity/clickhouse-backup/blob/master/test/integration/config-s3-embedded.yml#L20-L23
Thank you for your response. Now I have three questions:
1、I didn't find the corresponding environment variable for embedded_backup_disk. For example, timeout corresponds to CLICKHOUSE_TIMEOUT. Is embedded_backup_disk corresponding to CLICKHOUSE_EMBEDDED_BACKUP_DISK?
2、Do these need to be added as configurations within ClickHouse?
3、How to backup and restore? Is it done through API requests? Is there any documentation explaining the backup and restore principles?
I haven't found any API for performing both backup and upload simultaneously. Currently, it seems that the APIs for backup and upload are separate. Is that correct? And for incremental backup, is the only option the /backup/watch endpoint?
I didn't find the corresponding environment variable for embedded_backup_disk
Look to https://github.com/Altinity/clickhouse-backup/blob/master/ReadMe.md#default-config
there is the list of all available variable names
Do these need to be added as configurations within ClickHouse?
backup disk requires to be created explicitly for latest
released clickhouse-backup version
use spec.configuration.files
in your kind: ClickHouseInstallation
manifest if you use clickhouse-operator
spec:
configuration:
files:
config.d/backup_disk.xml: |
<clickhouse>
...
</clickhouse>
How to backup and restore? Is it done through API requests? Is there any documentation explaining the backup and restore principles?
Please read https://github.com/Altinity/clickhouse-backup/blob/master/Examples.md
I haven't found any API for performing both backup and upload simultaneously.
look to https://github.com/Altinity/clickhouse-backup/?tab=readme-ov-file#default-config
api
config section
env:
- name: API_ALLOW_PARALLEL
value: "true"
be careful because this use-case is rare, and you can make high workload for your system and degrade performance if you don't understand how backup works and how it affects
And for incremental backup, is the only option the /backup/watch endpoint?
Can this parameter be added in this way? Currently, it seems to be causing errors.
curl chi-clickhouse-cluster-dtstack-0-0:7171/backup/watch?watch_interval=1h -X POST
{"status":"acknowledged","operation":"watch","command":"watch --watch-interval=\"1h\""}
I saw the following error in the logs:
'error Watch error: fullInterval 24h is too long to keep 3 remote backups with watchInterval 1h logger=server.
I don't want to do a full backup every day. Can I remove this requirement?
What is the relationship between them? After performing an incremental backup, is a full backup still necessary? Will the previous incremental backups be deleted after a full backup?
Yes you can https://github.com/Altinity/clickhouse-backup/blob/master/ReadMe.md?plain=1#L677-L690
but you can't change exists running watch command first of all you need to POST /backup/kill look details in https://github.com/Altinity/clickhouse-backup/blob/master/ReadMe.md?plain=1#L644-L648
Can't I change the parameters? For example, how often to perform backups, how often to perform full backups? If I don't specify tables, does it mean it's a full database backup?
I've reviewed this example. Am I unable to change the parameters? Aren't they optional parameters?
curl -s 'localhost:7171/backup/watch?table=default.billing&watch_interval=1h&full_interval=24h' -X POST
but you can't change exists running watch command
How should I interpret this sentence? I didn't modify it, did I? I only changed the parameters.
Additionally, what is the reason for this log error? Is it because I allocated too little disk space? What is its calculation logic?
'error Watch error: fullInterval 24h is too long to keep 3 remote backups with watchInterval 1h logger=server.
but you can't change exists running watch command How should I interpret this sentence? I didn't modify it, did I? I only changed the parameters.
It means
Please avoid multiple parallel execution if you use
API_ALLOW_PARALLEL: true``
look to SELECT * FROM system.backup_actions
or GET /backup/actions
try to avoid multiple parallel execution of watch command
When you run watch first time, then it just run infinite internal commands to create full+incremental backups sequence
full
backupwatch_interval
increment1' diff from remote
full`watch_interval
increment2' diff from remote
increment1`full_interval
achieved be
look details in
https://github.com/Altinity/clickhouse-backup/?tab=readme-ov-file#cli-command---watch
and
https://github.com/Altinity/clickhouse-backup/blob/master/Examples.md#how-to-watch-backups-workwhen you call second API call with different parameters and API_ALLOW_PARALLEL
is true
then you will just execute two backup creation sequences in parallel, it will produce unnecessary workload to your system and degrade performance of clickhouse-server
if you have enough data
Additionally, what is the reason for this log error? Is it because I allocated too little disk space? What is its calculation logic? 'error Watch error: fullInterval 24h is too long to keep 3 remote backups with watchInterval 1h logger=server.
No, this is not related to disk size, you defined KEEP_BACKUPS_REMOTE: 3
but try to create full every 24h
and increment every 1h
, it requires KEEP_BACKUPS_REMOTE: 24
in your env
Don't be shy to make Pull Request with better error message
Please try to read and figure out with links which I shared
so, please kill exists watch command if it's running after that you can success execute
curl -X POST "chi-clickhouse-cluster-dtstack-0-0:7171/backup/watch?watch_interval=1h&full_interval=24h"
if you already applied
KEEP_BACKUPS_REMOTE: 24
in your env
section of kubernetes manifest
When I execute
curl -X POST "chi-clickhouse-cluster-dtstack-0-0:7171/backup/watch?watch_interval=8h&full_interval=24h"
it returns an error
"status":"acknowledged","operation":"watch","command":"watch --watch-interval=\"8h\" --full-interval=\"24h\""
Why is this happening? I don't want backups to be taken every hour; I want them to be taken every 8 hours.
config
<storage_configuration>
<disks>
<backups_s3>
<type>s3</type>
<endpoint>http://minio:9000/clickhouse/{cluster}/{shard}/</endpoint>
<!-- https://github.com/Altinity/clickhouse-backup/issues/691
<access_key_id>xxx</access_key_id>
<secret_access_key>xxxx</secret_access_key>
-->
<use_environment_credentials>1</use_environment_credentials>
<cache_enabled>false</cache_enabled>
<!-- to avoid slow startup -->
<send_metadata>false</send_metadata>
</backups_s3>
</disks>
</storage_configuration>
<backups>
<allowed_disk>backups_s3</allowed_disk>
<allowed_path>/var/lib/clickhouse/backups_embedded/</allowed_path>
</backups>
- name: API_ALLOW_PARALLEL
value: "true"
- name: CLICKHOUSE_EMBEDDED_BACKUP_DISK
value: "backups_s3"
- name: CLICKHOUSE_TIMEOUT
value: 4h
logs error info
2024/04/01 07:25:15.202484 info API call POST /backup/watch logger=server
2024/04/01 07:25:15.204612 debug api.status.Start -> status.commands[4] == {ActionRowStatus:{Command:watch --watch-interval="8h" --full-interval="24h" Status:in progress Start:2024-04-01 07:25:15 Finish: Error:} Ctx:context.Background.WithCancel Cancel:0x4a7ba0} logger=status
2024/04/01 07:25:15.205085 info clickhouse connection prepared: tcp://localhost:9000 run ping logger=clickhouse
2024/04/01 07:25:15.208751 info clickhouse connection open: tcp://localhost:9000 logger=clickhouse
2024/04/01 07:25:15.208807 info SELECT count() AS is_macros_exists FROM system.tables WHERE database='system' AND name='macros' SETTINGS empty_result_for_aggregation_by_empty_set=0 logger=clickhouse
2024/04/01 07:25:15.213018 info SELECT macro, substitution FROM system.macros logger=clickhouse
2024/04/01 07:25:15.214155 info SELECT count() AS is_macros_exists FROM system.tables WHERE database='system' AND name='macros' SETTINGS empty_result_for_aggregation_by_empty_set=0 logger=clickhouse
2024/04/01 07:25:15.216460 info SELECT macro, substitution FROM system.macros logger=clickhouse
2024/04/01 07:25:15.219344 info [s3:DEBUG] Request
GET /acos-service?versioning= HTTP/1.1
Host: minio:9000
User-Agent: aws-sdk-go-v2/1.24.1 os/linux lang/go#1.22.1 md/GOOS#linux md/GOARCH#amd64 api/s3#1.48.1
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: 4218d1e2-3bfa-4936-8d4f-d4ee6cf6bad0
Amz-Sdk-Request: attempt=1; max=3
Authorization: AWS4-HMAC-SHA256 Credential=acos/20240401/us-east-1/s3/aws4_request, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date, Signature=cbb3a6200b36fa32d62bd5d200342818868cfca1b3f584f89c4e696e45f98aff
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20240401T072515Z
2024/04/01 07:25:15.222684 info [s3:DEBUG] Response
HTTP/1.1 200 OK
Content-Length: 99
Accept-Ranges: bytes
Content-Security-Policy: block-all-mixed-content
Content-Type: application/xml
Date: Mon, 01 Apr 2024 07:25:44 GMT
Server: MinIO
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin
Vary: Accept-Encoding
X-Amz-Request-Id: 17C21739887BE7AF
X-Content-Type-Options: nosniff
X-Xss-Protection: 1; mode=block
2024/04/01 07:25:15.223232 debug /tmp/.clickhouse-backup-metadata.cache.S3 load 0 elements logger=s3
2024/04/01 07:25:15.223950 info [s3:DEBUG] Request
GET /acos-service?delimiter=%2F&list-type=2&max-keys=1000&prefix=backup%2Fshard-0%2F HTTP/1.1
Host: minio:9000
User-Agent: aws-sdk-go-v2/1.24.1 os/linux lang/go#1.22.1 md/GOOS#linux md/GOARCH#amd64 api/s3#1.48.1
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: 38e08fb4-fa0b-4688-8835-733a3e778e46
Amz-Sdk-Request: attempt=1; max=3
Authorization: AWS4-HMAC-SHA256 Credential=acos/20240401/us-east-1/s3/aws4_request, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date, Signature=7883b42cbe4c6313353b17c6e1e34a324830a3aa3705f150bc8cb5bbbce9745f
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20240401T072515Z
2024/04/01 07:25:15.225501 info [s3:DEBUG] Response
HTTP/1.1 200 OK
Content-Length: 282
Accept-Ranges: bytes
Content-Security-Policy: block-all-mixed-content
Content-Type: application/xml
Date: Mon, 01 Apr 2024 07:25:44 GMT
Server: MinIO
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin
Vary: Accept-Encoding
X-Amz-Request-Id: 17C2173988A1BFFB
X-Content-Type-Options: nosniff
X-Xss-Protection: 1; mode=block
2024/04/01 07:25:15.225849 debug /tmp/.clickhouse-backup-metadata.cache.S3 save 0 elements logger=s3
2024/04/01 07:25:15.226100 info clickhouse connection closed logger=clickhouse
2024/04/01 07:25:15.226130 info SELECT count() AS is_macros_exists FROM system.tables WHERE database='system' AND name='macros' SETTINGS empty_result_for_aggregation_by_empty_set=0 logger=clickhouse
2024/04/01 07:25:15.229716 info SELECT macro, substitution FROM system.macros logger=clickhouse
2024/04/01 07:25:15.231026 info clickhouse connection prepared: tcp://localhost:9000 run ping logger=clickhouse
2024/04/01 07:25:15.232102 info clickhouse connection open: tcp://localhost:9000 logger=clickhouse
2024/04/01 07:25:15.232779 info SELECT count() AS is_macros_exists FROM system.tables WHERE database='system' AND name='macros' SETTINGS empty_result_for_aggregation_by_empty_set=0 logger=clickhouse
2024/04/01 07:25:15.235794 info SELECT macro, substitution FROM system.macros logger=clickhouse
2024/04/01 07:25:15.237472 info clickhouse connection prepared: tcp://localhost:9000 run ping logger=clickhouse
2024/04/01 07:25:15.238425 info clickhouse connection open: tcp://localhost:9000 logger=clickhouse
2024/04/01 07:25:15.238468 info SELECT metadata_path FROM system.tables WHERE database = 'system' AND metadata_path!='' LIMIT 1; logger=clickhouse
2024/04/01 07:25:15.240480 info SELECT name, engine FROM system.databases WHERE NOT match(name,'^(system|INFORMATION_SCHEMA|information_schema|_temporary_and_external_tables)$') logger=clickhouse
2024/04/01 07:25:15.242805 info SHOW CREATE DATABASE `_internal_ddl_health_check_` logger=clickhouse
2024/04/01 07:25:15.243757 info SHOW CREATE DATABASE `alert` logger=clickhouse
2024/04/01 07:25:15.244580 info SHOW CREATE DATABASE `default` logger=clickhouse
2024/04/01 07:25:15.245376 info SHOW CREATE DATABASE `distributed_1` logger=clickhouse
2024/04/01 07:25:15.246125 info SHOW CREATE DATABASE `dtsmart` logger=clickhouse
2024/04/01 07:25:15.246955 info SHOW CREATE DATABASE `local_1` logger=clickhouse
2024/04/01 07:25:15.248182 info SELECT name, count(*) as is_present FROM system.settings WHERE name IN (?, ?) GROUP BY name with args [show_table_uuid_in_table_create_query_if_not_nil display_secrets_in_show_and_select] logger=clickhouse
2024/04/01 07:25:15.254069 info SELECT name FROM system.databases WHERE engine IN ('MySQL','PostgreSQL','MaterializedPostgreSQL') logger=clickhouse
2024/04/01 07:25:15.255386 info SELECT countIf(name='data_path') is_data_path_present, countIf(name='data_paths') is_data_paths_present, countIf(name='uuid') is_uuid_present, countIf(name='create_table_query') is_create_table_query_present, countIf(name='total_bytes') is_total_bytes_present FROM system.columns WHERE database='system' AND table='tables' logger=clickhouse
2024/04/01 07:25:15.258524 info SELECT database, name, engine , data_paths , uuid , create_table_query , coalesce(total_bytes, 0) AS total_bytes FROM system.tables WHERE is_temporary = 0 ORDER BY total_bytes DESC SETTINGS show_table_uuid_in_table_create_query_if_not_nil=1 logger=clickhouse
2024/04/01 07:25:15.283079 info SELECT metadata_path FROM system.tables WHERE database = 'system' AND metadata_path!='' LIMIT 1; logger=clickhouse
2024/04/01 07:25:15.285256 warn can't read /var/lib/clickhouse/metadata/dtsmart/dt_alert.sql: open /var/lib/clickhouse/metadata/dtsmart/dt_alert.sql: no such file or directory logger=clickhouse
2024/04/01 07:25:15.285313 warn can't read /var/lib/clickhouse/metadata/dtsmart/dt_arms_app.sql: open /var/lib/clickhouse/metadata/dtsmart/dt_arms_app.sql: no such file or directory logger=clickhouse
2024/04/01 07:25:15.285333 warn can't read /var/lib/clickhouse/metadata/dtsmart/dt_arms_service_group.sql: open /var/lib/clickhouse/metadata/dtsmart/dt_arms_service_group.sql: no such file or directory logger=clickhouse
2024/04/01 07:25:15.285357 warn can't read /var/lib/clickhouse/metadata/dtsmart/dt_arms_service_group_app.sql: open /var/lib/clickhouse/metadata/dtsmart/dt_arms_service_group_app.sql: no such file or directory logger=clickhouse
2024/04/01 07:25:15.285380 warn can't read /var/lib/clickhouse/metadata/dtsmart/dt_chart.sql: open /var/lib/clickhouse/metadata/dtsmart/dt_chart.sql: no such file or directory logger=clickhouse
2024/04/01 07:25:15.285405 warn can't read /var/lib/clickhouse/metadata/dtsmart/dt_collect_model.sql: open /var/lib/clickhouse/metadata/dtsmart/dt_collect_model.sql: no such file or directory logger=clickhouse
2024/04/01 07:25:15.285430 warn can't read /var/lib/clickhouse/metadata/dtsmart/dt_collect_model_task.sql: open /var/lib/clickhouse/metadata/dtsmart/dt_collect_model_task.sql: no such file or directory logger=clickhouse
2024/04/01 07:25:15.285448 warn can't read /var/lib/clickhouse/metadata/dtsmart/dt_dashboard.sql: open /var/lib/clickhouse/metadata/dtsmart/dt_dashboard.sql: no such file or directory logger=clickhouse
2024/04/01 07:25:15.285478 warn can't read /var/lib/clickhouse/metadata/dtsmart/dt_data_index.sql: open /var/lib/clickhouse/metadata/dtsmart/dt_data_index.sql: no such file or directory logger=clickhouse
2024/04/01 07:25:15.285501 warn can't read /var/lib/clickhouse/metadata/dtsmart/dt_host.sql: open /var/lib/clickhouse/metadata/dtsmart/dt_host.sql: no such file or directory logger=clickhouse
2024/04/01 07:25:15.285539 warn can't read /var/lib/clickhouse/metadata/dtsmart/dt_inspect_record.sql: open /var/lib/clickhouse/metadata/dtsmart/dt_inspect_record.sql: no such file or directory logger=clickhouse
2024/04/01 07:25:15.285565 warn can't read /var/lib/clickhouse/metadata/dtsmart/dt_inspect_template.sql: open /var/lib/clickhouse/metadata/dtsmart/dt_inspect_template.sql: no such file or directory logger=clickhouse
2024/04/01 07:25:15.285592 warn can't read /var/lib/clickhouse/metadata/dtsmart/dt_resource.sql: open /var/lib/clickhouse/metadata/dtsmart/dt_resource.sql: no such file or directory logger=clickhouse
2024/04/01 07:25:15.285621 warn can't read /var/lib/clickhouse/metadata/dtsmart/dt_resource_optimization_suggest.sql: open /var/lib/clickhouse/metadata/dtsmart/dt_resource_optimization_suggest.sql: no such file or directory logger=clickhouse
2024/04/01 07:25:15.285706 info SELECT count() as cnt FROM system.columns WHERE database='system' AND table='functions' AND name='create_query' SETTINGS empty_result_for_aggregation_by_empty_set=0 logger=clickhouse
2024/04/01 07:25:15.288068 info SELECT name, create_query FROM system.functions WHERE create_query!='' logger=clickhouse
2024/04/01 07:25:15.291454 info SELECT value FROM `system`.`build_options` where name='VERSION_INTEGER' logger=clickhouse
2024/04/01 07:25:15.292687 info SELECT countIf(name='type') AS is_disk_type_present, countIf(name='free_space') AS is_free_space_present, countIf(name='disks') AS is_storage_policy_present FROM system.columns WHERE database='system' AND table IN ('disks','storage_policies') logger=clickhouse
2024/04/01 07:25:15.295271 info SELECT d.path, any(d.name) AS name, any(d.type) AS type, min(d.free_space) AS free_space, groupUniqArray(s.policy_name) AS storage_policies FROM system.disks AS d LEFT JOIN (SELECT policy_name, arrayJoin(disks) AS disk FROM system.storage_policies) AS s ON s.disk = d.name GROUP BY d.path logger=clickhouse
2024/04/01 07:25:15.299456 info BACKUP TABLE `local_1`.`system_metrics`, TABLE `local_1`.`ddd`, TABLE `local_1`.`metrics_summary`, TABLE `local_1`.`arms_frontend_log`, TABLE `local_1`.`system_attribute_meta`, TABLE `local_1`.`easydo_agent_monitor`, TABLE `local_1`.`easydo_collect_task_monitor`, TABLE `distributed_1`.`dt_chart`, TABLE `distributed_1`.`dt_collect_model`, TABLE `distributed_1`.`dt_collect_model_task`, TABLE `distributed_1`.`dt_dashboard`, TABLE `distributed_1`.`dt_data_index`, TABLE `distributed_1`.`dt_host`, TABLE `distributed_1`.`dt_inspect_record`, TABLE `distributed_1`.`dt_inspect_template`, TABLE `distributed_1`.`dt_resource`, TABLE `distributed_1`.`dt_resource_optimization_suggest`, TABLE `distributed_1`.`easydo_agent_monitor`, TABLE `distributed_1`.`easydo_collect_task_monitor`, TABLE `distributed_1`.`metrics_summary`, TABLE `distributed_1`.`system_attribute_meta`, TABLE `distributed_1`.`system_metrics`, TABLE `dtsmart`.`dt_alert`, TABLE `dtsmart`.`dt_arms_app`, TABLE `dtsmart`.`dt_arms_service_group`, TABLE `dtsmart`.`dt_arms_service_group_app`, TABLE `dtsmart`.`dt_chart`, TABLE `dtsmart`.`dt_collect_model`, TABLE `dtsmart`.`dt_collect_model_task`, TABLE `dtsmart`.`dt_dashboard`, TABLE `dtsmart`.`dt_data_index`, TABLE `dtsmart`.`dt_host`, TABLE `dtsmart`.`dt_inspect_record`, TABLE `dtsmart`.`dt_inspect_template`, TABLE `dtsmart`.`dt_resource`, TABLE `dtsmart`.`dt_resource_optimization_suggest`, TABLE `distributed_1`.`dt_arms_app`, TABLE `distributed_1`.`dt_arms_service_group`, TABLE `distributed_1`.`dt_arms_service_group_app`, TABLE `distributed_1`.`dt_alert`, TABLE `distributed_1`.`ddd`, TABLE `distributed_1`.`arms_frontend_log` TO Disk(?,?) with args [backups_s3 shard0-full-20240401072515] logger=clickhouse
2024/04/01 07:25:17.316699 info SELECT sum(total_bytes) AS backup_data_size FROM system.tables WHERE concat(database,'.',name) IN ('local_1.system_metrics', 'local_1.ddd', 'local_1.metrics_summary', 'local_1.arms_frontend_log', 'local_1.system_attribute_meta', 'local_1.easydo_agent_monitor', 'local_1.easydo_collect_task_monitor', 'distributed_1.dt_chart', 'distributed_1.dt_collect_model', 'distributed_1.dt_collect_model_task', 'distributed_1.dt_dashboard', 'distributed_1.dt_data_index', 'distributed_1.dt_host', 'distributed_1.dt_inspect_record', 'distributed_1.dt_inspect_template', 'distributed_1.dt_resource', 'distributed_1.dt_resource_optimization_suggest', 'distributed_1.easydo_agent_monitor', 'distributed_1.easydo_collect_task_monitor', 'distributed_1.metrics_summary', 'distributed_1.system_attribute_meta', 'distributed_1.system_metrics', 'dtsmart.dt_alert', 'dtsmart.dt_arms_app', 'dtsmart.dt_arms_service_group', 'dtsmart.dt_arms_service_group_app', 'dtsmart.dt_chart', 'dtsmart.dt_collect_model', 'dtsmart.dt_collect_model_task', 'dtsmart.dt_dashboard', 'dtsmart.dt_data_index', 'dtsmart.dt_host', 'dtsmart.dt_inspect_record', 'dtsmart.dt_inspect_template', 'dtsmart.dt_resource', 'dtsmart.dt_resource_optimization_suggest', 'distributed_1.dt_arms_app', 'distributed_1.dt_arms_service_group', 'distributed_1.dt_arms_service_group_app', 'distributed_1.dt_alert', 'distributed_1.ddd', 'distributed_1.arms_frontend_log') logger=clickhouse
2024/04/01 07:25:17.320401 debug calculate parts list from embedded backup disk backup=shard0-full-20240401072515 logger=backuper operation=create
2024/04/01 07:25:17.328883 info SELECT value FROM `system`.`build_options` where name='VERSION_DESCRIBE' logger=clickhouse
2024/04/01 07:25:17.330708 info done backup=shard0-full-20240401072515 duration=2.093s logger=backuper operation=create_embedded
2024/04/01 07:25:17.330800 info clickhouse connection closed logger=clickhouse
2024/04/01 07:25:17.330883 info clickhouse connection prepared: tcp://localhost:9000 run ping logger=clickhouse
2024/04/01 07:25:17.331790 info clickhouse connection open: tcp://localhost:9000 logger=clickhouse
2024/04/01 07:25:17.331855 info SELECT countIf(name='type') AS is_disk_type_present, countIf(name='free_space') AS is_free_space_present, countIf(name='disks') AS is_storage_policy_present FROM system.columns WHERE database='system' AND table IN ('disks','storage_policies') logger=clickhouse
2024/04/01 07:25:17.335408 info SELECT d.path, any(d.name) AS name, any(d.type) AS type, min(d.free_space) AS free_space, groupUniqArray(s.policy_name) AS storage_policies FROM system.disks AS d LEFT JOIN (SELECT policy_name, arrayJoin(disks) AS disk FROM system.storage_policies) AS s ON s.disk = d.name GROUP BY d.path logger=clickhouse
2024/04/01 07:25:17.339800 info SELECT max(toInt64(bytes_on_disk * 1.02)) AS max_file_size FROM system.parts logger=clickhouse
2024/04/01 07:25:17.342822 info SELECT count() AS is_macros_exists FROM system.tables WHERE database='system' AND name='macros' SETTINGS empty_result_for_aggregation_by_empty_set=0 logger=clickhouse
2024/04/01 07:25:17.344884 info SELECT macro, substitution FROM system.macros logger=clickhouse
2024/04/01 07:25:17.345962 info SELECT count() AS is_macros_exists FROM system.tables WHERE database='system' AND name='macros' SETTINGS empty_result_for_aggregation_by_empty_set=0 logger=clickhouse
2024/04/01 07:25:17.348153 info SELECT macro, substitution FROM system.macros logger=clickhouse
2024/04/01 07:25:17.350285 info [s3:DEBUG] Request
GET /acos-service?versioning= HTTP/1.1
Host: minio:9000
User-Agent: aws-sdk-go-v2/1.24.1 os/linux lang/go#1.22.1 md/GOOS#linux md/GOARCH#amd64 api/s3#1.48.1
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: 226eae91-ce6b-4f02-9b7d-46655f01aa51
Amz-Sdk-Request: attempt=1; max=3
Authorization: AWS4-HMAC-SHA256 Credential=acos/20240401/us-east-1/s3/aws4_request, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date, Signature=edb3e2aa8ffb905ad8f8e15183ad09fc6b91fd7197c57e0b881a39395f8cf0a3
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20240401T072517Z
2024/04/01 07:25:17.353222 info [s3:DEBUG] Response
HTTP/1.1 200 OK
Content-Length: 99
Accept-Ranges: bytes
Content-Security-Policy: block-all-mixed-content
Content-Type: application/xml
Date: Mon, 01 Apr 2024 07:25:46 GMT
Server: MinIO
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin
Vary: Accept-Encoding
X-Amz-Request-Id: 17C2173A07785D3A
X-Content-Type-Options: nosniff
X-Xss-Protection: 1; mode=block
2024/04/01 07:25:17.353540 debug /tmp/.clickhouse-backup-metadata.cache.S3 load 0 elements logger=s3
2024/04/01 07:25:17.354045 info [s3:DEBUG] Request
GET /acos-service?delimiter=%2F&list-type=2&max-keys=1000&prefix=backup%2Fshard-0%2F HTTP/1.1
Host: minio:9000
User-Agent: aws-sdk-go-v2/1.24.1 os/linux lang/go#1.22.1 md/GOOS#linux md/GOARCH#amd64 api/s3#1.48.1
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: 932ec2d5-4139-4679-a3b5-2447606498c1
Amz-Sdk-Request: attempt=1; max=3
Authorization: AWS4-HMAC-SHA256 Credential=acos/20240401/us-east-1/s3/aws4_request, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date, Signature=ae8be870680c18d3bfc6f0cd27c46c53d2e527bd65a6f6f934f679f187fa5996
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20240401T072517Z
2024/04/01 07:25:17.355857 info [s3:DEBUG] Response
HTTP/1.1 200 OK
Content-Length: 282
Accept-Ranges: bytes
Content-Security-Policy: block-all-mixed-content
Content-Type: application/xml
Date: Mon, 01 Apr 2024 07:25:46 GMT
Server: MinIO
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin
Vary: Accept-Encoding
X-Amz-Request-Id: 17C2173A07993794
X-Content-Type-Options: nosniff
X-Xss-Protection: 1; mode=block
2024/04/01 07:25:17.356260 debug /tmp/.clickhouse-backup-metadata.cache.S3 save 0 elements logger=s3
2024/04/01 07:25:17.360278 warn /var/lib/clickhouse/backup/shard0-full-20240401072515/upload.state empty, will continue from scratch error: open /var/lib/clickhouse/backup/shard0-full-20240401072515/upload.state: no such file or directory logger=resumable
2024/04/01 07:25:17.360332 debug prepare table concurrent semaphore with concurrency=2 len(tablesForUpload)=42 backup=shard0-full-20240401072515 operation=upload
2024/04/01 07:25:17.360499 debug start dtsmart.dt_inspect_template with concurrency=2 len(table.Parts[...])=0 logger=uploadTableData
2024/04/01 07:25:17.360564 debug finish dtsmart.dt_inspect_template with concurrency=2 len(table.Parts[...])=0 uploadedFiles=map[], uploadedBytes=0 logger=uploadTableData
2024/04/01 07:25:17.360501 debug start local_1.easydo_collect_task_monitor with concurrency=2 len(table.Parts[...])=0 logger=uploadTableData
2024/04/01 07:25:17.360656 debug start dtsmart.dt_alert with concurrency=2 len(table.Parts[...])=0 logger=uploadTableData
2024/04/01 07:25:17.360664 debug finish local_1.easydo_collect_task_monitor with concurrency=2 len(table.Parts[...])=0 uploadedFiles=map[], uploadedBytes=0 logger=uploadTableData
2024/04/01 07:25:17.360668 debug finish dtsmart.dt_alert with concurrency=2 len(table.Parts[...])=0 uploadedFiles=map[], uploadedBytes=0 logger=uploadTableData
2024/04/01 07:25:17.360745 debug start dtsmart.dt_host with concurrency=2 len(table.Parts[...])=0 logger=uploadTableData
2024/04/01 07:25:17.360757 debug finish dtsmart.dt_host with concurrency=2 len(table.Parts[...])=0 uploadedFiles=map[], uploadedBytes=0 logger=uploadTableData
2024/04/01 07:25:17.360834 debug start local_1.easydo_agent_monitor with concurrency=2 len(table.Parts[...])=0 logger=uploadTableData
2024/04/01 07:25:17.360865 debug finish local_1.easydo_agent_monitor with concurrency=2 len(table.Parts[...])=0 uploadedFiles=map[], uploadedBytes=0 logger=uploadTableData
2024/04/01 07:25:17.360927 debug start local_1.ddd with concurrency=2 len(table.Parts[...])=0 logger=uploadTableData
2024/04/01 07:25:17.360946 debug finish local_1.ddd with concurrency=2 len(table.Parts[...])=0 uploadedFiles=map[], uploadedBytes=0 logger=uploadTableData
2024/04/01 07:25:17.360838 debug start dtsmart.dt_inspect_record with concurrency=2 len(table.Parts[...])=0 logger=uploadTableData
2024/04/01 07:25:17.360985 debug finish dtsmart.dt_inspect_record with concurrency=2 len(table.Parts[...])=0 uploadedFiles=map[], uploadedBytes=0 logger=uploadTableData
2024/04/01 07:25:17.361013 debug start local_1.arms_frontend_log with concurrency=2 len(table.Parts[...])=0 logger=uploadTableData
2024/04/01 07:25:17.361052 debug finish local_1.arms_frontend_log with concurrency=2 len(table.Parts[...])=0 uploadedFiles=map[], uploadedBytes=0 logger=uploadTableData
2024/04/01 07:25:17.361113 debug start dtsmart.dt_resource_optimization_suggest with concurrency=2 len(table.Parts[...])=0 logger=uploadTableData
2024/04/01 07:25:17.361156 debug finish dtsmart.dt_resource_optimization_suggest with concurrency=2 len(table.Parts[...])=0 uploadedFiles=map[], uploadedBytes=0 logger=uploadTableData
2024/04/01 07:25:17.361116 debug start dtsmart.dt_resource with concurrency=2 len(table.Parts[...])=0 logger=uploadTableData
2024/04/01 07:25:17.361237 debug finish dtsmart.dt_resource with concurrency=2 len(table.Parts[...])=0 uploadedFiles=map[], uploadedBytes=0 logger=uploadTableData
2024/04/01 07:25:17.361263 debug start local_1.system_attribute_meta with concurrency=2 len(table.Parts[...])=0 logger=uploadTableData
2024/04/01 07:25:17.361280 debug finish local_1.system_attribute_meta with concurrency=2 len(table.Parts[...])=0 uploadedFiles=map[], uploadedBytes=0 logger=uploadTableData
2024/04/01 07:25:17.361321 debug start local_1.system_metrics with concurrency=2 len(table.Parts[...])=0 logger=uploadTableData
2024/04/01 07:25:17.361337 debug finish local_1.system_metrics with concurrency=2 len(table.Parts[...])=0 uploadedFiles=map[], uploadedBytes=0 logger=uploadTableData
2024/04/01 07:25:17.361394 debug start local_1.metrics_summary with concurrency=2 len(table.Parts[...])=0 logger=uploadTableData
2024/04/01 07:25:17.361416 debug start dtsmart.dt_data_index with concurrency=2 len(table.Parts[...])=0 logger=uploadTableData
2024/04/01 07:25:17.361430 debug finish dtsmart.dt_data_index with concurrency=2 len(table.Parts[...])=0 uploadedFiles=map[], uploadedBytes=0 logger=uploadTableData
2024/04/01 07:25:17.361474 debug start dtsmart.dt_dashboard with concurrency=2 len(table.Parts[...])=0 logger=uploadTableData
2024/04/01 07:25:17.361490 debug finish dtsmart.dt_dashboard with concurrency=2 len(table.Parts[...])=0 uploadedFiles=map[], uploadedBytes=0 logger=uploadTableData
2024/04/01 07:25:17.361418 debug finish local_1.metrics_summary with concurrency=2 len(table.Parts[...])=0 uploadedFiles=map[], uploadedBytes=0 logger=uploadTableData
2024/04/01 07:25:17.361539 debug start dtsmart.dt_collect_model_task with concurrency=2 len(table.Parts[...])=0 logger=uploadTableData
2024/04/01 07:25:17.361550 debug finish dtsmart.dt_collect_model_task with concurrency=2 len(table.Parts[...])=0 uploadedFiles=map[], uploadedBytes=0 logger=uploadTableData
2024/04/01 07:25:17.361593 debug start dtsmart.dt_chart with concurrency=2 len(table.Parts[...])=0 logger=uploadTableData
2024/04/01 07:25:17.361602 debug start dtsmart.dt_collect_model with concurrency=2 len(table.Parts[...])=0 logger=uploadTableData
2024/04/01 07:25:17.361630 debug finish dtsmart.dt_collect_model with concurrency=2 len(table.Parts[...])=0 uploadedFiles=map[], uploadedBytes=0 logger=uploadTableData
2024/04/01 07:25:17.361603 debug finish dtsmart.dt_chart with concurrency=2 len(table.Parts[...])=0 uploadedFiles=map[], uploadedBytes=0 logger=uploadTableData
2024/04/01 07:25:17.361713 debug start dtsmart.dt_arms_service_group_app with concurrency=2 len(table.Parts[...])=0 logger=uploadTableData
2024/04/01 07:25:17.361727 debug finish dtsmart.dt_arms_service_group_app with concurrency=2 len(table.Parts[...])=0 uploadedFiles=map[], uploadedBytes=0 logger=uploadTableData
2024/04/01 07:25:17.361778 debug start dtsmart.dt_arms_service_group with concurrency=2 len(table.Parts[...])=0 logger=uploadTableData
2024/04/01 07:25:17.361794 debug finish dtsmart.dt_arms_service_group with concurrency=2 len(table.Parts[...])=0 uploadedFiles=map[], uploadedBytes=0 logger=uploadTableData
2024/04/01 07:25:17.361801 debug start dtsmart.dt_arms_app with concurrency=2 len(table.Parts[...])=0 logger=uploadTableData
2024/04/01 07:25:17.361817 debug finish dtsmart.dt_arms_app with concurrency=2 len(table.Parts[...])=0 uploadedFiles=map[], uploadedBytes=0 logger=uploadTableData
2024/04/01 07:25:17.361874 debug start distributed_1.dt_data_index with concurrency=2 len(table.Parts[...])=0 logger=uploadTableData
2024/04/01 07:25:17.361890 debug finish distributed_1.dt_data_index with concurrency=2 len(table.Parts[...])=0 uploadedFiles=map[], uploadedBytes=0 logger=uploadTableData
2024/04/01 07:25:17.361943 debug start distributed_1.dt_alert with concurrency=2 len(table.Parts[...])=0 logger=uploadTableData
2024/04/01 07:25:17.361895 debug start distributed_1.dt_arms_service_group_app with concurrency=2 len(table.Parts[...])=0 logger=uploadTableData
2024/04/01 07:25:17.361981 debug finish distributed_1.dt_arms_service_group_app with concurrency=2 len(table.Parts[...])=0 uploadedFiles=map[], uploadedBytes=0 logger=uploadTableData
2024/04/01 07:25:17.361952 debug finish distributed_1.dt_alert with concurrency=2 len(table.Parts[...])=0 uploadedFiles=map[], uploadedBytes=0 logger=uploadTableData
2024/04/01 07:25:17.362157 debug start distributed_1.dt_arms_app with concurrency=2 len(table.Parts[...])=0 logger=uploadTableData
2024/04/01 07:25:17.362170 debug start distributed_1.dt_arms_service_group with concurrency=2 len(table.Parts[...])=0 logger=uploadTableData
2024/04/01 07:25:17.362188 debug finish distributed_1.dt_arms_service_group with concurrency=2 len(table.Parts[...])=0 uploadedFiles=map[], uploadedBytes=0 logger=uploadTableData
2024/04/01 07:25:17.362252 debug start distributed_1.dt_chart with concurrency=2 len(table.Parts[...])=0 logger=uploadTableData
2024/04/01 07:25:17.362176 debug finish distributed_1.dt_arms_app with concurrency=2 len(table.Parts[...])=0 uploadedFiles=map[], uploadedBytes=0 logger=uploadTableData
2024/04/01 07:25:17.362429 debug start distributed_1.dt_resource_optimization_suggest with concurrency=2 len(table.Parts[...])=0 logger=uploadTableData
2024/04/01 07:25:17.362452 debug finish distributed_1.dt_resource_optimization_suggest with concurrency=2 len(table.Parts[...])=0 uploadedFiles=map[], uploadedBytes=0 logger=uploadTableData
2024/04/01 07:25:17.362514 debug start distributed_1.dt_resource with concurrency=2 len(table.Parts[...])=0 logger=uploadTableData
2024/04/01 07:25:17.362530 debug finish distributed_1.dt_resource with concurrency=2 len(table.Parts[...])=0 uploadedFiles=map[], uploadedBytes=0 logger=uploadTableData
2024/04/01 07:25:17.362609 debug start distributed_1.dt_inspect_template with concurrency=2 len(table.Parts[...])=0 logger=uploadTableData
2024/04/01 07:25:17.362624 debug finish distributed_1.dt_inspect_template with concurrency=2 len(table.Parts[...])=0 uploadedFiles=map[], uploadedBytes=0 logger=uploadTableData
2024/04/01 07:25:17.362689 debug start distributed_1.dt_inspect_record with concurrency=2 len(table.Parts[...])=0 logger=uploadTableData
2024/04/01 07:25:17.362703 debug finish distributed_1.dt_inspect_record with concurrency=2 len(table.Parts[...])=0 uploadedFiles=map[], uploadedBytes=0 logger=uploadTableData
2024/04/01 07:25:17.362753 debug start distributed_1.dt_host with concurrency=2 len(table.Parts[...])=0 logger=uploadTableData
2024/04/01 07:25:17.362764 debug finish distributed_1.dt_host with concurrency=2 len(table.Parts[...])=0 uploadedFiles=map[], uploadedBytes=0 logger=uploadTableData
2024/04/01 07:25:17.362810 debug start distributed_1.dt_collect_model with concurrency=2 len(table.Parts[...])=0 logger=uploadTableData
2024/04/01 07:25:17.362821 debug finish distributed_1.dt_collect_model with concurrency=2 len(table.Parts[...])=0 uploadedFiles=map[], uploadedBytes=0 logger=uploadTableData
2024/04/01 07:25:17.362869 debug start distributed_1.dt_dashboard with concurrency=2 len(table.Parts[...])=0 logger=uploadTableData
2024/04/01 07:25:17.362883 debug finish distributed_1.dt_dashboard with concurrency=2 len(table.Parts[...])=0 uploadedFiles=map[], uploadedBytes=0 logger=uploadTableData
2024/04/01 07:25:17.362937 debug start distributed_1.dt_collect_model_task with concurrency=2 len(table.Parts[...])=0 logger=uploadTableData
2024/04/01 07:25:17.362951 debug finish distributed_1.dt_collect_model_task with concurrency=2 len(table.Parts[...])=0 uploadedFiles=map[], uploadedBytes=0 logger=uploadTableData
2024/04/01 07:25:17.363023 debug start distributed_1.arms_frontend_log with concurrency=2 len(table.Parts[...])=0 logger=uploadTableData
2024/04/01 07:25:17.363052 debug finish distributed_1.arms_frontend_log with concurrency=2 len(table.Parts[...])=0 uploadedFiles=map[], uploadedBytes=0 logger=uploadTableData
2024/04/01 07:25:17.363117 debug start distributed_1.easydo_agent_monitor with concurrency=2 len(table.Parts[...])=0 logger=uploadTableData
2024/04/01 07:25:17.363143 debug finish distributed_1.easydo_agent_monitor with concurrency=2 len(table.Parts[...])=0 uploadedFiles=map[], uploadedBytes=0 logger=uploadTableData
2024/04/01 07:25:17.363212 debug start distributed_1.system_metrics with concurrency=2 len(table.Parts[...])=0 logger=uploadTableData
2024/04/01 07:25:17.363227 debug finish distributed_1.system_metrics with concurrency=2 len(table.Parts[...])=0 uploadedFiles=map[], uploadedBytes=0 logger=uploadTableData
2024/04/01 07:25:17.363305 debug start distributed_1.easydo_collect_task_monitor with concurrency=2 len(table.Parts[...])=0 logger=uploadTableData
2024/04/01 07:25:17.363320 debug finish distributed_1.easydo_collect_task_monitor with concurrency=2 len(table.Parts[...])=0 uploadedFiles=map[], uploadedBytes=0 logger=uploadTableData
2024/04/01 07:25:17.363392 debug start distributed_1.metrics_summary with concurrency=2 len(table.Parts[...])=0 logger=uploadTableData
2024/04/01 07:25:17.363407 debug finish distributed_1.metrics_summary with concurrency=2 len(table.Parts[...])=0 uploadedFiles=map[], uploadedBytes=0 logger=uploadTableData
2024/04/01 07:25:17.363507 debug start distributed_1.system_attribute_meta with concurrency=2 len(table.Parts[...])=0 logger=uploadTableData
2024/04/01 07:25:17.363522 debug finish distributed_1.system_attribute_meta with concurrency=2 len(table.Parts[...])=0 uploadedFiles=map[], uploadedBytes=0 logger=uploadTableData
2024/04/01 07:25:17.363623 debug start distributed_1.ddd with concurrency=2 len(table.Parts[...])=0 logger=uploadTableData
2024/04/01 07:25:17.363641 debug finish distributed_1.ddd with concurrency=2 len(table.Parts[...])=0 uploadedFiles=map[], uploadedBytes=0 logger=uploadTableData
2024/04/01 07:25:17.362270 debug finish distributed_1.dt_chart with concurrency=2 len(table.Parts[...])=0 uploadedFiles=map[], uploadedBytes=0 logger=uploadTableData
2024/04/01 07:25:17.363803 info clickhouse connection closed logger=clickhouse
2024/04/01 07:25:17.363825 error metrics.ExecuteWithMetrics(create_remote) return error: one of upload table go-routine return error: can't open /var/lib/clickhouse/disks/backups_s3/shard0-full-20240401072515/metadata/dtsmart/dt_inspect_template.sql: open /var/lib/clickhouse/disks/backups_s3/shard0-full-20240401072515/metadata/dtsmart/dt_inspect_template.sql: no such file or directory logger=metrics
2024/04/01 07:25:17.363860 info clickhouse connection prepared: tcp://localhost:9000 run ping logger=clickhouse
2024/04/01 07:25:17.365424 info clickhouse connection open: tcp://localhost:9000 logger=clickhouse
2024/04/01 07:25:17.365453 info SELECT countIf(name='type') AS is_disk_type_present, countIf(name='free_space') AS is_free_space_present, countIf(name='disks') AS is_storage_policy_present FROM system.columns WHERE database='system' AND table IN ('disks','storage_policies') logger=clickhouse
2024/04/01 07:25:17.369019 info SELECT d.path, any(d.name) AS name, any(d.type) AS type, min(d.free_space) AS free_space, groupUniqArray(s.policy_name) AS storage_policies FROM system.disks AS d LEFT JOIN (SELECT policy_name, arrayJoin(disks) AS disk FROM system.storage_policies) AS s ON s.disk = d.name GROUP BY d.path logger=clickhouse
2024/04/01 07:25:17.373231 info SELECT count() AS is_macros_exists FROM system.tables WHERE database='system' AND name='macros' SETTINGS empty_result_for_aggregation_by_empty_set=0 logger=clickhouse
2024/04/01 07:25:17.375608 info SELECT macro, substitution FROM system.macros logger=clickhouse
2024/04/01 07:25:17.376869 info SELECT count() AS is_macros_exists FROM system.tables WHERE database='system' AND name='macros' SETTINGS empty_result_for_aggregation_by_empty_set=0 logger=clickhouse
2024/04/01 07:25:17.379352 info SELECT macro, substitution FROM system.macros logger=clickhouse
2024/04/01 07:25:17.381145 info [s3:DEBUG] Request
GET /acos-service?versioning= HTTP/1.1
Host: minio:9000
User-Agent: aws-sdk-go-v2/1.24.1 os/linux lang/go#1.22.1 md/GOOS#linux md/GOARCH#amd64 api/s3#1.48.1
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: 00f06d8e-d121-467d-801f-7f8459ce753c
Amz-Sdk-Request: attempt=1; max=3
Authorization: AWS4-HMAC-SHA256 Credential=acos/20240401/us-east-1/s3/aws4_request, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date, Signature=2d93df77131521eb68188fd28a02563c929b601efea0e9984b5e0de27be927c6
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20240401T072517Z
2024/04/01 07:25:17.384393 info [s3:DEBUG] Response
HTTP/1.1 200 OK
Content-Length: 99
Accept-Ranges: bytes
Content-Security-Policy: block-all-mixed-content
Content-Type: application/xml
Date: Mon, 01 Apr 2024 07:25:46 GMT
Server: MinIO
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin
Vary: Accept-Encoding
X-Amz-Request-Id: 17C2173A0952385E
X-Content-Type-Options: nosniff
X-Xss-Protection: 1; mode=block
2024/04/01 07:25:17.384548 debug /tmp/.clickhouse-backup-metadata.cache.S3 load 0 elements logger=s3
2024/04/01 07:25:17.384987 info [s3:DEBUG] Request
GET /acos-service?delimiter=%2F&list-type=2&max-keys=1000&prefix=backup%2Fshard-0%2F HTTP/1.1
Host: minio:9000
User-Agent: aws-sdk-go-v2/1.24.1 os/linux lang/go#1.22.1 md/GOOS#linux md/GOARCH#amd64 api/s3#1.48.1
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: 621568fe-08bf-4780-9087-1f2b4f49cfad
Amz-Sdk-Request: attempt=1; max=3
Authorization: AWS4-HMAC-SHA256 Credential=acos/20240401/us-east-1/s3/aws4_request, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date, Signature=cad5c2a336642a3c46c8f1376d83855e1cbbc165c915499f4296a787e485544c
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20240401T072517Z
2024/04/01 07:25:17.386492 info [s3:DEBUG] Response
HTTP/1.1 200 OK
Content-Length: 282
Accept-Ranges: bytes
Content-Security-Policy: block-all-mixed-content
Content-Type: application/xml
Date: Mon, 01 Apr 2024 07:25:46 GMT
Server: MinIO
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin
Vary: Accept-Encoding
X-Amz-Request-Id: 17C2173A09702440
X-Content-Type-Options: nosniff
X-Xss-Protection: 1; mode=block
2024/04/01 07:25:17.386726 debug /tmp/.clickhouse-backup-metadata.cache.S3 save 0 elements logger=s3
2024/04/01 07:25:17.386817 debug b.skipIfTheSameRemoteBackupPresent return skip=false logger=RemoveBackupLocal
2024/04/01 07:25:17.386850 info SELECT metadata_path FROM system.tables WHERE database = 'system' AND metadata_path!='' LIMIT 1; logger=clickhouse
2024/04/01 07:25:17.389653 warn b.cleanLocalEmbedded return error: open /var/lib/clickhouse/preprocessed_configs/config.xml: no such file or directory logger=RemoveBackupLocal
2024/04/01 07:25:17.389720 info clickhouse connection closed logger=clickhouse
2024/04/01 07:25:17.389743 error metrics.ExecuteWithMetrics(delete) return error: open /var/lib/clickhouse/preprocessed_configs/config.xml: no such file or directory logger=metrics
2024/04/01 07:25:17.389766 error Watch error: too many errors create_remote: 1, delete local: 1, during watch full_interval: 24h, abort watching logger=server
2024/04/01 07:25:17.389794 debug api.status.stop -> status.commands[4] == {ActionRowStatus:{Command:watch --watch-interval="8h" --full-interval="24h" Status:error Start:2024-04-01 07:25:15 Finish:2024-04-01 07:25:17 Error:too many errors create_remote: 1, delete local: 1, during watch full_interval: 24h, abort watching} Ctx:<nil> Cancel:<nil>} logger=status
2024/04/01 07:25:17.857006 debug API call GET /metrics logger=server
/var/lib/clickhouse/preprocessed_configs/config.xml: no such file or directory
/var/lib/clickhouse
from container where run clickhouse-server
shall be available in the clickhouse-backup
container
Could you share without sensitive information?
kubectl get chi -n <your-namespace> clickhouse -o yaml
apiVersion: clickhouse.altinity.com/v1
kind: ClickHouseInstallation
metadata:
annotations:
meta.helm.sh/release-name: acos
meta.helm.sh/release-namespace: acos
creationTimestamp: "2024-04-01T06:07:44Z"
finalizers:
- finalizer.clickhouseinstallation.altinity.com
generation: 6
labels:
app.kubernetes.io/managed-by: Helm
name: clickhouse-cluster
namespace: acos
resourceVersion: "44148484"
uid: 445ab6c3-287b-4545-a9e0-85ea8ef1bae3
spec:
configuration:
clusters:
- layout:
replicasCount: 1
shardsCount: 1
name: xxxxx
secret:
value: xxxxx
users:
default/networks/ip: ::/0
default/password: xxxxx
zookeeper:
nodes:
- host: clickhouse-keeper
port: 2181
defaults:
templates:
podTemplate: clickhouse
serviceTemplate: clickhouse-clickhouse-cluster
templates:
podTemplates:
- metadata:
annotations:
prometheus.io/scrape: "true"
prometheus.io/tcp-probe: "true"
prometheus.io/tcp-probe-port: 8123,9000
labels:
app: clickhouse
name: clickhouse
spec:
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: app
operator: In
values:
- clickhouse
topologyKey: kubernetes.io/hostname
containers:
- image: altinity/clickhouse-server:23.8.8.21.altinitystable
name: clickhouse-cluster
resources:
limits:
cpu: 2
memory: 4G
requests:
cpu: 2
memory: 4G
volumeMounts:
- mountPath: /etc/clickhouse-server/config.xml
name: chi-clickhouse-cluster-config-file
subPath: config.xml
- mountPath: /var/lib/clickhouse/backups_embedded
name: backups-s3
- mountPath: /var/lib/clickhouse
name: ck-data-dir
- command:
- bash
- -xc
- /bin/clickhouse-backup server
env:
- name: S3_COMPRESSION_FORMAT
value: none
- name: API_ALLOW_PARALLEL
value: "true"
- name: CLICKHOUSE_EMBEDDED_BACKUP_DISK
value: backups_s3
- name: CLICKHOUSE_TIMEOUT
value: 4h
- name: CLICKHOUSE_PASSWORD
value: xxxxx
- name: CLICKHOUSE_USE_EMBEDDED_BACKUP_RESTORE
value: "true"
- name: LOG_LEVEL
value: debug
- name: ALLOW_EMPTY_BACKUPS
value: "true"
- name: API_LISTEN
value: 0.0.0.0:7171
- name: API_CREATE_INTEGRATION_TABLES
value: "true"
- name: BACKUPS_TO_KEEP_REMOTE
value: "24"
- name: REMOTE_STORAGE
value: s3
- name: S3_ACL
value: private
- name: S3_ENDPOINT
value: http://minio:9000
- name: S3_BUCKET
value: acos-service
- name: S3_PATH
value: backup/shard-{shard}
- name: S3_ACCESS_KEY
value: acos
- name: S3_SECRET_KEY
value: xxxxx
- name: S3_FORCE_PATH_STYLE
value: "false"
- name: S3_DEBUG
value: "true"
image: altinity/clickhouse-backup:master
imagePullPolicy: IfNotPresent
name: clickhouse-backup
ports:
- containerPort: 7171
name: backup-rest
securityContext:
fsGroup: 101
runAsGroup: 101
runAsUser: 101
volumes:
- configMap:
name: chi-clickhouse-cluster-config-file
name: chi-clickhouse-cluster-config-file
serviceTemplates:
- generateName: clickhouse-cluster
name: clickhouse-clickhouse-cluster
spec:
ports:
- name: native
port: 9000
targetPort: 9000
- name: http
port: 8123
targetPort: 8123
- name: prometheus
port: 8001
targetPort: 8001
type: ClusterIP
volumeClaimTemplates:
- name: ck-data-dir
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 50Gi
storageClassName: open-local-lvm-xfs
- name: backups-s3
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 50Gi
storageClassName: open-local-lvm-xfs
you don't need separate volume for
/var/lib/clickhouse/backups_embedded
remove
- mountPath: /var/lib/clickhouse/backups_embedded
name: backups-s3
and second volumeClaimTemplates
item
- name: backups-s3
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 50Gi
storageClassName: open-local-lvm-xfs
and add
- mountPath: /var/lib/clickhouse
name: ck-data-dir
to second container's item with name: clichouse-backup
deploy config
<storage_configuration>
<disks>
<backups_s3>
<type>s3</type>
<endpoint>http://minio:9000/acos/{cluster}/{shard}/</endpoint>
<!-- https://github.com/Altinity/clickhouse-backup/issues/691
<access_key_id>acos</access_key_id>
<secret_access_key>xxxxxx</secret_access_key>
-->
<use_environment_credentials>1</use_environment_credentials>
<cache_enabled>false</cache_enabled>
<!-- to avoid slow startup -->
<send_metadata>false</send_metadata>
</backups_s3>
</disks>
</storage_configuration>
If my MinIO bucket is private, I will encounter the following error.
<Error> Application: DB::Exception: Message: Access Denied., bucket acos, key dtstack/0/yvl/xsfmxmntolofwnmjjwxoqyqliydrf, object size 4: While checking access for disk backups_s3
Switching it to public resolves the issue, but I prefer to use it as private.
If my MinIO bucket is private, I will encounter the following error
uncomment credentials in XML config for backups_s3
or add into name: clickhouse-cluster
container env
section
- name: AWS_ACCESS_KEY_ID
value: your_access_key
- name: AWS_SECRET_ACCESS_KEY
valoue: your_my_super_secret_key
When I use
curl -X POST "chi-clickhouse-cluster-dtstack-0-0:7171/backup/watch?watch_interval=1h&full_interval=24h"
I can see that it has been uploaded to MinIO. However, when I use list to view it, I can only see the local backup and not the remote one.
curl chi-clickhouse-cluster-dtstack-1-0:7171/backup/list
{"name":"shard1-full-20240401100823","created":"2024-04-01 10:08:24","size":652722,"location":"local","required":"","desc":"embedded"}
What is currently stored in MinIO? Do I need to manually upload local backups to MinIO? I'm not quite understanding the logic here.
Another issue I encountered is that when I use incremental backup, I get an error:
{"command":"watch --watch-interval=\"8h\" --full-interval=\"24h\"","status":"error","start":"2024-04-01 10:03:43","finish":"2024-04-01 10:09:49","error":"too many errors create_remote: 4, delete local: 0, during watch full_interval: 24h, abort watching"}.
Why is this error happening? Could it be because I have set BACKUPS_TO_KEEP_REMOTE
to 3, and it is insufficient?
Why is this error happening?
Look to clickhouse-backup
container logs
2024/04/01 10:28:40.972282 debug object_disk.ReadMetadataFromFile(/var/lib/clickhouse/disks/backups_s3/shard0-full-20240401102709/metadata/local_1/easydo_collect_task_monitor.sql)
2024/04/01 10:28:40.972541 info [s3:DEBUG] Request
DELETE /acos/dtstack/0/iqp/wxziuqupsdmtsnomvpbjpurlicbih?x-id=DeleteObject HTTP/1.1
Host: minio:9000
User-Agent: aws-sdk-go-v2/1.24.1 os/linux lang/go#1.22.1 md/GOOS#linux md/GOARCH#amd64 api/s3#1.48.1
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: c312dcdc-c381-47a7-824b-4f95b36f610b
Amz-Sdk-Request: attempt=1; max=3
Authorization: AWS4-HMAC-SHA256 Credential=acos/20240401/us-east-1/s3/aws4_request, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date, Signature=304779e9f178a9aa556ae0fb108609d50145f671100c8ba4d65159cf02c660e5
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20240401T102840Z
2024/04/01 10:28:40.974231 info [s3:DEBUG] Response
HTTP/1.1 204 No Content
Accept-Ranges: bytes
Content-Security-Policy: block-all-mixed-content
Date: Mon, 01 Apr 2024 10:28:40 GMT
Server: MinIO
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin
Vary: Accept-Encoding
X-Amz-Request-Id: 17C221353051ED33
X-Content-Type-Options: nosniff
X-Xss-Protection: 1; mode=block
2024/04/01 10:28:40.974281 debug object_disk.ReadMetadataFromFile(/var/lib/clickhouse/disks/backups_s3/shard0-full-20240401102709/metadata/local_1/metrics_summary.sql)
2024/04/01 10:28:40.974509 info [s3:DEBUG] Request
DELETE /acos/dtstack/0/nxh/wshqwwmtfuwgrcojxamcucvenzznd?x-id=DeleteObject HTTP/1.1
Host: minio:9000
User-Agent: aws-sdk-go-v2/1.24.1 os/linux lang/go#1.22.1 md/GOOS#linux md/GOARCH#amd64 api/s3#1.48.1
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: 5961e37a-4ce9-4c6e-84f1-469e9f9ee229
Amz-Sdk-Request: attempt=1; max=3
Authorization: AWS4-HMAC-SHA256 Credential=acos/20240401/us-east-1/s3/aws4_request, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date, Signature=8a15eff3ad6b098f87e711c98284fdd04157a7adf7c1f056c41158cb009a1064
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20240401T102840Z
2024/04/01 10:28:40.976464 info [s3:DEBUG] Response
HTTP/1.1 204 No Content
Accept-Ranges: bytes
Content-Security-Policy: block-all-mixed-content
Date: Mon, 01 Apr 2024 10:28:40 GMT
Server: MinIO
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin
Vary: Accept-Encoding
X-Amz-Request-Id: 17C2213530707305
X-Content-Type-Options: nosniff
X-Xss-Protection: 1; mode=block
2024/04/01 10:28:40.976513 debug object_disk.ReadMetadataFromFile(/var/lib/clickhouse/disks/backups_s3/shard0-full-20240401102709/metadata/local_1/system_attribute_meta.sql)
2024/04/01 10:28:40.976743 info [s3:DEBUG] Request
DELETE /acos/dtstack/0/vyt/lxbvewmgrhrcunnnckytkoytispbd?x-id=DeleteObject HTTP/1.1
Host: minio:9000
User-Agent: aws-sdk-go-v2/1.24.1 os/linux lang/go#1.22.1 md/GOOS#linux md/GOARCH#amd64 api/s3#1.48.1
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: 5cd0e3ac-fe31-47fb-b62a-8af8c32601dc
Amz-Sdk-Request: attempt=1; max=3
Authorization: AWS4-HMAC-SHA256 Credential=acos/20240401/us-east-1/s3/aws4_request, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date, Signature=44b403836a7f37a2107392733138ee1c8bcf020fe1a0f021ce1fbc93b0940b17
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20240401T102840Z
2024/04/01 10:28:40.978707 info [s3:DEBUG] Response
HTTP/1.1 204 No Content
Accept-Ranges: bytes
Content-Security-Policy: block-all-mixed-content
Date: Mon, 01 Apr 2024 10:28:40 GMT
Server: MinIO
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin
Vary: Accept-Encoding
X-Amz-Request-Id: 17C22135309292BA
X-Content-Type-Options: nosniff
X-Xss-Protection: 1; mode=block
2024/04/01 10:28:40.978765 debug object_disk.ReadMetadataFromFile(/var/lib/clickhouse/disks/backups_s3/shard0-full-20240401102709/metadata/local_1/system_metrics.sql)
2024/04/01 10:28:40.979045 info [s3:DEBUG] Request
DELETE /acos/dtstack/0/txp/kgrfqgtcwpsmrzjzhhihnkqqlvyzp?x-id=DeleteObject HTTP/1.1
Host: minio:9000
User-Agent: aws-sdk-go-v2/1.24.1 os/linux lang/go#1.22.1 md/GOOS#linux md/GOARCH#amd64 api/s3#1.48.1
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: 9adaca0e-987a-4a95-b14e-d26d1ea11649
Amz-Sdk-Request: attempt=1; max=3
Authorization: AWS4-HMAC-SHA256 Credential=acos/20240401/us-east-1/s3/aws4_request, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date, Signature=744a67a4774dd105fb54e2f6dd4dc02c7b24a830ad40586c7a656b2d3adde12b
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20240401T102840Z
2024/04/01 10:28:40.980732 info [s3:DEBUG] Response
HTTP/1.1 204 No Content
Accept-Ranges: bytes
Content-Security-Policy: block-all-mixed-content
Date: Mon, 01 Apr 2024 10:28:40 GMT
Server: MinIO
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin
Vary: Accept-Encoding
X-Amz-Request-Id: 17C2213530B5B333
X-Content-Type-Options: nosniff
X-Xss-Protection: 1; mode=block
2024/04/01 10:28:40.980779 debug remove '/var/lib/clickhouse/backup/shard0-full-20240401102709' logger=RemoveBackupLocal
2024/04/01 10:28:40.980805 debug remove '/var/lib/clickhouse/disks/backups_s3/shard0-full-20240401102709' logger=RemoveBackupLocal
2024/04/01 10:28:40.986507 info done backup=shard0-full-20240401102709 duration=443ms location=local logger=RemoveBackupLocal operation=delete
2024/04/01 10:28:40.986607 info clickhouse connection closed logger=clickhouse
2024/04/01 10:28:40.986636 error Watch error: too many errors create_remote: 4, delete local: 0, during watch full_interval: 24h, abort watching logger=server
2024/04/01 10:28:40.986689 debug api.status.stop -> status.commands[9] == {ActionRowStatus:{Command:watch --watch-interval="12h" --full-interval="24h" Status:error Start:2024-04-01 10:22:35 Finish:2024-04-01 10:28:40 Error:too many errors create_remote: 4, delete local: 0, during watch full_interval: 24h, abort watching} Ctx:<nil> Cancel:<nil>} logger=status
When I use
curl -X POST "chi-clickhouse-cluster-dtstack-0-0:7171/backup/watch?watch_interval=1h&full_interval=24h"
I can see that it has been uploaded to MinIO. However, when I use list to view it, I can only see the local backup and not the remote one.curl chi-clickhouse-cluster-dtstack-1-0:7171/backup/list {"name":"shard1-full-20240401100823","created":"2024-04-01 10:08:24","size":652722,"location":"local","required":"","desc":"embedded"}
What is currently stored in MinIO? Do I need to manually upload local backups to MinIO? I'm not quite understanding the logic here.
Could you please help take a look at this?
Could you please help take a look at this?
You shared not the full log from clickhouse-backup container disable S3_DEBUG=true in env and share the logs
2024/04/01 12:19:53.136846 warn BackupList bd.Walk return error: operation error S3: ListObjectsV2, https response error StatusCode: 404, RequestID: 17C22746AC361F67, HostID: 2fd203aa-24e1-4c1a-b55a-529508ffeeb9, NoSuchBucket: logger=s3
check your current configuration
use for minio
- name: S3_FORCE_PATH_STYLE
value: true
2024/04/01 12:19:54.010273 warn open /var/lib/clickhouse/disks/backups_s3/shard0-full-20240401121953/data/distributed_1/arms_frontend_log: no such file or directory logger=getTableListByPatternLocal
you not applied my recommendation from https://github.com/Altinity/clickhouse-backup/issues/876#issuecomment-2029349320
2024/04/01 12:19:54.010273 警告打开 /var/lib/clickhouse/disks/backups_s3/shard0-full-20240401121953/data/distributed_1/arms_frontend_log:没有这样的文件或目录 logger=getTableListByPatternLocal
你没有应用我在#876 中的建议(评论)
I have implemented your advice.
apiVersion: v1
kind: Pod
metadata:
annotations:
cni.projectcalico.org/containerID: 850401bf36cd87eb449a943dde9538bd0690281860ec3455b302dacd492f2367
cni.projectcalico.org/podIP: 100.107.53.47/32
cni.projectcalico.org/podIPs: 100.107.53.47/32
meta.helm.sh/release-name: acos
meta.helm.sh/release-namespace: acos
prometheus.io/scrape: "true"
prometheus.io/tcp-probe: "true"
prometheus.io/tcp-probe-port: 8123,9000
creationTimestamp: "2024-04-01T12:16:11Z"
generateName: chi-clickhouse-cluster-dtstack-0-0-
labels:
app: clickhouse
app.kubernetes.io/managed-by: Helm
clickhouse.altinity.com/app: chop
clickhouse.altinity.com/chi: clickhouse-cluster
clickhouse.altinity.com/cluster: dtstack
clickhouse.altinity.com/namespace: acos
clickhouse.altinity.com/ready: "yes"
clickhouse.altinity.com/replica: "0"
clickhouse.altinity.com/shard: "0"
controller-revision-hash: chi-clickhouse-cluster-dtstack-0-0-564cc64f95
statefulset.kubernetes.io/pod-name: chi-clickhouse-cluster-dtstack-0-0-0
name: chi-clickhouse-cluster-dtstack-0-0-0
namespace: acos
ownerReferences:
- apiVersion: apps/v1
blockOwnerDeletion: true
controller: true
kind: StatefulSet
name: chi-clickhouse-cluster-dtstack-0-0
uid: b44d04d7-9eb4-45b9-ad54-891cb4776331
resourceVersion: "44525886"
uid: 5c3fe1cb-2849-4f58-b61c-8741b530717c
spec:
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: app
operator: In
values:
- clickhouse
topologyKey: kubernetes.io/hostname
containers:
- image: altinity/clickhouse-server:23.8.8.21.altinitystable
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 10
httpGet:
path: /ping
port: http
scheme: HTTP
initialDelaySeconds: 60
periodSeconds: 3
successThreshold: 1
timeoutSeconds: 1
name: clickhouse-cluster
ports:
- containerPort: 9000
name: tcp
protocol: TCP
- containerPort: 8123
name: http
protocol: TCP
- containerPort: 9009
name: interserver
protocol: TCP
readinessProbe:
failureThreshold: 3
httpGet:
path: /ping
port: http
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 3
successThreshold: 1
timeoutSeconds: 1
resources:
limits:
cpu: "2"
memory: 4G
requests:
cpu: "2"
memory: 4G
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /etc/clickhouse-server/config.xml
name: chi-clickhouse-cluster-config-file
subPath: config.xml
- mountPath: /etc/clickhouse-server/config.d/
name: chi-clickhouse-cluster-common-configd
- mountPath: /etc/clickhouse-server/users.d/
name: chi-clickhouse-cluster-common-usersd
- mountPath: /etc/clickhouse-server/conf.d/
name: chi-clickhouse-cluster-deploy-confd-dtstack-0-0
- mountPath: /var/lib/clickhouse
name: ck-data-dir
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: kube-api-access-cp66k
readOnly: true
- command:
- bash
- -xc
- /bin/clickhouse-backup server
env:
- name: S3_COMPRESSION_FORMAT
value: none
- name: API_ALLOW_PARALLEL
value: "true"
- name: CLICKHOUSE_EMBEDDED_BACKUP_DISK
value: backups_s3
- name: CLICKHOUSE_TIMEOUT
value: 4h
- name: CLICKHOUSE_PASSWORD
value: abc123
- name: CLICKHOUSE_USE_EMBEDDED_BACKUP_RESTORE
value: "true"
- name: LOG_LEVEL
value: debug
- name: ALLOW_EMPTY_BACKUPS
value: "true"
- name: API_LISTEN
value: 0.0.0.0:7171
- name: API_CREATE_INTEGRATION_TABLES
value: "true"
- name: BACKUPS_TO_KEEP_REMOTE
value: "3"
- name: REMOTE_STORAGE
value: s3
- name: S3_ACL
value: private
- name: S3_ENDPOINT
value: http://minio:9000
- name: S3_BUCKET
value: acos-service
- name: S3_PATH
value: backup/shard-{shard}
- name: S3_ACCESS_KEY
value: acos
- name: S3_SECRET_KEY
value: DmPWhh6f
- name: S3_FORCE_PATH_STYLE
value: "false"
- name: S3_DEBUG
value: "false"
image: altinity/clickhouse-backup:master
imagePullPolicy: IfNotPresent
name: clickhouse-backup
ports:
- containerPort: 7171
name: backup-rest
protocol: TCP
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /var/lib/clickhouse
name: ck-data-dir
- mountPath: /etc/clickhouse-server/config.d/
name: chi-clickhouse-cluster-common-configd
- mountPath: /etc/clickhouse-server/users.d/
name: chi-clickhouse-cluster-common-usersd
- mountPath: /etc/clickhouse-server/conf.d/
name: chi-clickhouse-cluster-deploy-confd-dtstack-0-0
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: kube-api-access-cp66k
readOnly: true
dnsPolicy: ClusterFirst
enableServiceLinks: true
hostAliases:
- hostnames:
- chi-clickhouse-cluster-dtstack-0-0
ip: 127.0.0.1
hostname: chi-clickhouse-cluster-dtstack-0-0-0
nodeName: 172-16-121-243
preemptionPolicy: PreemptLowerPriority
priority: 0
restartPolicy: Always
schedulerName: default-scheduler
securityContext:
fsGroup: 101
runAsGroup: 101
runAsUser: 101
serviceAccount: default
serviceAccountName: default
subdomain: chi-clickhouse-cluster-dtstack-0-0
terminationGracePeriodSeconds: 30
tolerations:
- effect: NoExecute
key: node.kubernetes.io/not-ready
operator: Exists
tolerationSeconds: 300
- effect: NoExecute
key: node.kubernetes.io/unreachable
operator: Exists
tolerationSeconds: 300
volumes:
- name: ck-data-dir
persistentVolumeClaim:
claimName: ck-data-dir-chi-clickhouse-cluster-dtstack-0-0-0
- configMap:
defaultMode: 420
name: chi-clickhouse-cluster-config-file
name: chi-clickhouse-cluster-config-file
- configMap:
defaultMode: 420
name: chi-clickhouse-cluster-common-configd
name: chi-clickhouse-cluster-common-configd
- configMap:
defaultMode: 420
name: chi-clickhouse-cluster-common-usersd
name: chi-clickhouse-cluster-common-usersd
- configMap:
defaultMode: 420
name: chi-clickhouse-cluster-deploy-confd-dtstack-0-0
name: chi-clickhouse-cluster-deploy-confd-dtstack-0-0
- name: kube-api-access-cp66k
projected:
defaultMode: 420
sources:
- serviceAccountToken:
expirationSeconds: 3607
path: token
- configMap:
items:
- key: ca.crt
path: ca.crt
name: kube-root-ca.crt
- downwardAPI:
items:
- fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
path: namespace
status:
conditions:
- lastProbeTime: null
lastTransitionTime: "2024-04-01T12:16:07Z"
status: "True"
type: Initialized
- lastProbeTime: null
lastTransitionTime: "2024-04-01T12:16:22Z"
status: "True"
type: Ready
- lastProbeTime: null
lastTransitionTime: "2024-04-01T12:16:22Z"
status: "True"
type: ContainersReady
- lastProbeTime: null
lastTransitionTime: "2024-04-01T12:16:11Z"
status: "True"
type: PodScheduled
containerStatuses:
- containerID: containerd://12b94ae358c957219c946341262af0c1dff9663e255d81888f08c91dc2dede84
image: docker.io/altinity/clickhouse-backup:master
imageID: docker.io/altinity/clickhouse-backup@sha256:c70ede914745fd01da1de51f2fe52e8d38cb371a2660305da5c4685c25140cd2
lastState: {}
name: clickhouse-backup
ready: true
restartCount: 0
started: true
state:
running:
startedAt: "2024-04-01T12:16:12Z"
- containerID: containerd://95eabb9103b27b1184bc6007835c70a200df4496228bf744ba0ebdfac1b9caef
image: docker.io/altinity/clickhouse-server:23.8.8.21.altinitystable
imageID: docker.io/altinity/clickhouse-server@sha256:4a25044e9ceab10a6b7f938f7f7c0ae1e56fd9050e3ae260849287cb4cbfebd8
lastState: {}
name: clickhouse-cluster
ready: true
restartCount: 0
started: true
state:
running:
startedAt: "2024-04-01T12:16:11Z"
hostIP: 172.16.121.243
phase: Running
podIP: 100.107.53.47
podIPs:
- ip: 100.107.53.47
qosClass: Burstable
startTime: "2024-04-01T12:16:07Z"
ok. i see
image: altinity/clickhouse-backup:master
please use altinity/clickhouse-backup:latest
instead
and share logs from scracth with
- name: S3_DEBUG
value: "true"
@dxygit1 are you sure you install minio properly?
i see in logs
GET /acos-service?delimiter=%2F&list-type=2&max-keys=1000&prefix=backup%2Fshard-0%2F HTTP/1.1
Host: minio:9000
2024/04/02 01:26:47.921971 info [s3:DEBUG] Response
HTTP/1.1 404 Not Found
it should return 200
do you use minio
from
https://github.com/Altinity/clickhouse-backup/blob/master/Examples.md#how-to-use-clickhouse-backup-in-kubernetes AS IS?
could you share?
kubectl get svc --all-namespaces | grep minio
I'm really sorry, I have installed MinIO correctly, but I haven't created the 'acos-service' bucket. I'll try it out. Thank you for your assistance.
I still have a question. When the content of an incremental backup is uploaded to MinIO, how can we restore the incremental part? I understand that restoring the full backup can be done through the interface, but I haven't found any documentation on how to restore the incremental part. Did I overlook something?
During download incremental backup all required parts from chain of required backups will downloaded and locally you will have full backup with whole set of data parts
Okay, I've seen it. Thank you very much for your response.
Is the data stored here incremental? What is the purpose of this data? Will it be deleted during the next full backup?
acos/dstak/0 this is your backup data
also check S3_PATH in your settings and look to minio where stored backup_name/metadata.json
check
SELECT * FROM system.backup_list
inside clickhouse-server container
I understand that this is my backup data, but I would like to know what the contents of this backup data are. Is it a full backup of my current database data, or is it incremental data? What is its purpose? I assume I won't need to use the data in this bucket when restoring.
I have another bucket that has corresponding full and incremental backups
Or is the content of these two buckets the same?
so do you have two different buckets?
fist for backups_s3
disk which contains backup data
and second acos-service
for clickhouse-backup metadata which also contains references to keys inside
backup_name/shadow/db/table/part_name/*.tar
files
One is configured in the ClickHouse configuration file. corresponding data
One is configured with environment variables in ClickHouse Backup. corresponding data
Does this mean that these are two copies of the same data?
Does this mean that these are two copies of the same data?
No this is different data
backups_s3
disk contains physical backuped data
clickhouse-backup
bucket which you defined in S3 section contains metadata which allow restore backups_s3
properly
v2.5 will support empty value embedded_backup_disk: ""
and will store both metadata+data in the same bucket
does clickhouse-backup define that only metadata is stored in S3, and during restoration, it will search for the physical data in the backups_s3 disk?
v2.5 will support empty value embedded_backup_disk: "" and will store both metadata+data in the same bucket
When will v2.5 be released approximately?
does clickhouse-backup define that only metadata is stored in S3, and during restoration, it will search for the physical data in the backups_s3 disk?
When create
, then clickhouse-backup
execute BACKUP
SQL command on clickhouse-server
which upload physical data or execute CopyObject to s3 endpoint which you defined for backup_s3
disk and create metadata files on /var/lib/clickhouse/disks/backup_s3/backup_name/
folder, many small files which contains references to s3 keys inside s3 endpoint,
then clickhouse-backup
create /var/lib/clickhouse/disks/backups_s3/backup_name/metadata.json
and /var/lib/clickhouse/disks/backups_s3/backup_name/metadata/db/table_name.json
When upload
, then clickhouse-backup
upload content of /var/lib/clickhouse/disks/backups_s3/backup_name/
to s3 endpoint which you defined in s3 section of /etc/clickhouse-backup/config.yml or overrides in S3_*
environment variables
When download
then clickhouse-backup
download whole content from s3 endpoint /backup_name/ into /var/lib/clickhouse/disks/backups_s3 or any other disk which you defined as backup in clickhouse-server configuration
When restore
then clickhouse-backup
execute RESTORE
SQL command which read s3 keys from /var/lib/clickhouse/disks/backups_s3
and download data parts into /var/lib/clickhouse/tmp for local
disks or execute CopyObject
for object disks with <type>s3/azure</type>
After that will execute internal processes like ALTER TABLE ... ATTACH PART but more consistent (embedded backup allow backup *Log table engines)
It is clear?
Okay, I understand now. Thank you very much for your help.
error info
deploy