search

Altinity / clickhouse-backup

Tool for easy backup and restore for ClickHouse® using object storage for backup files.
https://altinity.com
Other
1.29k stars 227 forks source link

An error occurred while uploading backup to Cos #1053

Open liubo-it opened 1 day ago

liubo-it commented 1 day ago

command：

/data/tingyun/clickhouse-backup/clickhouse-backup   upload 2024-12-02 --config=/data/tingyun/clickhouse-backup/apm.yml

config

general:
  remote_storage: cos
  max_file_size: 107374182400
  disable_progress_bar: false
  backups_to_keep_local: 1
  backups_to_keep_remote: 32
  log_level: info
  allow_empty_backups: false
clickhouse:
  username: default
  password: xxx
  host: wukong-release-apmck-iossd-505.txbj.tingyun.com
  port: 9000
  disk_mapping: {}
  skip_tables:
  - system.*
  - information_schema.*
  - INFORMATION_SCHEMA.*
  timeout: 5m
  freeze_by_part: false
  secure: false
  skip_verify: false
  sync_replicated_tables: true
  skip_sync_replica_timeouts: true
  log_sql_queries: false
  check_replicas_before_attach: false
  check_parts_columns: false
cos:
  url: "https://wukong-apm-clickhouse-1251882848.cos.ap-beijing.myqcloud.com"
  timeout: 3m
  secret_id: "xxx"
  secret_key: "xxx"
  path: "apm/03/wukong-release-apmck-iossd-505.txbj.tingyun.com"
  compression_format: tar
  compression_level: 1

description An error occurred when I used upload to upload backup data to Cos.

error

2024-12-02 20:34:53.508 INF pkg/backup/upload.go:168 > done duration=65ms operation=upload_data progress=141/158 size=860B table=apm.svr_metric_data_baseline_application_instance_dist version=2.6.3
2024-12-02 20:34:53.555 INF pkg/backup/upload.go:168 > done duration=75ms operation=upload_data progress=142/158 size=650B table=aspm.aspm_risk_trace_local_request_count_dist version=2.6.3
2024-12-02 20:34:53.585 INF pkg/backup/upload.go:168 > done duration=77ms operation=upload_data progress=143/158 size=1.67KiB table=apm.svr_component_op_trace_dist version=2.6.3
2024-12-02 20:34:53.611 INF pkg/backup/upload.go:168 > done duration=56ms operation=upload_data progress=144/158 size=694B table=apm.svr_log_message_dist version=2.6.3
2024-12-02 20:34:53.631 INF pkg/backup/upload.go:168 > done duration=46ms operation=upload_data progress=145/158 size=634B table=aspm.aspm_risk_trace_local_user_count_dist version=2.6.3
2024-12-02 20:34:53.655 INF pkg/backup/upload.go:168 > done duration=44ms operation=upload_data progress=146/158 size=1.64KiB table=apm.svr_action_data_day_dist version=2.6.3
2024-12-02 20:34:53.672 INF pkg/backup/upload.go:168 > done duration=41ms operation=upload_data progress=147/158 size=844B table=apm.svr_component_trace_all version=2.6.3
2024-12-02 20:34:53.692 INF pkg/backup/upload.go:168 > done duration=37ms operation=upload_data progress=148/158 size=596B table=apm.svr_jar_class_index_dist version=2.6.3
2024-12-02 20:34:53.725 INF pkg/backup/upload.go:168 > done duration=53ms operation=upload_data progress=149/158 size=846B table=apm.svr_component_trace_dist version=2.6.3
2024-12-02 20:34:53.744 INF pkg/backup/upload.go:168 > done duration=52ms operation=upload_data progress=150/158 size=689B table=aspm.aspm_sqlquery_dist version=2.6.3
2024-12-02 20:34:53.783 INF pkg/backup/upload.go:168 > done duration=58ms operation=upload_data progress=151/158 size=1.37KiB table=apm.svr_action_data_baseline_server_service_group_dist version=2.6.3
2024-12-02 20:34:53.830 INF pkg/backup/upload.go:168 > done duration=86ms operation=upload_data progress=152/158 size=1.12KiB table=aspm.aspm_trace_risk_dist version=2.6.3
2024-12-02 20:34:53.856 INF pkg/backup/upload.go:168 > done duration=72ms operation=upload_data progress=153/158 size=541B table=apm.svr_health_targets_dist version=2.6.3
2024-12-02 20:34:53.882 INF pkg/backup/upload.go:168 > done duration=52ms operation=upload_data progress=154/158 size=1.83KiB table=apm.svr_error_trace_dist version=2.6.3
2024-12-02 20:34:53.902 INF pkg/backup/upload.go:168 > done duration=46ms operation=upload_data progress=155/158 size=1.36KiB table=apm.svr_action_data_baseline_server_biz_system_dist version=2.6.3
2024-12-02 20:34:53.917 INF pkg/backup/upload.go:168 > done duration=35ms operation=upload_data progress=156/158 size=782B table=apm.svr_health_events_dist version=2.6.3
2024-12-02 20:34:53.951 INF pkg/backup/upload.go:168 > done duration=48ms operation=upload_data progress=157/158 size=553B table=aspm.aspm_trace_total_dist version=2.6.3
2024-12-02 20:34:53.982 INF pkg/backup/upload.go:168 > done duration=64ms operation=upload_data progress=158/158 size=1.00KiB table=apm.svr_global_topo_edge_test_dist version=2.6.3
2024-12-02 20:34:59.230 INF pkg/backup/upload.go:168 > done duration=1m8.479s operation=upload_data progress=57/158 size=1.50GiB table=apm.svr_component_op_trace_local version=2.6.3
2024-12-02 20:35:54.099 INF pkg/backup/upload.go:168 > done duration=2m5.841s operation=upload_data progress=53/158 size=3.08GiB table=apm.svr_metric_data_day_local version=2.6.3
2024-12-02 20:36:06.058 INF pkg/backup/upload.go:168 > done duration=2m17.196s operation=upload_data progress=56/158 size=2.00GiB table=apm.svr_log_message_local version=2.6.3
2024-12-02 20:36:13.941 INF pkg/backup/upload.go:168 > done duration=2m27.133s operation=upload_data progress=2/158 size=10.23GiB table=apm.svr_action_data_baseline_server_application_action_local version=2.6.3
2024-12-02 20:39:36.830 INF pkg/backup/upload.go:168 > done duration=5m37.499s operation=upload_data progress=59/158 size=3.52GiB table=apm.svr_health_targets_local version=2.6.3
2024-12-02 20:41:13.074 INF pkg/backup/upload.go:168 > done duration=7m25.816s operation=upload_data progress=36/158 size=15.22GiB table=apm.sys_gauge_data_hour_local version=2.6.3
2024-12-02 20:43:14.404 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: file /1-20240825_0_18845_31/resp_time.bin: writing data: io: read/write on closed pipe
2024-12-02 20:43:14.405 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: context canceled
2024-12-02 20:43:14.405 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: context canceled
2024-12-02 20:43:14.405 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: context canceled
2024-12-02 20:43:14.405 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: context canceled
2024-12-02 20:43:14.405 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: context canceled
2024-12-02 20:43:14.405 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: context canceled
2024-12-02 20:43:14.405 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: context canceled
2024-12-02 20:43:14.405 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: context canceled
2024-12-02 20:43:14.405 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: context canceled
2024-12-02 20:43:14.405 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: context canceled
2024-12-02 20:43:14.405 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: context canceled
2024-12-02 20:43:14.405 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: context canceled
2024-12-02 20:43:14.405 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: context canceled
2024-12-02 20:43:14.406 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: context canceled
2024-12-02 20:43:14.405 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: context canceled
2024-12-02 20:43:14.405 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: context canceled
2024-12-02 20:43:14.405 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: context canceled
2024-12-02 20:43:14.405 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: context canceled
2024-12-02 20:43:14.405 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: context canceled
2024-12-02 20:43:14.405 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: context canceled
2024-12-02 20:43:14.405 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: context canceled
2024-12-02 20:43:14.405 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: context canceled
2024-12-02 20:43:14.405 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: context canceled
2024-12-02 20:43:14.405 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: context canceled
2024-12-02 20:43:14.405 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: context canceled
2024-12-02 20:43:14.405 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: context canceled
Slach commented 1 day ago

could you remove max_file_size from general section

general:
  max_file_size: 107374182400

and increase timeout to 10m

cos:
 timeout: 10m
 debug: true

and try again and share logs?

liubo-it commented 2 hours ago

could you remove max_file_size from general section

general:
  max_file_size: 107374182400

and increase timeout to 10m

cos:
 timeout: 10m
 debug: true

and try again and share logs?


PUT /apm%2F03%2Fwukong-release-apmck-iossd-505.txbj.tingyun.com%2F2024-12-04%2Fshadow%2Faspm%2Faspm_trace_risk_log_local%2Fdefault_9%252D20241103_0_266_123.tar HTTP/1.1^M
Host: wukong-apm-clickhouse-1251882848.cos.ap-beijing.myqcloud.com^M
Authorization: q-sign-algorithm=sha1&q-ak=xxx&q-sign-time=1733314541;1733318141&q-key-time=1733314541;1733318141&q-header-list=host&q-url-param-list=&q-signature=7b1f41e622448e25d5a4e74be37fd8187a944a99^M
User-Agent: cos-go-sdk-v5/0.7.55^M
^M

PUT /apm%2F03%2Fwukong-release-apmck-iossd-505.txbj.tingyun.com%2F2024-12-04%2Fshadow%2Faspm%2Faspm_trace_risk_log_local%2Fdefault_9%252D20241201_116_116_0.tar HTTP/1.1^M Host: wukong-apm-clickhouse-1251882848.cos.ap-beijing.myqcloud.com^M Authorization: q-sign-algorithm=sha1&q-ak=xxx&q-sign-time=1733314541;1733318141&q-key-time=1733314541;1733318141&q-header-list=host&q-url-param-list=&q-signature=1664c28baaa5fd22f4eada1388a26dc9b8ad91f6^M User-Agent: cos-go-sdk-v5/0.7.55^M ^M

PUT /apm%2F03%2Fwukong-release-apmck-iossd-505.txbj.tingyun.com%2F2024-12-04%2Fshadow%2Faspm%2Faspm_instance_event_log%2Fdefault_12%252D20241110_0_0_0.tar HTTP/1.1^M Host: wukong-apm-clickhouse-1251882848.cos.ap-beijing.myqcloud.com^M Authorization: q-sign-algorithm=sha1&q-ak=xxx&q-sign-time=1733314541;1733318141&q-key-time=1733314541;1733318141&q-header-list=host&q-url-param-list=&q-signature=8a97d26562b12052e844e01cfeaf60cad84eadc0^M User-Agent: cos-go-sdk-v5/0.7.55^M ^M

/UploadCompressedStream Authorization: q-sign-algorithm=sha1&q-ak=xxx&q-sign-time=1733316961;1733320561&q-key-time=1733316961;1733320561&q-header-list=host&q-url-param-list=&q-signature=3577b2c3c1d14aebfc21fbbc2d166509e2b0fd4e^M User-Agent: cos-go-sdk-v5/0.7.55^M ^M

HTTP/1.1 200 OK^M Content-Length: 20992^M Accept-Ranges: bytes^M Connection: keep-alive^M Content-Type: application/x-tar^M Date: Wed, 04 Dec 2024 12:56:01 GMT^M Etag: "972a07539bb4f13912f38d87f451bd05"^M Last-Modified: Wed, 04 Dec 2024 12:56:01 GMT^M Server: tencent-cos^M X-Cos-Hash-Crc64ecma: 5261643989400821985^M X-Cos-Request-Id: Njc1MDUxNjFfNzEwN2YxZV9iZDM4Xzc3ZWRkZDg=^M ^M

PUT /apm%2F03%2Fwukong-release-apmck-iossd-505.txbj.tingyun.com%2F2024-12-04%2Fshadow%2Fapm%2Fsvr_component_data_min_local%2Fdefault_14%252D20241124_0_24124_39.tar HTTP/1.1^M Host: wukong-apm-clickhouse-1251882848.cos.ap-beijing.myqcloud.com^M Authorization: q-sign-algorithm=sha1&q-ak=xxx&q-sign-time=1733316961;1733320561&q-key-time=1733316961;1733320561&q-header-list=host&q-url-param-list=&q-signature=eabbed16fbe847ab9a2a4a4c1ac688c906d507e0^M User-Agent: cos-go-sdk-v5/0.7.55^M ^M

2024-12-04 20:56:01.917 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: file /10-20241124_0_31339_15/component_exclusive.bin: writing data: io: read/write on closed pipe 2024-12-04 20:56:01.918 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: file /6-20241201_20622_23749_5/action_guid.bin: writing data: io: read/write on closed pipe 2024-12-04 20:56:01.918 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: file /6-20241201_0_16817_7/action_guid.bin: writing data: io: read/write on closed pipe 2024-12-04 20:56:01.918 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: file /6-20241124_0_15275_8/action_guid.bin: writing data: io: read/write on closed pipe 2024-12-04 20:56:01.918 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: file /6-20241124_15276_28242_6/action_guid.bin: writing data: io: read/write on closed pipe 2024-12-04 20:56:01.918 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: file /12-20241124_0_46582_22/component_count.bin: writing data: io: read/write on closed pipe 2024-12-04 20:56:01.918 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: file /4-20241124_0_53903_17/action_guid.bin: writing data: io: read/write on closed pipe 2024-12-04 20:56:01.918 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: file /6-20241201_16818_20621_5/action_guid.bin: writing data: io: read/write on closed pipe 2024-12-04 20:56:01.918 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: file /6-20241124_54293_56925_5/action_guid.bin: writing data: io: read/write on closed pipe 2024-12-04 20:56:01.918 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: file /6-20241124_28243_40264_6/action_guid.bin: writing data: io: read/write on closed pipe 2024-12-04 20:56:01.918 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: file /6-20241124_40265_54292_6/action_guid.bin: writing data: io: read/write on closed pipe 2024-12-04 20:56:01.918 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: file /6-20240929_0_23827_16/success_count.bin: writing data: io: read/write on closed pipe 2024-12-04 20:56:01.918 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: file /6-20241103_0_55050_15/resp_time.bin: writing data: io: read/write on closed pipe 2024-12-04 20:56:01.918 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: file /6-20241006_0_48477_19/resp_time.bin: writing data: io: read/write on closed pipe 2024-12-04 20:56:01.918 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: file /6-20241013_0_50243_17/resp_time.bin: writing data: io: read/write on closed pipe 2024-12-04 20:56:01.918 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: file /6-20241027_0_45805_14/resp_time.bin: writing data: io: read/write on closed pipe 2024-12-04 20:56:01.918 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: file /6-20241020_0_48849_14/resp_time.bin: writing data: io: read/write on closed pipe 2024-12-04 20:56:01.918 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: file /6-20240825_0_57638_16/resp_time.bin: writing data: io: read/write on closed pipe 2024-12-04 20:56:01.918 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: file /6-20240915_0_55707_15/count.bin: writing data: io: read/write on closed pipe 2024-12-04 20:56:01.918 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: file /6-20240922_28971_57292_14/resp_time_micro.bin: writing data: io: read/write on closed pipe 2024-12-04 20:56:01.918 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: file /6-20240901_0_54628_12/count.bin: writing data: io: read/write on closed pipe 2024-12-04 20:56:01.918 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: file /6-20240908_21782_53096_11/resp_time_micro.bin: writing data: io: read/write on closed pipe 2024-12-04 20:56:01.918 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: file /6-20241110_20970_60208_19/resp_time_micro.bin: writing data: io: read/write on closed pipe 2024-12-04 20:56:01.918 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: file /6-20240908_0_21781_12/success_count.bin: writing data: io: read/write on closed pipe 2024-12-04 20:56:01.918 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: file /6-20240922_0_28970_14/resp_time_micro.bin: writing data: io: read/write on closed pipe 2024-12-04 20:56:01.918 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: context canceled



It seems that there are still issues