clickhouse-backup list does not list remote backups

[resuni]

I am looking at using clickhouse-backup to back up our ClickHouse 18.16.1 cluster to S3-compatible non-Amazon storage. clickhouse-backup create_backup runs without error, and I can confirm the backup exists on our S3-compatible storage using s3cmd. However, when I run clickhouse-backup list, only the local backup appears in the list. Furthermore, the delete and restore commands will return an error when attempting to delete or restore a remote backup.

In order to troubleshoot this, I've set the log level to "debug", but nothing in the debug messages appears to indicate anything is wrong when running the create_backup command. How do I go about troubleshooting this?

My config:

general:
  remote_storage: s3
  max_file_size: 1099511627776
  disable_progress_bar: false
  backups_to_keep_local: 0
  backups_to_keep_remote: 0
  log_level: debug
  allow_empty_backups: false
clickhouse:
  username: default
  password: ""
  host: localhost
  port: 8080
  disk_mapping: {}
  skip_tables:
  - system.*
  timeout: 5m
  freeze_by_part: false
  secure: false
  skip_verify: false
  sync_replicated_tables: true
  skip_sync_replica_timeouts: true
  log_sql_queries: false
s3:
  access_key: "REDACTED"
  secret_key: "REDACTED"
  bucket: "REDACTED"
  endpoint: "REDACTED"
  region: REDACTED
  acl: private
  force_path_style: false
  path: "REDACTED"
  disable_ssl: false
  part_size: 536870912
  compression_level: 1
  compression_format: tar
  sse: ""
  disable_cert_verification: false
  storage_class: STANDARD

[Slach]

which clickhouse-backup -version do you use?

could you share results of the following command S3_FORCE_PATH_STYLE=true clickhouse-backup list remote -c your_config.yml ?

[resuni]

I downloaded the latest 1.0.0 RPM from the releases page.

Version:         1.0.0
Git Commit:      37f3bd78adec2aadc1de10f9323fe426a5e12dc4
Build Date:      2021-06-16

S3_FORCE_PATH_STYLE=true clickhouse-backup list remote -c /etc/clickhouse-backup/config.yml returns nothing. I ran create_remote again to be sure, and still nothing.

[Slach]

@resuni could you try run docker?

docker run --rm -e S3_DEBUG=true -v /etc/clickhouse-backup/config.yml:/etc/clickhouse-backup/config.yml altinity/clickhouse-backup:master create_remote test_backup

docker run --rm -e S3_DEBUG=true -v /etc/clickhouse-backup/config.yml:/etc/clickhouse-backup/config.yml altinity/clickhouse-backup:master list remote

if not, I attach binary which supports S3_DEBUG clickhouse-backup.zip

unpack it and run

S3_DEBUG=true clickhouse-backup list remote -c /etc/clickhouse-backup/config.yml

and share the results

[resuni]

I opted to use the binary in your attached zip, since I don't have Docker installed on this machine. I've redacted the Host, Authorization, and X-Amz-Content-Sha256 headers. Let me know if you think there may be something in those headers that you need.

# S3_DEBUG=true ./clickhouse-backup list remote -c /etc/clickhouse-backup/config.yml
2021/08/06 16:13:19  info DEBUG: Request s3/ListObjectsV2 Details:
---[ REQUEST POST-SIGN ]-----------------------------
GET /?delimiter=%!F(MISSING)&list-type=2&max-keys=1000&prefix=%!F(MISSING)dev%!F(MISSING) HTTP/1.1
Host: REDACTED
User-Agent: aws-sdk-go/1.37.15 (go1.16.6; linux; amd64)
Authorization: REDACTED
X-Amz-Content-Sha256: REDACTED
X-Amz-Date: 20210806T161319Z
Accept-Encoding: gzip

-----------------------------------------------------
2021/08/06 16:13:19  info DEBUG: Response s3/ListObjectsV2 Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
Content-Length: 277
Connection: keep-alive
Content-Type: application/xml
Date: Fri, 06 Aug 2021 16:13:21 GMT
Server: StorageGRID/11.4.0.2
X-Amz-Id-2: 12133881
X-Amz-Request-Id: 1628266335752929

-----------------------------------------------------

[resuni]

I figured it out. In my clickhouse-backup config.yml, I specified the path with a leading slash. For whatever reason, uploading backups worked just fine with this configuration, but reading from the S3 storage did not. Once I removed the leading slash from the path, everything started working as expected.

[Slach]

@AlexAkulov look like we can clouse issue?

[Slach]

Close issue due inactivity