sighupio / furyagent

Apache License 2.0
9 stars 2 forks source link

furyagent: pushing etcd backups to S3 fails silently #38

Closed omissis closed 2 years ago

omissis commented 2 years ago

We have received a report saying that even though the command to backup etcd to an S3 bucket runs successfully, checking the S3 bucket show the file with an older timestamp.

Command used:

furyagent backup etcd --config=/etc/fury/furyagent.yml

Logs:

{"level":"info","msg":"created temporary db file","path":"/tmp/etcd-backup.db.part"}
{"level":"info","msg":"fetching snapshot","endpoint":"https://localhost:2379/"}
{"level":"info","msg":"fetched snapshot","endpoint":"https://localhost:2379/","took":"2.437981545s"}
{"level":"info","msg":"saved","path":"/tmp/etcd-backup.db"}
 2022-09-01 18:05:24.578177 I | storage.go:239: uploading /tmp/etcd-backup.db1662048324 to etcd/k8s-master-03/snapshot.db

Checking the bucket content with the aws s3 ls command shows an older date for the file.

/etc/fury/furyagent.yml content:

storage:
  provider: s3
  url: "http://s3.eu-central-1.amazonaws.com"
  aws_access_key: "REEDACTED"
  aws_secret_key: "REDACTED"
  bucketName: "REDACTED"
  region: "eu-central-1"
clusterComponent:
  nodeName: k8s-master-03
  etcd:
    certDir: /etc/etcd/pki
    dataDir: /var/lib/etcd
    caCertFilename: ca.pem
    caKeyFilename: ca-key.pem
    clientCertFilename: etcdctl-client.pem
    clientKeyFilename: etcdctl-client-key.pem
    endpoint: https://localhost:2379
    snapshotFile: /tmp/etcd-backup.db
    backupRetention: 36h
    backupFrequency: 15m
  master:
    certDir: /etc/kubernetes/pki
    caCertFilename: ca.crt
    caKeyFilename: ca.key
    saPubFilename: sa.pub
    saKeyFilename: sa.key
    proxyCaCertFilename: front-proxy-ca.crt
    proxyKeyCertFilename: front-proxy-ca.key
    #etcdCaCertFilename: etcd-ca.crt
    etcdClientCertFilename: etcd-client.crt
    etcdClientKeyFilename: etcd-client.key
    backupRetention: 24h
    backupFrequency: 15m