ETCD-636: Add `etcd-backup-server` sidecar

Elbehery commented 2 months ago

This PR adds backup-server scheduling logic.

This is part of https://issues.redhat.com/browse/ETCD-636

cc @openshift/openshift-team-etcd

Note : This PR is built atop of https://github.com/openshift/cluster-etcd-operator/pull/1303 which is a rebased version from https://github.com/openshift/cluster-etcd-operator/pull/1298

I have tested manually using https://github.com/openshift/cluster-etcd-operator/pull/1298, and it works as expected.

The rebased version in order to use https://github.com/openshift/cluster-etcd-operator/pull/1297 which was merged while https://github.com/openshift/cluster-etcd-operator/pull/1298 still WIP

openshift-ci[bot] commented 2 months ago

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Elbehery

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files: - ~~[OWNERS](https://github.com/openshift/cluster-etcd-operator/blob/master/OWNERS)~~ [Elbehery] Approvers can indicate their approval by writing `/approve` in a comment Approvers can cancel approval by writing `/approve cancel` in a comment

openshift-ci-robot commented 2 months ago

@Elbehery: This pull request references ETCD-635 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the spike to target the "4.17.0" version, but no target version was set.

In response to [this](https://github.com/openshift/cluster-etcd-operator/pull/1301): >This PR adds backup-server scheduling logic. > >This is part of https://issues.redhat.com/browse/ETCD-636 > >cc @openshift/openshift-team-etcd Instructions for interacting with me using PR comments are available [here](https://prow.ci.openshift.org/command-help?repo=openshift%2Fcluster-etcd-operator). If you have questions or suggestions related to my behavior, please file an issue against the [openshift-eng/jira-lifecycle-plugin](https://github.com/openshift-eng/jira-lifecycle-plugin/issues/new) repository.

Elbehery commented 2 months ago

/label tide/merge-method-squash

openshift-ci-robot commented 2 months ago

@Elbehery: This pull request references ETCD-635 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the spike to target the "4.17.0" version, but no target version was set.

In response to [this](https://github.com/openshift/cluster-etcd-operator/pull/1301): >This PR adds backup-server scheduling logic. > > > >This is part of https://issues.redhat.com/browse/ETCD-636 > >cc @openshift/openshift-team-etcd > > >Note : This PR is built atop of https://github.com/openshift/cluster-etcd-operator/pull/1303 which is a rebased version from https://github.com/openshift/cluster-etcd-operator/pull/1298 > >I have tested manually using https://github.com/openshift/cluster-etcd-operator/pull/1298, and it works as expected. > >The rebased version in order to use https://github.com/openshift/cluster-etcd-operator/pull/1297 which was merged while >https://github.com/openshift/cluster-etcd-operator/pull/1298 still WIP > > Instructions for interacting with me using PR comments are available [here](https://prow.ci.openshift.org/command-help?repo=openshift%2Fcluster-etcd-operator). If you have questions or suggestions related to my behavior, please file an issue against the [openshift-eng/jira-lifecycle-plugin](https://github.com/openshift-eng/jira-lifecycle-plugin/issues/new) repository.

Elbehery commented 2 months ago

/hold

Elbehery commented 2 months ago

/retest-required

Elbehery commented 2 months ago

/retest-required

Elbehery commented 2 months ago

Tested with a 4.17 cluster and this PR atop of it

used the following CR

apiVersion: config.openshift.io/v1alpha1
kind: Backup
metadata:
  name: default
  annotations:
    default: "true"
spec:
  etcd:
    schedule: "*/5 * * * *"
    timeZone: "UTC"

master node 1 pod/etcd-ip-10-0-64-151.us-east-2.compute.internal

sh-5.1# ls -l /var/backup/etcd/current-backup/
total 107764
-rw-------. 1 root root 110268448 Jul 28 21:20 snapshot_2024-07-28_212000.db
-rw-------. 1 root root     76822 Jul 28 21:20 static_kuberesources_2024-07-28_212000.tar.gz
sh-5.1#

master node 2 pod/etcd-ip-10-0-59-231.us-east-2.compute.internal

sh-5.1# ls -l /var/backup/etcd/current-backup/
total 105240
-rw-------. 1 root root 107683872 Jul 28 21:25 snapshot_2024-07-28_212500.db
-rw-------. 1 root root     76799 Jul 28 21:25 static_kuberesources_2024-07-28_212500.tar.gz
sh-5.1#

master node 3 pod/etcd-ip-10-0-97-178.us-east-2.compute.internal

sh-5.1# ls -l /var/backup/etcd/current-backup/
total 108828
-rw-------. 1 root root 111357984 Jul 28 21:25 snapshot_2024-07-28_212500.db
-rw-------. 1 root root     76803 Jul 28 21:25 static_kuberesources_2024-07-28_212500.tar.gz
sh-5.1#

Elbehery commented 2 months ago

logs from etcd-backup-server container

melbeher@melbeher-mac Downloads % oc logs -f pod/etcd-ip-10-0-64-151.us-east-2.compute.internal -n openshift-etcd -c etcd-backup-server 
I0728 20:54:23.644800       1 backupserver.go:42] hello from backup server :) 
I0728 20:54:23.647960       1 backupserver.go:82] hello from backup server Run() :) 
I0728 20:54:23.648011       1 backupserver.go:88] hello from backup server - scheduler has been init ;)  
I0728 20:54:23.648170       1 backupserver.go:110] hello from backup server -before schedule backup()  
I0728 20:54:23.648180       1 backupserver.go:133] hello from backup server -inside-begin schedule backup()  
I0728 20:54:23.648224       1 backupserver.go:139] hello from backup server -inside-end schedule backup()  
E0728 20:54:23.648234       1 backupserver.go:140] hello from backup server -inside-end schedule backup()  error [<nil>]
I0728 20:54:23.648238       1 backupserver.go:115] hello from backup server -after schedule backup()  
2024/07/28 20:54:23 [tasker] next tick on 2024/07/28 20:55:00
2024/07/28 20:55:00 [tasker] running 1 due tasks
2024/07/28 20:55:00 [tasker] next tick on 2024/07/28 20:56:00
2024/07/28 20:55:00 [tasker] task [*/5 * * * *][#1] running
I0728 20:55:00.692796       1 backuputils.go:41] hello from backup server -inside backupOptions backup()  
I0728 20:55:00.698008       1 backuputils.go:49] hello from backup server -inside backupOptions backup() - before checkAndCreateDir() 
I0728 20:55:00.698437       1 backuputils.go:54] hello from backup server -inside backupOptions backup() - after checkAndCreateDir() 
I0728 20:55:00.698493       1 backuputils.go:61] hello from backup server -inside backupOptions backup() - before saveSnapshot() 
{"level":"info","ts":"2024-07-28T20:55:00.698889Z","logger":"etcd-client","caller":"v3@v3.5.10/maintenance.go:212","msg":"opened snapshot stream; downloading"}
{"level":"info","ts":"2024-07-28T20:55:01.226809Z","logger":"etcd-client","caller":"v3@v3.5.10/maintenance.go:220","msg":"completed snapshot read; closing"}
I0728 20:55:01.226952       1 etcdclientutils.go:70] fetched snapshot, took: 528.339787ms
I0728 20:55:01.227021       1 etcdclientutils.go:75] saved snapshot to path /var/backup/etcd/current-backup/snapshot_2024-07-28_205500.db
I0728 20:55:01.227041       1 backuputils.go:67] hello from backup server -inside backupOptions backup() - after saveSnapshot() 
I0728 20:55:01.227045       1 backuputils.go:70] hello from backup server -inside backupOptions backup() - before archiveLatestResources() 
I0728 20:55:01.227051       1 backuputils.go:21] Static Pod Resources are being stored in: /var/backup/etcd/current-backup/static_kuberesources_2024-07-28_205500.tar.gz
I0728 20:55:01.227259       1 backuputils.go:30]    Adding the latest revision for podName kube-apiserver-pod: /etc/kubernetes/static-pod-resources/kube-apiserver-pod-9
I0728 20:55:01.227377       1 backuputils.go:30]    Adding the latest revision for podName kube-controller-manager-pod: /etc/kubernetes/static-pod-resources/kube-controller-manager-pod-8
I0728 20:55:01.227461       1 backuputils.go:30]    Adding the latest revision for podName kube-scheduler-pod: /etc/kubernetes/static-pod-resources/kube-scheduler-pod-7
I0728 20:55:01.227566       1 backuputils.go:30]    Adding the latest revision for podName etcd-pod: /etc/kubernetes/static-pod-resources/etcd-pod-9
I0728 20:55:01.256715       1 backuputils.go:75] hello from backup server -inside backupOptions backup() - after archiveLatestResources() 
2024/07/28 20:55:01 [tasker] task [*/5 * * * *][#1] ran successfully
2024/07/28 20:56:00 [tasker] next tick on 2024/07/28 20:57:00
2024/07/28 20:57:00 [tasker] next tick on 2024/07/28 20:58:00
2024/07/28 20:58:00 [tasker] next tick on 2024/07/28 20:59:00
2024/07/28 20:59:00 [tasker] next tick on 2024/07/28 21:00:00
2024/07/28 21:00:00 [tasker] running 1 due tasks
2024/07/28 21:00:00 [tasker] next tick on 2024/07/28 21:01:00
2024/07/28 21:00:00 [tasker] task [*/5 * * * *][#1] running
I0728 21:00:00.909098       1 backuputils.go:41] hello from backup server -inside backupOptions backup()  
I0728 21:00:00.914293       1 backuputils.go:49] hello from backup server -inside backupOptions backup() - before checkAndCreateDir() 
I0728 21:00:00.928238       1 backuputils.go:54] hello from backup server -inside backupOptions backup() - after checkAndCreateDir() 
I0728 21:00:00.928337       1 backuputils.go:61] hello from backup server -inside backupOptions backup() - before saveSnapshot() 
{"level":"info","ts":"2024-07-28T21:00:00.928499Z","logger":"etcd-client","caller":"v3@v3.5.10/maintenance.go:212","msg":"opened snapshot stream; downloading"}
{"level":"info","ts":"2024-07-28T21:00:01.490485Z","logger":"etcd-client","caller":"v3@v3.5.10/maintenance.go:220","msg":"completed snapshot read; closing"}
I0728 21:00:01.490632       1 etcdclientutils.go:70] fetched snapshot, took: 562.196236ms
I0728 21:00:01.490744       1 etcdclientutils.go:75] saved snapshot to path /var/backup/etcd/current-backup/snapshot_2024-07-28_210000.db
I0728 21:00:01.490803       1 backuputils.go:67] hello from backup server -inside backupOptions backup() - after saveSnapshot() 
I0728 21:00:01.490838       1 backuputils.go:70] hello from backup server -inside backupOptions backup() - before archiveLatestResources() 
I0728 21:00:01.490870       1 backuputils.go:21] Static Pod Resources are being stored in: /var/backup/etcd/current-backup/static_kuberesources_2024-07-28_210000.tar.gz
I0728 21:00:01.491042       1 backuputils.go:30]    Adding the latest revision for podName kube-apiserver-pod: /etc/kubernetes/static-pod-resources/kube-apiserver-pod-9
I0728 21:00:01.491176       1 backuputils.go:30]    Adding the latest revision for podName kube-controller-manager-pod: /etc/kubernetes/static-pod-resources/kube-controller-manager-pod-8
I0728 21:00:01.491308       1 backuputils.go:30]    Adding the latest revision for podName kube-scheduler-pod: /etc/kubernetes/static-pod-resources/kube-scheduler-pod-7
I0728 21:00:01.491477       1 backuputils.go:30]    Adding the latest revision for podName etcd-pod: /etc/kubernetes/static-pod-resources/etcd-pod-9
I0728 21:00:01.501215       1 backuputils.go:75] hello from backup server -inside backupOptions backup() - after archiveLatestResources() 
2024/07/28 21:00:01 [tasker] task [*/5 * * * *][#1] ran successfully
2024/07/28 21:01:00 [tasker] next tick on 2024/07/28 21:02:00
2024/07/28 21:02:00 [tasker] next tick on 2024/07/28 21:03:00
2024/07/28 21:03:01 [tasker] next tick on 2024/07/28 21:04:00
2024/07/28 21:04:00 [tasker] next tick on 2024/07/28 21:05:00
2024/07/28 21:05:00 [tasker] running 1 due tasks
2024/07/28 21:05:00 [tasker] next tick on 2024/07/28 21:06:00
2024/07/28 21:05:00 [tasker] task [*/5 * * * *][#1] running
I0728 21:05:00.122797       1 backuputils.go:41] hello from backup server -inside backupOptions backup()  
I0728 21:05:00.127720       1 backuputils.go:49] hello from backup server -inside backupOptions backup() - before checkAndCreateDir() 
I0728 21:05:00.138848       1 backuputils.go:54] hello from backup server -inside backupOptions backup() - after checkAndCreateDir() 
I0728 21:05:00.138916       1 backuputils.go:61] hello from backup server -inside backupOptions backup() - before saveSnapshot() 
{"level":"info","ts":"2024-07-28T21:05:00.139058Z","logger":"etcd-client","caller":"v3@v3.5.10/maintenance.go:212","msg":"opened snapshot stream; downloading"}
{"level":"info","ts":"2024-07-28T21:05:00.6419Z","logger":"etcd-client","caller":"v3@v3.5.10/maintenance.go:220","msg":"completed snapshot read; closing"}
I0728 21:05:00.641965       1 etcdclientutils.go:70] fetched snapshot, took: 502.961514ms
I0728 21:05:00.642016       1 etcdclientutils.go:75] saved snapshot to path /var/backup/etcd/current-backup/snapshot_2024-07-28_210500.db
I0728 21:05:00.642036       1 backuputils.go:67] hello from backup server -inside backupOptions backup() - after saveSnapshot() 
I0728 21:05:00.642041       1 backuputils.go:70] hello from backup server -inside backupOptions backup() - before archiveLatestResources() 
I0728 21:05:00.642046       1 backuputils.go:21] Static Pod Resources are being stored in: /var/backup/etcd/current-backup/static_kuberesources_2024-07-28_210500.tar.gz
I0728 21:05:00.642160       1 backuputils.go:30]    Adding the latest revision for podName kube-apiserver-pod: /etc/kubernetes/static-pod-resources/kube-apiserver-pod-9
I0728 21:05:00.642233       1 backuputils.go:30]    Adding the latest revision for podName kube-controller-manager-pod: /etc/kubernetes/static-pod-resources/kube-controller-manager-pod-8
I0728 21:05:00.642299       1 backuputils.go:30]    Adding the latest revision for podName kube-scheduler-pod: /etc/kubernetes/static-pod-resources/kube-scheduler-pod-7
I0728 21:05:00.642395       1 backuputils.go:30]    Adding the latest revision for podName etcd-pod: /etc/kubernetes/static-pod-resources/etcd-pod-9
I0728 21:05:00.651788       1 backuputils.go:75] hello from backup server -inside backupOptions backup() - after archiveLatestResources() 
2024/07/28 21:05:00 [tasker] task [*/5 * * * *][#1] ran successfully
2024/07/28 21:06:00 [tasker] next tick on 2024/07/28 21:07:00
2024/07/28 21:07:00 [tasker] next tick on 2024/07/28 21:08:00
2024/07/28 21:08:00 [tasker] next tick on 2024/07/28 21:09:00
2024/07/28 21:09:00 [tasker] next tick on 2024/07/28 21:10:00
2024/07/28 21:10:00 [tasker] running 1 due tasks
2024/07/28 21:10:00 [tasker] next tick on 2024/07/28 21:11:00
2024/07/28 21:10:00 [tasker] task [*/5 * * * *][#1] running
I0728 21:10:00.278670       1 backuputils.go:41] hello from backup server -inside backupOptions backup()  
I0728 21:10:00.283732       1 backuputils.go:49] hello from backup server -inside backupOptions backup() - before checkAndCreateDir() 
I0728 21:10:00.294635       1 backuputils.go:54] hello from backup server -inside backupOptions backup() - after checkAndCreateDir() 
I0728 21:10:00.294653       1 backuputils.go:61] hello from backup server -inside backupOptions backup() - before saveSnapshot() 
{"level":"info","ts":"2024-07-28T21:10:00.294766Z","logger":"etcd-client","caller":"v3@v3.5.10/maintenance.go:212","msg":"opened snapshot stream; downloading"}
{"level":"info","ts":"2024-07-28T21:10:00.790883Z","logger":"etcd-client","caller":"v3@v3.5.10/maintenance.go:220","msg":"completed snapshot read; closing"}
I0728 21:10:00.790988       1 etcdclientutils.go:70] fetched snapshot, took: 496.274432ms
I0728 21:10:00.791062       1 etcdclientutils.go:75] saved snapshot to path /var/backup/etcd/current-backup/snapshot_2024-07-28_211000.db
I0728 21:10:00.791091       1 backuputils.go:67] hello from backup server -inside backupOptions backup() - after saveSnapshot() 
I0728 21:10:00.791099       1 backuputils.go:70] hello from backup server -inside backupOptions backup() - before archiveLatestResources() 
I0728 21:10:00.791107       1 backuputils.go:21] Static Pod Resources are being stored in: /var/backup/etcd/current-backup/static_kuberesources_2024-07-28_211000.tar.gz
I0728 21:10:00.791226       1 backuputils.go:30]    Adding the latest revision for podName kube-apiserver-pod: /etc/kubernetes/static-pod-resources/kube-apiserver-pod-9
I0728 21:10:00.791362       1 backuputils.go:30]    Adding the latest revision for podName kube-controller-manager-pod: /etc/kubernetes/static-pod-resources/kube-controller-manager-pod-8
I0728 21:10:00.791475       1 backuputils.go:30]    Adding the latest revision for podName kube-scheduler-pod: /etc/kubernetes/static-pod-resources/kube-scheduler-pod-7
I0728 21:10:00.791575       1 backuputils.go:30]    Adding the latest revision for podName etcd-pod: /etc/kubernetes/static-pod-resources/etcd-pod-9
I0728 21:10:00.799634       1 backuputils.go:75] hello from backup server -inside backupOptions backup() - after archiveLatestResources() 
2024/07/28 21:10:00 [tasker] task [*/5 * * * *][#1] ran successfully
2024/07/28 21:11:00 [tasker] next tick on 2024/07/28 21:12:00
2024/07/28 21:12:00 [tasker] next tick on 2024/07/28 21:13:00
2024/07/28 21:13:00 [tasker] next tick on 2024/07/28 21:14:00
2024/07/28 21:14:00 [tasker] next tick on 2024/07/28 21:15:00
2024/07/28 21:15:00 [tasker] running 1 due tasks
2024/07/28 21:15:00 [tasker] next tick on 2024/07/28 21:16:00
2024/07/28 21:15:00 [tasker] task [*/5 * * * *][#1] running
I0728 21:15:00.380629       1 backuputils.go:41] hello from backup server -inside backupOptions backup()  
I0728 21:15:00.385770       1 backuputils.go:49] hello from backup server -inside backupOptions backup() - before checkAndCreateDir() 
I0728 21:15:00.397106       1 backuputils.go:54] hello from backup server -inside backupOptions backup() - after checkAndCreateDir() 
I0728 21:15:00.397121       1 backuputils.go:61] hello from backup server -inside backupOptions backup() - before saveSnapshot() 
{"level":"info","ts":"2024-07-28T21:15:00.397268Z","logger":"etcd-client","caller":"v3@v3.5.10/maintenance.go:212","msg":"opened snapshot stream; downloading"}
{"level":"info","ts":"2024-07-28T21:15:00.880988Z","logger":"etcd-client","caller":"v3@v3.5.10/maintenance.go:220","msg":"completed snapshot read; closing"}
I0728 21:15:00.881061       1 etcdclientutils.go:70] fetched snapshot, took: 483.880474ms
I0728 21:15:00.881125       1 etcdclientutils.go:75] saved snapshot to path /var/backup/etcd/current-backup/snapshot_2024-07-28_211500.db
I0728 21:15:00.881147       1 backuputils.go:67] hello from backup server -inside backupOptions backup() - after saveSnapshot() 
I0728 21:15:00.881152       1 backuputils.go:70] hello from backup server -inside backupOptions backup() - before archiveLatestResources() 
I0728 21:15:00.881158       1 backuputils.go:21] Static Pod Resources are being stored in: /var/backup/etcd/current-backup/static_kuberesources_2024-07-28_211500.tar.gz
I0728 21:15:00.881276       1 backuputils.go:30]    Adding the latest revision for podName kube-apiserver-pod: /etc/kubernetes/static-pod-resources/kube-apiserver-pod-9
I0728 21:15:00.881392       1 backuputils.go:30]    Adding the latest revision for podName kube-controller-manager-pod: /etc/kubernetes/static-pod-resources/kube-controller-manager-pod-8
I0728 21:15:00.881479       1 backuputils.go:30]    Adding the latest revision for podName kube-scheduler-pod: /etc/kubernetes/static-pod-resources/kube-scheduler-pod-7
I0728 21:15:00.881560       1 backuputils.go:30]    Adding the latest revision for podName etcd-pod: /etc/kubernetes/static-pod-resources/etcd-pod-9
I0728 21:15:00.891125       1 backuputils.go:75] hello from backup server -inside backupOptions backup() - after archiveLatestResources() 
2024/07/28 21:15:00 [tasker] task [*/5 * * * *][#1] ran successfully
2024/07/28 21:16:00 [tasker] next tick on 2024/07/28 21:17:00
2024/07/28 21:17:00 [tasker] next tick on 2024/07/28 21:18:00
2024/07/28 21:18:00 [tasker] next tick on 2024/07/28 21:19:00
2024/07/28 21:19:00 [tasker] next tick on 2024/07/28 21:20:00
2024/07/28 21:20:00 [tasker] running 1 due tasks
2024/07/28 21:20:00 [tasker] next tick on 2024/07/28 21:21:00
2024/07/28 21:20:00 [tasker] task [*/5 * * * *][#1] running
I0728 21:20:00.484513       1 backuputils.go:41] hello from backup server -inside backupOptions backup()  
I0728 21:20:00.490397       1 backuputils.go:49] hello from backup server -inside backupOptions backup() - before checkAndCreateDir() 
I0728 21:20:00.501854       1 backuputils.go:54] hello from backup server -inside backupOptions backup() - after checkAndCreateDir() 
I0728 21:20:00.501872       1 backuputils.go:61] hello from backup server -inside backupOptions backup() - before saveSnapshot() 
{"level":"info","ts":"2024-07-28T21:20:00.501985Z","logger":"etcd-client","caller":"v3@v3.5.10/maintenance.go:212","msg":"opened snapshot stream; downloading"}
{"level":"info","ts":"2024-07-28T21:20:01.006747Z","logger":"etcd-client","caller":"v3@v3.5.10/maintenance.go:220","msg":"completed snapshot read; closing"}
I0728 21:20:01.006846       1 etcdclientutils.go:70] fetched snapshot, took: 504.916818ms
I0728 21:20:01.006916       1 etcdclientutils.go:75] saved snapshot to path /var/backup/etcd/current-backup/snapshot_2024-07-28_212000.db
I0728 21:20:01.006942       1 backuputils.go:67] hello from backup server -inside backupOptions backup() - after saveSnapshot() 
I0728 21:20:01.006949       1 backuputils.go:70] hello from backup server -inside backupOptions backup() - before archiveLatestResources() 
I0728 21:20:01.006957       1 backuputils.go:21] Static Pod Resources are being stored in: /var/backup/etcd/current-backup/static_kuberesources_2024-07-28_212000.tar.gz
I0728 21:20:01.007133       1 backuputils.go:30]    Adding the latest revision for podName kube-apiserver-pod: /etc/kubernetes/static-pod-resources/kube-apiserver-pod-9
I0728 21:20:01.007246       1 backuputils.go:30]    Adding the latest revision for podName kube-controller-manager-pod: /etc/kubernetes/static-pod-resources/kube-controller-manager-pod-8
I0728 21:20:01.007407       1 backuputils.go:30]    Adding the latest revision for podName kube-scheduler-pod: /etc/kubernetes/static-pod-resources/kube-scheduler-pod-7
I0728 21:20:01.007532       1 backuputils.go:30]    Adding the latest revision for podName etcd-pod: /etc/kubernetes/static-pod-resources/etcd-pod-9
I0728 21:20:01.015822       1 backuputils.go:75] hello from backup server -inside backupOptions backup() - after archiveLatestResources() 
2024/07/28 21:20:01 [tasker] task [*/5 * * * *][#1] ran successfully
2024/07/28 21:21:00 [tasker] next tick on 2024/07/28 21:22:00
2024/07/28 21:22:00 [tasker] next tick on 2024/07/28 21:23:00
2024/07/28 21:23:00 [tasker] next tick on 2024/07/28 21:24:00
2024/07/28 21:24:00 [tasker] next tick on 2024/07/28 21:25:00
2024/07/28 21:25:00 [tasker] running 1 due tasks
2024/07/28 21:25:00 [tasker] next tick on 2024/07/28 21:26:00
2024/07/28 21:25:00 [tasker] task [*/5 * * * *][#1] running
I0728 21:25:00.594254       1 backuputils.go:41] hello from backup server -inside backupOptions backup()  
I0728 21:25:00.599360       1 backuputils.go:49] hello from backup server -inside backupOptions backup() - before checkAndCreateDir() 
I0728 21:25:00.610523       1 backuputils.go:54] hello from backup server -inside backupOptions backup() - after checkAndCreateDir() 
I0728 21:25:00.610538       1 backuputils.go:61] hello from backup server -inside backupOptions backup() - before saveSnapshot() 
{"level":"info","ts":"2024-07-28T21:25:00.610648Z","logger":"etcd-client","caller":"v3@v3.5.10/maintenance.go:212","msg":"opened snapshot stream; downloading"}
{"level":"info","ts":"2024-07-28T21:25:01.128775Z","logger":"etcd-client","caller":"v3@v3.5.10/maintenance.go:220","msg":"completed snapshot read; closing"}
I0728 21:25:01.128854       1 etcdclientutils.go:70] fetched snapshot, took: 518.259971ms
I0728 21:25:01.128928       1 etcdclientutils.go:75] saved snapshot to path /var/backup/etcd/current-backup/snapshot_2024-07-28_212500.db
I0728 21:25:01.128976       1 backuputils.go:67] hello from backup server -inside backupOptions backup() - after saveSnapshot() 
I0728 21:25:01.128984       1 backuputils.go:70] hello from backup server -inside backupOptions backup() - before archiveLatestResources() 
I0728 21:25:01.128991       1 backuputils.go:21] Static Pod Resources are being stored in: /var/backup/etcd/current-backup/static_kuberesources_2024-07-28_212500.tar.gz
I0728 21:25:01.129141       1 backuputils.go:30]    Adding the latest revision for podName kube-apiserver-pod: /etc/kubernetes/static-pod-resources/kube-apiserver-pod-9
I0728 21:25:01.129270       1 backuputils.go:30]    Adding the latest revision for podName kube-controller-manager-pod: /etc/kubernetes/static-pod-resources/kube-controller-manager-pod-8
I0728 21:25:01.129418       1 backuputils.go:30]    Adding the latest revision for podName kube-scheduler-pod: /etc/kubernetes/static-pod-resources/kube-scheduler-pod-7
I0728 21:25:01.129545       1 backuputils.go:30]    Adding the latest revision for podName etcd-pod: /etc/kubernetes/static-pod-resources/etcd-pod-9
I0728 21:25:01.138756       1 backuputils.go:75] hello from backup server -inside backupOptions backup() - after archiveLatestResources() 
2024/07/28 21:25:01 [tasker] task [*/5 * * * *][#1] ran successfully
2024/07/28 21:26:00 [tasker] next tick on 2024/07/28 21:27:00
2024/07/28 21:27:00 [tasker] next tick on 2024/07/28 21:28:00

Elbehery commented 2 months ago

e2e-aws-ovn-etcd-scaling failures are not related to this PR

/override e2e-aws-ovn-etcd-scaling

openshift-ci[bot] commented 2 months ago

@Elbehery: /override requires failed status contexts, check run or a prowjob name to operate on. The following unknown contexts/checkruns were given:

e2e-aws-ovn-etcd-scaling

Only the following failed contexts/checkruns were expected:

ci/prow/e2e-agnostic-ovn
ci/prow/e2e-agnostic-ovn-upgrade
ci/prow/e2e-aws-etcd-certrotation
ci/prow/e2e-aws-etcd-recovery
ci/prow/e2e-aws-ovn-etcd-scaling
ci/prow/e2e-aws-ovn-serial
ci/prow/e2e-aws-ovn-single-node
ci/prow/e2e-metal-ovn-ha-cert-rotation-shutdown
ci/prow/e2e-metal-ovn-sno-cert-rotation-shutdown
ci/prow/e2e-operator
ci/prow/e2e-operator-fips
ci/prow/images
ci/prow/unit
ci/prow/verify
ci/prow/verify-deps
pull-ci-openshift-cluster-etcd-operator-master-e2e-agnostic-ovn
pull-ci-openshift-cluster-etcd-operator-master-e2e-agnostic-ovn-upgrade
pull-ci-openshift-cluster-etcd-operator-master-e2e-aws-etcd-certrotation
pull-ci-openshift-cluster-etcd-operator-master-e2e-aws-etcd-recovery
pull-ci-openshift-cluster-etcd-operator-master-e2e-aws-ovn-etcd-scaling
pull-ci-openshift-cluster-etcd-operator-master-e2e-aws-ovn-serial
pull-ci-openshift-cluster-etcd-operator-master-e2e-aws-ovn-single-node
pull-ci-openshift-cluster-etcd-operator-master-e2e-metal-ovn-ha-cert-rotation-shutdown
pull-ci-openshift-cluster-etcd-operator-master-e2e-metal-ovn-sno-cert-rotation-shutdown
pull-ci-openshift-cluster-etcd-operator-master-e2e-operator
pull-ci-openshift-cluster-etcd-operator-master-e2e-operator-fips
pull-ci-openshift-cluster-etcd-operator-master-images
pull-ci-openshift-cluster-etcd-operator-master-unit
pull-ci-openshift-cluster-etcd-operator-master-verify
pull-ci-openshift-cluster-etcd-operator-master-verify-deps
tide

If you are trying to override a checkrun that has a space in it, you must put a double quote on the context.

In response to [this](https://github.com/openshift/cluster-etcd-operator/pull/1301#issuecomment-2254796049): >`e2e-aws-ovn-etcd-scaling` failures are not related to this PR > >/override e2e-aws-ovn-etcd-scaling Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.

Elbehery commented 2 months ago

/override ci/prow/e2e-aws-ovn-etcd-scaling

openshift-ci[bot] commented 2 months ago

@Elbehery: Overrode contexts on behalf of Elbehery: ci/prow/e2e-aws-ovn-etcd-scaling

In response to [this](https://github.com/openshift/cluster-etcd-operator/pull/1301#issuecomment-2254796830): >/override ci/prow/e2e-aws-ovn-etcd-scaling Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.

openshift-ci[bot] commented 2 months ago

@Elbehery: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-metal-ovn-sno-cert-rotation-shutdown	4fc5bdc16335b8d849ab0800b6c214203b5031c6	link	false	`/test e2e-metal-ovn-sno-cert-rotation-shutdown`
ci/prow/e2e-metal-ovn-ha-cert-rotation-shutdown	4fc5bdc16335b8d849ab0800b6c214203b5031c6	link	false	`/test e2e-metal-ovn-ha-cert-rotation-shutdown`
ci/prow/e2e-aws-etcd-recovery	4fc5bdc16335b8d849ab0800b6c214203b5031c6	link	false	`/test e2e-aws-etcd-recovery`
ci/prow/e2e-aws-etcd-certrotation	4fc5bdc16335b8d849ab0800b6c214203b5031c6	link	false	`/test e2e-aws-etcd-certrotation`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).

Elbehery commented 2 months ago

/assign @hasbro17

Elbehery commented 1 month ago

/hold cancel

Elbehery commented 1 month ago

/hold

Elbehery commented 1 month ago

this PR has been split into

openshift-merge-robot commented 1 month ago

PR needs rebase.

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.

openshift-ci-robot commented 1 month ago

@Elbehery: This pull request references ETCD-636 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.18.0" version, but no target version was set.

In response to [this](https://github.com/openshift/cluster-etcd-operator/pull/1301): >This PR adds backup-server scheduling logic. > > > >This is part of https://issues.redhat.com/browse/ETCD-636 > >cc @openshift/openshift-team-etcd > > >Note : This PR is built atop of https://github.com/openshift/cluster-etcd-operator/pull/1303 which is a rebased version from https://github.com/openshift/cluster-etcd-operator/pull/1298 > >I have tested manually using https://github.com/openshift/cluster-etcd-operator/pull/1298, and it works as expected. > >The rebased version in order to use https://github.com/openshift/cluster-etcd-operator/pull/1297 which was merged while >https://github.com/openshift/cluster-etcd-operator/pull/1298 still WIP > > Instructions for interacting with me using PR comments are available [here](https://prow.ci.openshift.org/command-help?repo=openshift%2Fcluster-etcd-operator). If you have questions or suggestions related to my behavior, please file an issue against the [openshift-eng/jira-lifecycle-plugin](https://github.com/openshift-eng/jira-lifecycle-plugin/issues/new) repository.

Elbehery commented 2 weeks ago

closing this in favor of

/close

Elbehery commented 2 weeks ago

closing this in favor of

/close

Elbehery commented 2 weeks ago

closing this in favor of

https://github.com/openshift/cluster-etcd-operator/pull/1304 https://github.com/openshift/cluster-etcd-operator/pull/1305 https://github.com/openshift/cluster-etcd-operator/pull/1306 /close

openshift-ci[bot] commented 2 weeks ago

@Elbehery: Closed this PR.

In response to [this](https://github.com/openshift/cluster-etcd-operator/pull/1301#issuecomment-2339984280): >closing this in favor of > >- https://github.com/openshift/cluster-etcd-operator/pull/1304 >- https://github.com/openshift/cluster-etcd-operator/pull/1305 >- https://github.com/openshift/cluster-etcd-operator/pull/1306 > >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.

openshift-ci[bot] commented 2 weeks ago

@Elbehery: Closed this PR.

In response to [this](https://github.com/openshift/cluster-etcd-operator/pull/1301#issuecomment-2339984593): >closing this in favor of > >- https://github.com/openshift/cluster-etcd-operator/pull/1304 >- https://github.com/openshift/cluster-etcd-operator/pull/1305 >- https://github.com/openshift/cluster-etcd-operator/pull/1306 > >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.

openshift / cluster-etcd-operator

ETCD-636: Add `etcd-backup-server` sidecar #1301