search

RamenDR / ramen

Apache License 2.0
72 stars 52 forks source link

e2e: add subscription workload #1357

Closed par97 closed 3 months ago

par97 commented 3 months ago

This PR is to enable subscription application with busybox deployment for e2e test.

keslerzhu commented 3 months ago

I have verified this PR works in my test env:

(ramen) fusion@s194:~/keslerzhu/ramen/e2e$ go test -kubeconfig-hub ./kubeconfig/hub/kubeconfig -kubeconfig-c1 ./kubeconfig/c1/kubeconfig -kubeconfig-c2 ./kubeconfig/c2/kubeconfig
2024-05-06T15:28:41.441+0800    INFO    e2e/main_test.go:57 TestSuites
2024-05-06T15:28:41.441+0800    INFO    deployers/subscription.go:30    enter Deploy Deployment/Subscription
2024-05-06T15:28:41.538+0800    INFO    dractions/actions.go:28 enter EnableProtection Deployment/Subscription
2024-05-06T15:28:41.541+0800    INFO    dractions/retry.go:40   could not get placement decision, retry in 30 seconds
2024-05-06T15:29:11.546+0800    INFO    dractions/retry.go:29   got placementdecision name subscription-busybox-decision-1
2024-05-06T15:29:11.546+0800    INFO    dractions/actions.go:55 get placementdecision subscription-busybox-decision-1
2024-05-06T15:29:11.549+0800    INFO    dractions/actions.go:63 placementdecision clusterName: rdr-dr1
2024-05-06T15:29:11.549+0800    INFO    dractions/actions.go:74 update placement subscription-busybox annotation
2024-05-06T15:29:11.558+0800    INFO    dractions/actions.go:80 create drpc subscription-busybox
2024-05-06T15:29:11.585+0800    INFO    dractions/retry.go:69   drpc  subscription-busybox is not ready yet, retry in 30 seconds
2024-05-06T15:29:41.613+0800    INFO    dractions/retry.go:62   drpc subscription-busybox LastGroupSyncTime is nil
2024-05-06T15:29:41.613+0800    INFO    dractions/retry.go:69   drpc  subscription-busybox is not ready yet, retry in 30 seconds
2024-05-06T15:30:11.645+0800    INFO    dractions/retry.go:62   drpc subscription-busybox LastGroupSyncTime is nil
2024-05-06T15:30:11.645+0800    INFO    dractions/retry.go:69   drpc  subscription-busybox is not ready yet, retry in 30 seconds
2024-05-06T15:30:41.663+0800    INFO    dractions/retry.go:56   drpc subscription-busybox is ready
2024-05-06T15:30:41.663+0800    INFO    dractions/actions.go:132    enter DRActions Failover
2024-05-06T15:30:41.667+0800    INFO    dractions/retry.go:56   drpc subscription-busybox is ready
2024-05-06T15:30:41.667+0800    INFO    dractions/actions.go:151    get drpc subscription-busybox
2024-05-06T15:30:41.670+0800    INFO    dractions/actions.go:158    get drpolicy dr-policy
2024-05-06T15:30:41.675+0800    INFO    dractions/retry.go:29   got placementdecision name subscription-busybox-decision-1
2024-05-06T15:30:41.676+0800    INFO    dractions/retry.go:136  get placementdecision subscription-busybox-decision-1
2024-05-06T15:30:41.678+0800    INFO    dractions/retry.go:144  placementdecision clusterName: rdr-dr1
2024-05-06T15:30:41.678+0800    INFO    dractions/actions.go:170    failover to cluster: rdr-dr2
2024-05-06T15:30:41.678+0800    INFO    dractions/actions.go:175    update drpc subscription-busybox
2024-05-06T15:30:46.700+0800    INFO    dractions/retry.go:124  current drpc  subscription-busybox phase is FailingOver, expecting FailedOver, retry in 30 seconds
2024-05-06T15:31:16.704+0800    INFO    dractions/retry.go:114  drpc  subscription-busybox phase is FailedOver
2024-05-06T15:31:16.707+0800    INFO    dractions/retry.go:91   drpc subscription-busybox condition PeerReady is not True
2024-05-06T15:31:16.707+0800    INFO    dractions/retry.go:69   drpc  subscription-busybox is not ready yet, retry in 30 seconds
2024-05-06T15:31:46.733+0800    INFO    dractions/retry.go:91   drpc subscription-busybox condition PeerReady is not True
2024-05-06T15:31:46.733+0800    INFO    dractions/retry.go:69   drpc  subscription-busybox is not ready yet, retry in 30 seconds
2024-05-06T15:32:16.740+0800    INFO    dractions/retry.go:91   drpc subscription-busybox condition PeerReady is not True
2024-05-06T15:32:16.740+0800    INFO    dractions/retry.go:69   drpc  subscription-busybox is not ready yet, retry in 30 seconds
2024-05-06T15:32:46.770+0800    INFO    dractions/retry.go:91   drpc subscription-busybox condition PeerReady is not True
2024-05-06T15:32:46.770+0800    INFO    dractions/retry.go:69   drpc  subscription-busybox is not ready yet, retry in 30 seconds
2024-05-06T15:33:16.776+0800    INFO    dractions/retry.go:91   drpc subscription-busybox condition PeerReady is not True
2024-05-06T15:33:16.776+0800    INFO    dractions/retry.go:69   drpc  subscription-busybox is not ready yet, retry in 30 seconds
2024-05-06T15:33:46.804+0800    INFO    dractions/retry.go:91   drpc subscription-busybox condition PeerReady is not True
2024-05-06T15:33:46.804+0800    INFO    dractions/retry.go:69   drpc  subscription-busybox is not ready yet, retry in 30 seconds
2024-05-06T15:34:16.818+0800    INFO    dractions/retry.go:56   drpc subscription-busybox is ready
2024-05-06T15:34:16.818+0800    INFO    dractions/actions.go:189    enter DRActions Relocate
2024-05-06T15:34:16.823+0800    INFO    dractions/retry.go:56   drpc subscription-busybox is ready
2024-05-06T15:34:16.823+0800    INFO    dractions/actions.go:208    get drpc subscription-busybox
2024-05-06T15:34:16.826+0800    INFO    dractions/actions.go:215    get drpolicy dr-policy
2024-05-06T15:34:16.832+0800    INFO    dractions/retry.go:29   got placementdecision name subscription-busybox-decision-1
2024-05-06T15:34:16.832+0800    INFO    dractions/retry.go:136  get placementdecision subscription-busybox-decision-1
2024-05-06T15:34:16.835+0800    INFO    dractions/retry.go:144  placementdecision clusterName: rdr-dr2
2024-05-06T15:34:16.835+0800    INFO    dractions/actions.go:227    relocate to cluster: rdr-dr1
2024-05-06T15:34:16.835+0800    INFO    dractions/actions.go:232    update drpc subscription-busybox
2024-05-06T15:34:21.856+0800    INFO    dractions/retry.go:124  current drpc  subscription-busybox phase is Initiating, expecting Relocated, retry in 30 seconds
2024-05-06T15:34:51.864+0800    INFO    dractions/retry.go:124  current drpc  subscription-busybox phase is Relocating, expecting Relocated, retry in 30 seconds
2024-05-06T15:35:21.868+0800    INFO    dractions/retry.go:124  current drpc  subscription-busybox phase is Relocating, expecting Relocated, retry in 30 seconds
2024-05-06T15:35:51.896+0800    INFO    dractions/retry.go:124  current drpc  subscription-busybox phase is Relocating, expecting Relocated, retry in 30 seconds
2024-05-06T15:36:21.901+0800    INFO    dractions/retry.go:114  drpc  subscription-busybox phase is Relocated
2024-05-06T15:36:21.905+0800    INFO    dractions/retry.go:91   drpc subscription-busybox condition PeerReady is not True
2024-05-06T15:36:21.905+0800    INFO    dractions/retry.go:69   drpc  subscription-busybox is not ready yet, retry in 30 seconds
2024-05-06T15:36:51.931+0800    INFO    dractions/retry.go:56   drpc subscription-busybox is ready
2024-05-06T15:36:51.931+0800    INFO    dractions/actions.go:93 enter DRActions DisableProtection
2024-05-06T15:36:51.931+0800    INFO    dractions/actions.go:112    delete drpc subscription-busybox
2024-05-06T15:36:51.947+0800    INFO    dractions/actions.go:118    get placement subscription-busybox
2024-05-06T15:36:51.950+0800    INFO    dractions/actions.go:126    updated placement subscription-busybox annotation
2024-05-06T15:36:51.966+0800    INFO    deployers/subscription.go:61    enter Undeploy Deployment/Subscription
PASS
ok      github.com/ramendr/ramen/e2e    490.696s
(ramen) fusion@s194:~/keslerzhu/ramen/e2e$
ShyamsundarR commented 3 months ago

@par97 for this PR to work in the e2e environment, we need to create a Channel. As the script sets up the RDR env, and deploys and configures Ramen but expects the next tool to create the Channel. (see failure here: https://github.com/RamenDR/ramen/pull/1373 )

In the case of e2e we would need another option, or just take in a .yaml file with the options for the git link to the Channel sources and create a channel from the code to begin with. Example YAML can look like: https://github.com/RamenDR/ramen/blob/main/test/configs/deployment-k8s-regional-rbd.yaml

par97 commented 3 months ago

@ShyamsundarR The channel is a prereq to run the subscription + deployment workload. As documented in https://github.com/RamenDR/ramen/blob/main/docs/testing.md, we need first add a channel:

Ramen basic test use the [ocm-ramen-samples repo](https://github.com/RamenDR/ocm-ramen-samples). Before running the tests, you need to deploy a channel pointing this repo:

kubectl apply -k https://github.com/RamenDR/ocm-ramen-samples.git/channel?ref=main --context hub

I have plan to add check for the channel in the Validate Suite.

To add such channel, we could have a few options:

  1. still doc this step as prereq.
  2. execute above kubectl apply command in the e2e-rdr.sh, or directly kubectl apply the saved local yaml file.
  3. create these resources (namespace, channel) in go code.
ShyamsundarR commented 3 months ago

@ShyamsundarR The channel is a prereq to run the subscription + deployment workload. As documented in https://github.com/RamenDR/ramen/blob/main/docs/testing.md, we need first add a channel:

Ramen basic test use the [ocm-ramen-samples repo](https://github.com/RamenDR/ocm-ramen-samples). Before running the tests, you need to deploy a channel pointing this repo:

kubectl apply -k https://github.com/RamenDR/ocm-ramen-samples.git/channel?ref=main --context hub

I have plan to add check for the channel in the Validate Suite.

To add such channel, we could have a few options:

1. still doc this step as prereq.

2. execute above kubectl apply command in the e2e-rdr.sh, or directly kubectl apply the saved local yaml file.

3. create these resources (namespace, channel) in go code.

I am suggesting (3) because in local setups or when experimenting with non-standard channels (forms of ocm-ramen-sample) it is easier to pass in the git source and create a channel for e2e to use.

ShyamsundarR commented 3 months ago

I added this issue to track some review comments that we can fix subsequently, so that we can make progress on using what is available (most will potentially be resolved once we get to the Sub+CephFS and/or AppSet+Workload as we generalize the framework and utils functions)

ShyamsundarR commented 3 months ago

@par97 other than this one comment I think we are ready to merge this in, once you clarify I can merge this PR based on the clarification.