planning / todo list of backlog items

[AmitKumarDas]

• mayastor install
• cstor pool day 2 ops
• dmaas new features
• openebs upgrade
• kubera charts testing
• align resources with components/storage engines
• ability to bring up temp k8s clusters to test node cordon, k8s upgrade test cases

[AmitKumarDas]

User References:

In our environment we have 3 openebs nodes running on 3 different esxi hosts. We need to patch esxi hosts. Here the steps we are planning to follow:
1. Cordon and drain node.
2. Shutdown node and move (vmotion) it to a different host.
3. Power-on the node and uncordon it once in ready state.
4. Confirm both CSP and CVR must be in Healthy state after reboot.
5. Repeat same procedure with other nodes.
We would like to get your opinion on above steps and ensure this if this will cause any outage to end users. Any suggestions will be welcome.

[AmitKumarDas]

Will this help to setup & teardown clusters ? https://github.com/banzaicloud/pipeline cc @gprasath We shall check later i.e. post Jun 15

[AmitKumarDas]

Can these helps with our goal to setup & teardown k8s clusters real easy & quick:

• https://typhoon.psdn.io/
• https://cluster-api.sigs.k8s.io/reference/providers.html
• https://kind.sigs.k8s.io
  • a wrapper to set this up based on our customizations can help run good number of testcases
  • a sample wrapper can be found here:
  • https://cluster-api.sigs.k8s.io/developer/e2e.html#developing-e2e-tests

cc @ajeshbaby @gprasath

These are the current / popular bootstrap tools:

• kubeadm
• kops
• tectonic
• kubespray

[AmitKumarDas]

Following are the list of painpoints discussed with the team before starting off with Kubera-1.11 scope:

Items are assigned a priority level A priority level of 1 indicates the highest significance while 5 indicates lowest

- Which pipelines are owned by e2e team

• merges to ci.mayadata.io will be owned by the Director dev team
  • new testcases added to release pipelines will be added (via PullRequest) to ci.mayadata.io (weekly once) by the e2e team
  • merge rights are owned by Director dev team
  • failure in pipelines will be monitored by Director dev team
• release pipelines are owned by e2e team
• new updates:
  • As discussed e2e team will try to better the infrastructure & remove the need for manual involvement. This will take good effort & time. E2E team will start building the required tooling to get this done.

- Integration Test Submissions from dev teams

• OpenEBS dev team is not submitting new any testcases cc @kmova
• There are no test cases from DAO teams - cc @vishnuitta @avishnu
• There are no test cases submitted from Director team - cc @avishnu - NOTE: These tests are added to release pipelines

- Pipelines are not run on a regular basis for ci builds of OpenEBS

• Openshift pipeline will be scheduled at night to run ci builds & later release builds
• Konvoy pipeline will be scheduled to run for release builds only - Once release builds are out, above pipelines will be dedicated to run release builds only

- How to reduce the time taken to run e2e pipelines - [P 4]

• Some of the pipelines in OpenEBS community takes 4 hours to run to completion
• OnPrem pipelines will aggravate this
• More number of testcases will aggravate this - Solutions/Ideas:
  • Run module specific test cases in the pipelines
  • Have more control on executing stages/jobs, etc
  • Gitlab does not provide enough tunables to do this

- Pipelines do not provide the root cause of problems/failures [P 4]

• Needs to be fixed
• Director logs may be helpful to find root cause in the long run
• Move cluster cleanup to the start of pipeline run for on prem setup
• Ability to push pod logs, resource yaml descriptions to some github repo for later analysis
• Add cluster health or product health testcases before running the pipelines

- Rancher pipeline is flaky [P 1]

• All Director & OpenEBS components/controllers should expose Prometheus metrics that can indicate if any particular controller is aggressive in querying kubernetes api server.
• Perhaps run some pipelines/ monitor by keeping cstor & other openebs controllers in openebs namespace instead of current kube-system namespace
• E2E team will be looking into various infra aspects & sort this out

- Proper handover process [P 1]

• DOP & OpenEBS enterprise builds should not be tied with Director Online & OpenEBS community
• In other words, EE builds will happen on the scheduled date
• 17th of a month should be the first RC build
• 22nd of a month is the code freeze date & should be respected to maintain the enterprise quality

- Challenges with current GUI testing [P 3]

• tests are not declarative
• Should we depend entirely on Selenium
• Selenium-based tests need a connection to AWS instances
• Running selenium tests requires the use of AWS spot instances
• Running selenium tests in on-prem is flaky
• Need to eliminate manual testing of Director OnPrem

- Docs to create an pipeline setup that includes bringing up K8s cluster [P 4]

• OpenEBS e2e team is not aware of the process followed to test Director OnPrem
• Kubera e2e team is not aware of the process followed to test OpenEBS community

- What should be the focus area of e2e team? [P 1]

• Should e2e team focus on coverage percentage?
• pick up some amount of coverage
• Should e2e focus on verifying stateful workloads?
• Should e2e focus on verifying/hardening user-specific workflows?
  • pick up one or two cases
• Should e2e focus on testing DOP?
• Should e2e focus on testing DAO?

- Improve the infrastructure required to run pipelines? [P 4]

• e.g. How to make them pure declarative implementations?
• e.g. How to generate documentation from declarative infra?
• e.g. How to setup pipeline infra on a laptop, on Travis, on GitHub Actions, etc?

- Improve test case implementations that provide granular control over running these tests [P 2]

• e.g. How to make them pure declarative implementations?
• e.g. How to expose metrics from each testcase run automatically?
• e.g. How to run a testcase on laptop, on Travis, on GitHub Actions, etc?
• e.g. How to run a suite of testcases on laptop, on Travis, on Github Actions, etc?
• e.g. How to generate documentation from declarative testcases?

- Expose e2e metrics over a dashboard [P 2]

- Performance testing of mayastor? [P 5]

https://docs.google.com/document/d/17osIP5WAhKYm9MpuBORgPcxOGiidvT4fowdeYhFF8hE/

[AmitKumarDas]

GUI Testing Thoughts/Innovations:

• https://blog.logrocket.com/cypress-io-the-selenium-killer/

[AmitKumarDas]

Closing this since things are being tracked under project plan