rook: add test script

[lunkan93]

What kind of PR is this?

Required: Mark one of the following that is applicable:

• [x] kind/feature
• [ ] kind/improvement
• [ ] kind/deprecation
• [ ] kind/documentation
• [ ] kind/clean-up
• [ ] kind/bug
• [ ] kind/other

Optional: Mark one or more of the following that are applicable:

[!important] Critical security fixes should be marked with kind/security Breaking changes should be marked kind/admin-change or kind/dev-change depending on type

• [ ] kind/admin-change
• [ ] kind/dev-change
• [ ] kind/security
• [ ] [kind/adr]()

What does this PR do / why do we need this PR?

Added a test script for rook-ceph.

...

• Fixes #279

Additional information to reviewers

Screenshots

Checklist

• [x] Proper commit message prefix on all commits
• Change checks:
  • [x] The change is transparent
  • [ ] The change is disruptive
  • [ ] The change requires no migration steps
  • [ ] The change requires migration steps
• Metrics checks:
  • [ ] The metrics are still exposed and present in Grafana after the change
  • [ ] The metrics names didn't change (Grafana dashboards and Prometheus alerts are not affected)
  • [ ] The metrics names did change (Grafana dashboards and Prometheus alerts were fixed)
• Logs checks:
  • [ ] The logs do not show any errors after the change
• Network Policy checks:
  • [ ] Any changed pod is covered by Network Policies
  • [ ] The change does not cause any dropped packages in the NetworkPolicy Dashboard
• Pod Security Policy checks:
  • [ ] Any changed pod is covered by Pod Security Admission
  • [ ] Any changed pod is covered by Gatekeeper Pod Security Policies
  • [ ] The change does not cause any pods to be blocked by Pod Security Admission or Policies
• Falco checks:
  • [ ] The change does not cause any alerts to be generated by Falco

[lunkan93]

I will need to fix some more things to be able to run it in the pipeline, currently looking into that :sweat_smile:

[lunkan93]

I've tried to address your feedback @Xartos - let me know if I missed something :smile:

Example output with latest changes

❯ ./scripts/test-rook.sh both Testing rook-ceph in sc.. Testing Deployments ==================== csi-rbdplugin-provisioner exists ✔ ready ✔ rook-ceph-operator exists ✔ ready ✔ rook-ceph-toolbox exists ✔ ready ✔ rook-ceph-crashcollector-simon-sc-worker-0 exists ✔ ready ✔ rook-ceph-crashcollector-simon-sc-worker-1 exists ✔ ready ✔ rook-ceph-crashcollector-simon-sc-worker-2 exists ✔ ready ✔ rook-ceph-crashcollector-simon-sc-worker-3 exists ✔ ready ✔ rook-ceph-crashcollector-simon-sc-worker-5 exists ✔ ready ✔ Testing Mons ==================== 3/3 deployments exist ✔ rook-ceph-mon-a ready ✔ rook-ceph-mon-b ready ✔ rook-ceph-mon-c ready ✔ Testing Mgrs ==================== 1/1 deployments exist ✔ rook-ceph-mgr-a ready ✔ Testing OSDs ==================== 5/5 deployments exist ✔ rook-ceph-osd-0 ready ✔ rook-ceph-osd-1 ready ✔ rook-ceph-osd-2 ready ✔ rook-ceph-osd-3 ready ✔ rook-ceph-osd-4 ready ✔ Testing DaemonSets ==================== csi-rbdplugin exists ✔ ready ✔ Testing Jobs ==================== rook-ceph-osd-prepare-simon-sc-worker-0 exists ✔ completed ✔ rook-ceph-osd-prepare-simon-sc-worker-1 exists ✔ completed ✔ rook-ceph-osd-prepare-simon-sc-worker-2 exists ✔ completed ✔ rook-ceph-osd-prepare-simon-sc-worker-3 exists ✔ completed ✔ rook-ceph-osd-prepare-simon-sc-worker-5 exists ✔ completed ✔ Testing CephClusters ==================== Cluster health is not ok: HEALTH_WARN ❌ Cluster phase is ready ✔ Testing ServiceMonitors ==================== Checking targets up to 5 times to avoid flakes... 1 Running tests... serviceMonitor/rook-ceph/rook-ceph-mgr/0 ✔ Testing rook-ceph in wc.. Testing Deployments ==================== csi-rbdplugin-provisioner exists ✔ ready ✔ rook-ceph-operator exists ✔ ready ✔ rook-ceph-toolbox exists ✔ ready ✔ rook-ceph-crashcollector-simon-wc-worker-0 exists ✔ ready ✔ rook-ceph-crashcollector-simon-wc-worker-1 exists ✔ ready ✔ rook-ceph-crashcollector-simon-wc-worker-2 exists ✔ ready ✔ Testing Mons ==================== 3/3 deployments exist ✔ rook-ceph-mon-a ready ✔ rook-ceph-mon-b ready ✔ rook-ceph-mon-c ready ✔ Testing Mgrs ==================== 1/1 deployments exist ✔ rook-ceph-mgr-a ready ✔ Testing OSDs ==================== 3/3 deployments exist ✔ rook-ceph-osd-0 ready ✔ rook-ceph-osd-1 ready ✔ rook-ceph-osd-2 ready ✔ Testing DaemonSets ==================== csi-rbdplugin exists ✔ ready ✔ Testing Jobs ==================== rook-ceph-osd-prepare-simon-wc-worker-0 exists ✔ completed ✔ rook-ceph-osd-prepare-simon-wc-worker-1 exists ✔ completed ✔ rook-ceph-osd-prepare-simon-wc-worker-2 exists ✔ completed ✔ Testing CephClusters ==================== Cluster health is not ok: HEALTH_WARN ❌ Cluster phase is ready ✔ Testing ServiceMonitors ==================== Checking targets up to 5 times to avoid flakes... 1 Running tests... serviceMonitor/rook-ceph/rook-ceph-mgr/0 ✔ Summary ==================== Successes: 58 Failures: 2 Something failed Json output of failed test resources ==================================== { "MON_CLOCK_SKEW": { "message": "clock skew detected on mon.b, mon.c", "severity": "HEALTH_WARN" } } { "MON_DISK_LOW": { "message": "mon a is low on available space", "severity": "HEALTH_WARN" } } Unhealthy/missing prometheus targets ====================================