META - Extension System Testing

nebari-dev / nebari

🪴 Nebari - your open source data science platform

https://nebari.dev

BSD 3-Clause "New" or "Revised" License

281 stars 93 forks source link

META - Extension System Testing #1862

Open costrouc opened 1 year ago

costrouc commented 1 year ago

Context

#1833 is a huge PR. It need significant testing. Here is an exhaustive checklist

Value and/or benefit

Ensure this PR is successful and all previous behavior is preserved.

### Providers
- [ ] https://github.com/nebari-dev/nebari/issues/1951
- [ ] https://github.com/nebari-dev/nebari/issues/1952
- [ ] https://github.com/nebari-dev/nebari/issues/1953
- [ ] https://github.com/nebari-dev/nebari/issues/1954

### Extra
- [ ] https://github.com/nebari-dev/nebari/issues/1955
- [ ] https://github.com/nebari-dev/nebari/issues/1958

When testing out a checklist item follow https://github.com/nebari-dev/nebari/issues/new?assignees=&labels=type%3A+release+%F0%9F%8F%B7&template=testing-checklist.md&title=Testing+checklist+for+%3Cversion%3E.

costrouc commented 1 year ago

Breaking up tasks:

stream for cli testing "framework" @costrouc
stream for cloud testing "framework" @aktech
configuration "fixture" for specifying things like ("aws" + "gpu" + "github"), documentation or some easy way to allow more configuration matrix tests @iameskild
specifying expectations of minimal "working check" e.g. for gpu with nvidia-smi working does that count?
tear-downs for all clouds (load-balancers and storage cleaned up)
schema fields all having descriptions and additional testing where needed. https://github.com/nebari-dev/nebari/pull/1833/files#diff-f7cda9f473c67dd06224a3ab445bb68065976dc6593511fab331179d60a50013R69 (stretch goal) not needed for this issue.

aktech commented 1 year ago

One note the cleaning of resources, it doesn't cleans quite well by the way:

2023-08-01 13:27:23      INFO   49 utils: destroying Nebari took 3220.010 [s]
2023-08-01 13:27:23     ERROR  186 destroy: Stage=stages/07-kubernetes-services failed to fully destroy
2023-08-01 13:27:23     ERROR  186 destroy: Stage=stages/05-kubernetes-keycloak failed to fully destroy
2023-08-01 13:27:23     ERROR  186 destroy: Stage=stages/04-kubernetes-ingress failed to fully destroy
2023-08-01 13:27:23     ERROR  186 destroy: Stage=stages/03-kubernetes-initialize failed to fully destroy
2023-08-01 13:27:23     ERROR  186 destroy: Stage=stages/02-infrastructure failed to fully destroy
2023-08-01 13:27:23     ERROR  189 destroy: ERROR: not all nebari stages were destroyed properly. For cloud deployments of Nebari typically only stages 01 and 02 need to succeed to properly destroy everything