Enable non-gpu autoscaling with Karpenter - Githubissues

dandi / dandi-hub

Infrastructure and code for the dandihub

https://hub.dandiarchive.org

Other

11 stars 23 forks source link

Enable non-gpu autoscaling with Karpenter #111

Closed asmacdo closed 8 months ago

asmacdo commented 8 months ago

Currently, autoscaling is restricted to a single PySpark profile (as an example).

Acceptance Criteria:

Remove PySpark profile
Enable profiles: https://github.com/dandi/dandi-hub/blob/do-eks/helm/jupyterhub/dandihub.yaml#L90-L134
- Tiny 0.5 CPU / 1 GB
- Base 6 CPU / 16 GB up to 12C/32G
- Medium 12C/32G upto 24C/64G.
- Large 24C/64G upto 48C/96G.
Enable Karpenter Node types to accommodate profiles