Add autoscaling support

[bacongobbler]

This PR adds autoscaling support to spin k8s scaffold. It sets some some defaults for CPU and memory utilization if not set by the user, which can be controlled through the --autoscaler-target-cpu-utilization and --autoscaler-target-memory-utilization flags, respectively.

TODO:

• [x] add docs to README
• [x] write tests

><> spin k8s scaffold -f bacongobbler/hello-rust:1.0.0 --autoscaler hpa --cpu-limit 500m --memory-limit 500Mi
apiVersion: core.spinoperator.dev/v1
kind: SpinApp
metadata:
  name: hello-rust
spec:
  image: "bacongobbler/hello-rust:1.0.0"
  executor: containerd-shim-spin
  enableAutoscaling: true
  resources:
    limits:
      cpu: 500m
      memory: 500Mi
---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: hello-rust-autoscaler
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: hello-rust
  minReplicas: 2
  maxReplicas: 3
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 60
  - type: Resource
    resource:
     name: memory
      target:
        type: Utilization
        averageUtilization: 60

[bacongobbler]

I think it's probably best to avoid adding KEDA autoscaling support if at all possible. The text/template approach we're currently using for spin k8s scaffold is going to get really unmaintainable really fast.

[bacongobbler]

nvm - found a way to add KEDA support.

[macolso]

What happens if the user already have HPA or KEDA configured on the cluster, runs scaffold with the new flags and then hits apply? Will an upgrade be ran on their behalf?

[bacongobbler]

If they have a SpinApp named foo in the default namespace, then they run

spin k8s scaffold --from foo:1.0.0 --autoscaler hpa --cpu-limit 500m --memory-limit 500Mi | kubectl apply -f -

Then yes, kubectl will patch the SpinApp/foo resource as well as the HorizontalPodAutoscaler/foo-autoscaler resource in the default namespace.

If they don't want to upgrade existing resources in their cluster, they can use kubectl create -f -, which will return an error if the resource already exists.

How come?

[macolso]

I was curious what the UX would look like if a dev tested locally using spin scaffold and then went to deploy later via kubectl apply but forgot they had been testing autoscale behavior locally (and didn't intend to bring those changes to their cloud environment)

[bacongobbler]

I'm not quite sure I follow what the behavior would be in your proposed scenario. spin k8s scaffold renders a YAML file to your terminal (or to a file if you use the --output flag). It does not install anything in your cluster.

If a user was testing locally using spin scaffold, all they'd see is text displayed on their screen.

Typically a kube config file would have two profiles: one for local development and one for production. They'd use a plugin like kubectx to switch between a local dev environment and a production environment. e.g.

# local development
$ k3s cluster create ...
$ spin k8s scaffold ... | kubectl apply -f -
# switch to prod environment
$ kubectx production
Switched to context "production"
$ spin k8s scaffold ... | kubectl apply -f -

The user would have to either re-invoke spin k8s scaffold with the same flags as they ran in the development context, OR they would have to run kubectl apply -f my-rendered-output-with-k3s.yaml against production. Both cases I'm not really sure how we'd guard against that - they had to explicitly switch contexts AND reuse a file they had previously written to, AND use a command that intentionally patches/upgrades resources in the cluster.

Can you help me understand the scenario you're proposing? I want to understand how a user could bring changes from their development context into their production cluster. Perhaps there's some scenario I'm misunderstanding that we need to guard against.