search

cloudfoundry / cf-crd-explorations

Apache License 2.0
3 stars 2 forks source link

Use experimental CF CRs to drive workload runtime #3

Closed tcdowney closed 3 years ago

tcdowney commented 3 years ago

Background

We want to get a better understanding of what building a CF v3 API that is backed by custom resources actually looks like. Assume you already have a "staged" app/runnable app image. Consider making an app we can use to demo and experiment with. Other teams have found spring-pet-clinic to be a good candidate since it is Java (what most CF devs use) and relies on a MySQL database. Using the CRDs (probably App, Process, Droplet) we created in #1, let's see how useful they are in driving the underlying workload runtime (e.g. Knative, Eirini LRP, Kubernetes Deployments, etc.). We should build some experimental controllers to reconcile these CRs and create the necessary workload resources.

The goal of this spike is to answer the following questions:

Scenario: Applying pre-made CRs makes a running workload

Given a set of example custom resources and controllers installed on the cluster, When Ikubectl apply <app/process/droplet>, Then I see that my application is running (and can curl it/access it, etc)

(Note: Out of band, this requires an LB service to access the application).

Deliverable

Dev Notes

Dev Provided Acceptance Steps:

  1. Claim an environment
  2. Follow the instructions in the README (summarized below)
    1. Deploy the https://github.com/cloudfoundry/cf-for-k8s/tree/eirini-controller-enabled branch of cf-for-k8s
    2. make install this repo
    3. make run the controller
    4. Update the sample Route CR to reference the apps domain you used when deploying cf-for-k8s
    5. kubectl -n cf-workloads apply --recursive -f config/samples/. to apply the CRs
    6. Run the curl command in the README to update the Droplet status
  3. See the pod is up and running, kubectl -n cf-workloads get pods, and see if you can curl it at the address you specified in the sample Route
  4. Update the App yaml (config/samples/apps_v1alpha1_app.yaml) to point to the other app env secret name (my-other-app-guid-env) and kubectl apply the App again. You should see these new environment variables reflected on the Eirini LRP resource and pods
  5. You can use the same curl as earlier to update the Droplet status to a different image and see the Eirini LRP / pods get updated to point to that image
  6. Your imagination is your limit 🙃
tcdowney commented 3 years ago

Work is currently on the https://github.com/cloudfoundry/cf-crd-explorations/tree/rebased-workloads-from-crds-spike-tme-155 branch.

tcdowney commented 3 years ago

How to update the Status subresource for Droplets

  1. kubectl proxy &
  2. 
NAMESPACE=cf-workloads
DROPLET_NAME=kpack-droplet-guid

curl -k -s -X PATCH -H "Accept: application/json, /" \ -H "Content-Type: application/merge-patch+json" \ 127.0.0.1:8001/apis/apps.cloudfoundry.org/v1alpha1/namespaces/$NAMESPACE/droplets/$DROPLET_NAME/status \ --data '{"status":{"image": {"reference": "relintdockerhubpushbot/dora", "pullSecretName": ""}, "conditions": []}}'

heycait commented 3 years ago

Tim and I did another spike exploration on this branch where the App Controller reads the Processes and references the Droplet to create the LRPs for each process.

tcdowney commented 3 years ago

One thing @heycait and I are thinking through is how can we get changes to either an App, Process, or Droplet to trigger updating the Eirini LRP.

A couple thoughts we want to explore are:

  1. What does an App Controller that watches Apps and all Process/Droplets that it owns look like (which we're exploring on the spike-app-controller branch linked in the comment above)
  2. What does propagating state from the App and Droplet to the Processes look like (which loosely is what main is doing). In this case individual App and Droplet reconcilers would react to their corresponding changes and update the relevant Processes which then triggers the Process controller to update the LRP
  3. ???

Still thinking all this through and exploring... just noting down what we're thinking about

heycait commented 3 years ago

Tim found some guidance one how to design controllers. See this doc.

Example: If you need to update all Services in response to a Node being added - Reconciler Services but Watch Nodes (transformed to Service object name / Namespaces) instead of Reconciling Nodes and updating Services

We should have the Process watch for changes to the Application and Droplet instead. We are therefore going back to the original spike with all the things happening within the Process Controller.

The next step is to determine how we can get the controller to watch for changes to these other objects and trigger Reconciles on them. As the code stands now, the Process Controller only Reconciles on changes to the Process itself. However, this presents a problem of ordering. If the Process is applied first, followed by the App and Droplet, the Process would never creating a running LRP and would need to be deleted and applied again.

angelachin commented 3 years ago

I know this spike is assuming that an app/process/droplet CR already exists, but I'm curious to know if we've thought about the order in which each of these gets created? Right now, it seems like they are referring to each other mainly through fields in the spec, but I'm wondering how much we can rely on knowing what to fill in for those fields ahead of time. Is the shim setting all the spec fields?

tcdowney commented 3 years ago

@angelachin ideally the order in which they're created doesn't actually matter and the controller will just no-op if an app isn't runnable yet (e.g. a staged Droplet isn't assigned / doesn't exist yet) and it will re-reconcile once the resources it is watching change. I am imagining the shim / controllers does set the spec fields on the users behalf in response to imperative actions taken via the CF APIs. For example, a user might make the following request to set the current Droplet on an App:

curl "https://api.example.org/v3/apps/[guid]/relationships/current_droplet" \
  -X PATCH \
  -H "Authorization: bearer [token]" \
  -H "Content-type: application/json" \
  -d '{ "data": { "guid": "[droplet_guid]" } }'

The shim in this case would update the Spec.CurrentDropletRef field on the corresponding App. The user could also do this themselves by directly updating the App CR.

For many of the V3 CF APIs, relationships are represented this way and set by API users: https://v3-apidocs.cloudfoundry.org/version/3.101.0/index.html#relationships

I realize that while this is most convenient for representing the existing CF APIs it is perhaps not the most "Kubernetes native" way to model this, though.

It may be a bit trickier to reason about, but alternatively we could use the selector.matchLabels pattern and have parent object claim children that match the selectors. An App would adopt Droplets, Processes, Builds, etc. like how a ReplicaSet adopts Pods.

I think there are some unanswered questions here about what happens if multiple Apps try to adopt the same resources and it may make validations around uniqueness/ownership more difficult (I'm picturing orphaned resources being a problem).

angelachin commented 3 years ago

Cool. I think the struggle with relationship mapping could be helped a lot by having a "AppManifest"-like CRD that is the entry point for both our shim and a user directly editing with kubectl. That way that resource could own all of these child resources and show the relationship between them.

If the shim is setting the spec fields, that makes sense to me. The main thing we want to avoid is having a controller editing a spec field-- I believe that general k8s controller development assumes controllers should only ever be creating new resources or updating the status fields.

tcdowney commented 3 years ago

The main thing we want to avoid is having a controller editing a spec field-- I believe that general k8s controller development assumes controllers should only ever be creating new resources or updating the status fields.

I'm not sure I 100% agree with this.

From the K8s api-conventions doc:

The specification is a complete description of the desired state, including configuration settings provided by the user, default values expanded by the system, and properties initialized or otherwise changed after creation by other ecosystem components (e.g., schedulers, auto-scalers), and is persisted in stable storage with the API object.

IMO the Spec/Status distinction just becomes a Desired State vs Actual State question and if controllers have input into what the desired state should be then it should be valid for them to modify an object's Spec. I think we'll have instances of this where we need to be able to set reasonable defaults for things like memory/disk limits, health checks, etc. that the user can also set but are free to omit.

There's examples of this in the default types like how users can set clusterIP on a Service Spec, but if they omit it the controller will populate it or how when creating a Job the Pod selector is both user specifiable / controller defaultable.

This is just my understanding though, and I'm happy to be shown examples / best practices saying otherwise.

angelachin commented 3 years ago

I think we'll have instances of this where we need to be able to set reasonable defaults for things like memory/disk limits, health checks, etc. that the user can also set but are free to omit.

Yeah I agree. I think my initial wording/position isn't entirely correct. I think what I want to call out is to think critically about a controller updating the spec field so that we don't clobber something that a user has provided. I agree that I think it gets back to Desired vs Actual state, and for the most part since controllers are reconciling resources, I'd expect them to be updating the actual state and thus the status fields.

tcdowney commented 3 years ago

Yep, makes sense @angelachin!

As an aside, I've added a couple of diagrams to our Miro board to help folks visualize what the controller is doing and what data ends up on the Eirini LRP.

For posterity, copies are attached below, but any updates will be made on the Miro.

spike-process-controller-small

spike-eirini-lrp-composition

angelachin commented 3 years ago

Looks like all of this work in on the main branch, so closing this issue out.