Query for multiple root elements by label

alexbarbato commented 2 years ago

Thanks so much for your great work here!!!

Use Case

It'd be so amazing if there were the ability to query by label for "multiple root elements". For example say we have multiple objects all correlated by label, but with their own lineage's below them. They are related and I'd like to view them all at the same time, with a single query.

Rough proposed API

kubectl lineage --selector app=my-awesome-app as an example Shorthand being -l

Rough result

Query for elements that don't have owner-refs and define them as "roots" and apply the standard algorithm and render from there.

alexbarbato commented 2 years ago

I'd potentially even be down to try and help implement this if seemingly useful!

tohjustin commented 2 years ago

Thanks so much for your great work here!!!

Hi @alexbarbato, really appreciate it! 🙂

Just want to clarify a bit more on your request. Given the following command, is this what you are looking for?

kube-lineage --selector $LABEL_SELECTOR

Query all objects in the cluster that matches the $LABEL_SELECTOR label selector expression.
From the result above, mark objects that doesn't have any owner references as a root object.
Print the "lineage" of each root object.

Currently I'm a bit hesitant to implement this right away due to the following reasons 😥

The proposed API doesn't fit nicely into the existing input format that kube-lineage currently conforms to, which is to be as similar as possible to kubectl get.
- COMMAND KIND[.VERSION][.GROUP] [NAME] [flags]
- COMMAND KIND[.VERSION][.GROUP]/NAME [flags]
The "mark objects that doesn't have any owner references as a root object" part above doesn't align well with the design goal which is to be as generic as possible & not implement any "special logic" for a particular relationship type (OwnerReference/ControllerReference in this case).

Would it be possible to provide more information to help me understand your use-case better?

Thank you! 🙏

tohjustin commented 2 years ago

By the way, would any of these potential features satisfy your use case? (currently in the backlog)

Support printing multiple root objects similar to kubectl get
```
kube-lineage $KIND_1,$KIND_2
kube-lineage $KIND $NAME_1,$NAME_2
kube-lineage $KIND_1/$NAME_1 $KIND_2/$NAME_2
```
1. Query the collection of objects specified in the command input.
2. From the result above, mark all objects as a root object.
3. Print the "lineage" of each root object.
Support printing a subset of objects via --selector/-l flag, similar to kubectl get $KIND --selector $LABEL_SELECTOR
```
kube-lineage $KIND           --selector $LABEL_SELECTOR
kube-lineage $KIND_1,$KIND_2 --selector $LABEL_SELECTOR
```
1. Query the collection of objects specified in the command input that matches the $LABEL_SELECTOR label selector expression.
2. From the result above, mark all objects as a root object.
3. Print the "lineage" of each root object.
Support recognizing custom resource relationships (eg. ArgoCD's Application, Helm Controller's HelmRelease etc.)
```
kube-lineage $CUSTOM_RESOURCE_TYPE/$NAME
kube-lineage $CUSTOM_RESOURCE_TYPE $NAME
```
1. Print the lineages of each custom resource object

alexbarbato commented 2 years ago

Thanks so much @tohjustin :D

I totally understand your reservations and it makes sense to me what you're trying to accomplish.

Technically what you're doing in Scenario 2 around querying for multiple KINDS with a label selector would make what I'm trying to do possible in 1 command versus the two it is now.

To give you a bit more context, here's the workflow I'm following right now.

Create a Cartographer supply chain. (I'm using the Tanzu Application Platform Beta)
This results in two key CRD's being created when using their "Default/Out of the Box Supply Chain" - Workload and a serving.service.knative.dev.
- The Workload created here is the Cartographer CRD that effectively reflects a delivery path.
- The serving.service.knative.dev is the deployed application that results as a part of the Cartographer Supply Chain.

I end up having to query for both CRD's because they are all associated by a specific label versus owner refs across both objects to get a more holistic picture of the supply chain running and the resulting KNative service that is created.

I definitely wouldn't recommend you build something exclusive for my use case, but hopefully the context gives you a bit better understanding of what I'm trying to accomplish.

alexbarbato commented 2 years ago

It is potentially worth mentioning that the Supply Chain above may have any other number of related CRD's that it doesn't "own", but were created as a part of the Workload and associated by label.

So right now it's only two KINDS I'm querying, but that could grow.

tohjustin commented 2 years ago

Hi @alexbarbato, really appreciate for the detailed description of your use-case! 🙏

I'm glad to hear that scenario 2 would be able to handle your current use case. While waiting for the features demonstrated in scenario 1 & 2 to implemented, would the following bash function work for you?

kubelineagebylabel() {
  KINDS=workload.carto.run,serving.service.knative.dev
  for RESOURCE in $(kubectl get $KINDS -l $1 -o name); do kubectl lineage $RESOURCE; done
}

$ kubelineagebylabel app.tanzu.vmware.com/workload-type=web

tohjustin commented 2 years ago

This also the first time I've heard of the Cartographer project, it looks really interesting after taking a quick look at its documentation. I have a few more questions regarding how it works if you don't mind:

Is the "Create a Cartographer supply chain" step that you've mentioned earlier equivalent to creating a ClusterSupplyChain custom resource?
And the "associated by a specific label" refers to the label selector that you define in ClusterSupplyChain.spec.selector?

Because if that's the case, would something like this make sense to you?

$ kube-lineage clustersupplychain $SUPPLY_CHAIN_NAME

kube-lineage could then implement custom logic to find all the Workload objects w/ matching labels + other resources like serving.service.knative.dev (depending on the ClusterSupplyChain/$SUPPLY_CHAIN_NAME spec) & mark them as related to the provided ClusterSupplyChain object, which is essentially scenario 3.

alexbarbato commented 2 years ago

Is the "Create a Cartographer supply chain" step that you've mentioned earlier equivalent to creating a ClusterSupplyChain custom resource?

Yup, effectively (the Tanzu Application Platform Beta ships with default supply chains so in my case I'm not actually writing these by hand, but I could)

And the "associated by a specific label" refers to the label selector that you define in ClusterSupplyChain.spec.selector?

So yes and no, I'd actually be looking for something like carto.run/workload-name=NAME which is a label applied to the components that are a part of the Cartographer run. I don't actually care tooo much about the top level ClusterSupplyChain. Depending on how you implemented that custom logic it could totally work as you described so I guess I'd have to dig a bit deeper to really give you a firm 👍 or 👎

rombert commented 2 years ago

FWIW, I would find an implementation of Scenario 2 as described in https://github.com/tohjustin/kube-lineage/issues/2#issuecomment-955153758 useful.

tohjustin / kube-lineage