Label pod with workflow's labels

[hadim]

Currently, the pods of a given workflow are labelled with workflows.argoproj.io/completed and workflows.argoproj.io/workflow. Could we also add the labels of the workflow to that list, something like:

workflows.argoproj.io/workflow/label/project = "xxxx"
workflows.argoproj.io/workflow/label/tenant = "xxxx"

[alexec]

I think you can label your pods as follows. Look at examples/pod-metadata.yaml

[hadim]

Thanks. I am aware of this and should have mentioned it actually in my first message. I am looking for an automated way of doing it.

But maybe it's a too specific feature for Argo? (feel free to close if you think it is)

As a workaround, I'll use examples/pod-metadata.yaml.

[alexec]

Would you like to submit a PR to add this feature?

[hadim]

I'll consider it for sure if the examples/pod-metadata.yaml workaround becomes too cumbersome.

[arghya88]

@alexec I am interested to contribute this.Can you assign this to me?

[hadim]

I started to look at various entry points in the code to integrate this. I found https://github.com/argoproj/argo/blob/91bce2574fab15f4fab4bc4df9e50563aa748838/workflow/controller/workflowpod.go#L131 to be a potentially good candidate to add the labels of the workflow spec on-the-fly during pod creation. Maybe @alexec would like to see this elsewhere?

Another question is do we want to force by default? or is it best to add an option in the configmap about it?

[hadim]

@arghya88 still interested working on that?

[terrytangyuan]

@hadim You can now define podMetadata at workflow level and then those metadata will also appear in pods. See https://github.com/argoproj/argo-workflows/blob/master/examples/pod-metadata-wf-field.yaml for an example. Is this what you need? Or are you looking for inheriting existing labels from workflow to pods?

[terrytangyuan]

If you need the latter, what is your use case? The workflow is the owner of your pods so you can also obtain the labels from the referenced workflow.

[hadim]

I was looking for a way to automatically set those labels from the labels of the workflow. But yeah I'll use podMetadata if that's not something you think is worse adding.

[terrytangyuan]

The labels workflows.argoproj.io/completed and workflows.argoproj.io/workflow are there for other purposes. I am not sure if it's necessary to inherit other labels/metadata as well since they can be obtained easily by owner reference. @alexec WDYT?

[ludydoo]

You can use a MutatingWebhookConfiguration to add labels to pods that have a Workflow owner reference

// main.go 
package main

import (
    "context"
    "flag"
    "fmt"
    "github.com/google/martian/log"
    "github.com/sirupsen/logrus"
    kwhhttp "github.com/slok/kubewebhook/v2/pkg/http"
    kwhlogrus "github.com/slok/kubewebhook/v2/pkg/log/logrus"
    kwhmodel "github.com/slok/kubewebhook/v2/pkg/model"
    kwhmutating "github.com/slok/kubewebhook/v2/pkg/webhook/mutating"
    corev1 "k8s.io/api/core/v1"
    "k8s.io/apimachinery/pkg/apis/meta/v1"
    "net/http"
    "os"
    "strings"
    "time"
)

type config struct {
    certFile string
    keyFile  string
}

func initFlags() *config {
    cfg := &config{}

    fl := flag.NewFlagSet(os.Args[0], flag.ExitOnError)
    fl.StringVar(&cfg.certFile, "tls-cert-file", "", "TLS certificate file")
    fl.StringVar(&cfg.keyFile, "tls-key-file", "", "TLS key file")

    if err := fl.Parse(os.Args[1:]); err != nil {
        err := fmt.Errorf("could not parse flags: %w", err)
        log.Errorf(err.Error())
        panic(err)
    }

    log.Infof("Using tls cert file at path: %s", cfg.certFile)
    log.Infof("Using tls key file at path: %s", cfg.keyFile)

    return cfg
}

func checkArgoWorkflowLabelNotPresent(pod *corev1.Pod) (*kwhmutating.MutatorResult, error) {
    if pod.Labels == nil {
        return &kwhmutating.MutatorResult{}, nil
    }
    if _, ok := pod.Labels["argo"]; ok {
        return &kwhmutating.MutatorResult{}, fmt.Errorf("forbidden: pod cannot have label argo=workflow as it is not owned by an Argo Workflow")
    }
    return &kwhmutating.MutatorResult{}, nil
}

func run() error {

    logrusLogEntry := logrus.NewEntry(logrus.New())
    logrusLogEntry.Logger.SetLevel(logrus.DebugLevel)
    logger := kwhlogrus.NewLogrus(logrusLogEntry)

    logger.Infof("running...")
    cfg := initFlags()

    logger.Infof("creating mutator func")
    var mf kwhmutating.MutatorFunc = func(ctx context.Context, review *kwhmodel.AdmissionReview, object v1.Object) (*kwhmutating.MutatorResult, error) {
        pod, ok := object.(*corev1.Pod)
        if !ok {
            return &kwhmutating.MutatorResult{}, nil
        }
        if pod.OwnerReferences == nil || len(pod.OwnerReferences) == 0 {
            return checkArgoWorkflowLabelNotPresent(pod)
        }
        var found = false
        for _, reference := range pod.OwnerReferences {
            if reference.Kind == "Workflow" && strings.Contains(reference.APIVersion, "argoproj.io/") {
                found = true
                break
            }
        }
        if !found {
            return checkArgoWorkflowLabelNotPresent(pod)
        }
        if pod.Labels == nil {
            pod.Labels = make(map[string]string)
        }
        currentValue, ok := pod.Labels["argo"]
        if ok && currentValue != "workflow" {
            return &kwhmutating.MutatorResult{}, fmt.Errorf("invalid label value argo='%s'", currentValue)
        }
        pod.Labels["argo"] = "workflow"
        return &kwhmutating.MutatorResult{MutatedObject: pod}, nil
    }

    logger.Infof("creating webhook config")
    mcfg := kwhmutating.WebhookConfig{
        ID:      "argo-workflow-labeler",
        Mutator: mf,
        Logger:  logger,
    }

    logger.Infof("creating webhook")
    wh, err := kwhmutating.NewWebhook(mcfg)
    if err != nil {
        return fmt.Errorf("error creating webhook: %w", err)
    }

    logger.Infof("creating http handler")
    whHandler, err := kwhhttp.HandlerFor(kwhhttp.HandlerConfig{Webhook: wh, Logger: logger})
    if err != nil {
        return fmt.Errorf("error creating webhook handler: %w", err)
    }

    // Serve.
    logger.Infof("Listening on :8080")
    err = http.ListenAndServeTLS(":8080", cfg.certFile, cfg.keyFile, whHandler)
    if err != nil {
        return fmt.Errorf("error serving webhook: %w", err)
    }

    return nil

}

func main() {

    log.Infof("sleeping a bit...")
    time.Sleep(5 * time.Second)

    log.Infof("starting app")
    err := run()
    if err != nil {
        log.Errorf("error running app: %v", err)
        time.Sleep(time.Second)
        os.Exit(1)
    }
}

# Dockerfile
FROM golang:1.16 as builder
WORKDIR /workspace
COPY go.mod go.mod
COPY go.sum go.sum
RUN go mod download
COPY main.go main.go
RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 GO111MODULE=on go build -a -o argo-workflow-labeler main.go
FROM gcr.io/distroless/static:nonroot
COPY --from=builder /workspace/argo-workflow-labeler /argo-workflow-labeler
USER nonroot:nonroot
CMD ["/argo-workflow-labeler"]

# Makefile
VERSION ?= latest
IMAGE ?= my-container-registry/argo-workflow-labeler

# Build the docker image
pack:
    docker build . -t ${IMAGE}:${VERSION}

# Push the docker image
push: pack
    docker push ${IMAGE}:${VERSION}

# Certificate
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: argo-workflow-labeler-tls
  namespace: argo
spec:
  secretName: argo-workflow-labeler-tls
  dnsNames:
    - argo-workflow-labeler.argo.svc
    - argo-workflow-labeler.argo.svc.cluster.local
  commonName: argo-workflow-labeler.argo.svc
  issuerRef:
    name: selfsigned
---
# Issuer
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: selfsigned
  namespace: argo
spec:
  selfSigned: {}
---
# Service
apiVersion: v1
kind: Service
metadata:
  name: argo-workflow-labeler
  namespace: argo
spec:
  ports:
    - port: 443
      targetPort: 8080
      protocol: TCP
      name: webhook
  selector:
    app: argo-workflow-labeler
---
# ServiceAccount
apiVersion: v1
kind: ServiceAccount
metadata:
  name: argo-workflow-labeler
  namespace: argo
---
# MutatingWebhookConfiguration
apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  name: argo-workflow-labeler
  labels:
    app: argo-workflow-labeler
    kind: mutator
  annotations:
    cert-manager.io/inject-ca-from: argo/argo-workflow-labeler-tls
webhooks:
  - name: argo-workflow-labeler.example.com
    admissionReviewVersions:
      - v1
    sideEffects: NoneOnDryRun
    clientConfig:
      service:
        port: 443
        name: argo-workflow-labeler
        namespace: argo
        path: "/mutate"
    objectSelector:
      matchExpressions:
        - key: app
          operator: NotIn
          values: [ argo-workflow-labeler ]
    rules:
      - operations: [ "CREATE" ]
        apiGroups: [ "" ]
        apiVersions: [ "v1" ]
        resources: [ "pods" ]
---
# Deployment
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: argo-workflow-labeler
  namespace: argo
spec:
  host: argo-workflow-labeler.argo.svc.cluster.local
  trafficPolicy:
    tls:
      mode: DISABLE
  exportTo:
    - "*"
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: argo-workflow-labeler
  namespace: argo
spec:
  selector:
    matchLabels:
      app: argo-workflow-labeler
  template:
    metadata:
      labels:
        app: argo-workflow-labeler
        control-plane: argo-workflows
      annotations:
        traffic.sidecar.istio.io/excludeInboundPorts: "8080"
    spec:
      serviceAccountName: argo-workflow-labeler
      containers:
        - name: argo-workflow-labeler
          image: digitalbackbonecr.azurecr.io/argo-workflow-labeler:latest
          resources:
            requests:
              cpu: 50m
              memory: 100Mi
            limits:
              cpu: 50m
              memory: 100Mi
          ports:
            - containerPort: 8080
              name: webhook
          args:
            - -tls-cert-file=/etc/webhook-certs/tls.crt
            - -tls-key-file=/etc/webhook-certs/tls.key
          securityContext:
            privileged: false
            readOnlyRootFilesystem: true
            allowPrivilegeEscalation: false
            runAsNonRoot: true
            runAsUser: 1002
            runAsGroup: 1002
          volumeMounts:
            - mountPath: /etc/webhook-certs
              name: webhook-tls
      volumes:
        - name: webhook-tls
          secret:
            secretName: argo-workflow-labeler-tls

[stefansedich]

We ran into this issue and wanted a way for labels on templates to automagically get propogated from the template to the worklowMetadata/podMetadata.

Thought about it or a while and considered a PR to argo-workflows to add this support however I was able use Kyverno to create a mutating policy that would load the related template for a workflow and propogate all labels for us.

Ideally going forward our teams will set podMetadata/workflowMetadata but at least with this solution we can ensure the tags set only on templates today get set on the workflow pods themselves by default.

[FernandoArteaga]

using controller.workflowDefaults.podMetadata you can set labels or annotations to the Workflow Pod.

controller:
  workflowDefaults:
      podMetadata:
        labels:
          foo: "bar"

[kkrboyaa]

using controller.workflowDefaults.podMetadata you can set labels or annotations to the Workflow Pod.

controller:
  workflowDefaults:
      podMetadata:
        labels:
          foo: "bar"

This is doing exactly what I needed. After adding this now I have labels on all pods. If you want to have them both on the Workflow and on the Pods use a combination of podMetadata and metadata. Add this to your values.yaml:

controller:
  workflowDefaults:
    metadata:
      labels:
        foo: "bar"
    spec:
      podMetadata:
        labels:
          foo: "bar"