Open hadim opened 3 years ago
I think you can label your pods as follows. Look at examples/pod-metadata.yaml
Thanks. I am aware of this and should have mentioned it actually in my first message. I am looking for an automated way of doing it.
But maybe it's a too specific feature for Argo? (feel free to close if you think it is)
As a workaround, I'll use examples/pod-metadata.yaml
.
Would you like to submit a PR to add this feature?
I'll consider it for sure if the examples/pod-metadata.yaml
workaround becomes too cumbersome.
@alexec I am interested to contribute this.Can you assign this to me?
I started to look at various entry points in the code to integrate this. I found https://github.com/argoproj/argo/blob/91bce2574fab15f4fab4bc4df9e50563aa748838/workflow/controller/workflowpod.go#L131 to be a potentially good candidate to add the labels of the workflow spec on-the-fly during pod creation. Maybe @alexec would like to see this elsewhere?
Another question is do we want to force by default? or is it best to add an option in the configmap about it?
@arghya88 still interested working on that?
@hadim You can now define podMetadata at workflow level and then those metadata will also appear in pods. See https://github.com/argoproj/argo-workflows/blob/master/examples/pod-metadata-wf-field.yaml for an example. Is this what you need? Or are you looking for inheriting existing labels from workflow to pods?
If you need the latter, what is your use case? The workflow is the owner of your pods so you can also obtain the labels from the referenced workflow.
I was looking for a way to automatically set those labels from the labels of the workflow. But yeah I'll use podMetadata
if that's not something you think is worse adding.
The labels workflows.argoproj.io/completed
and workflows.argoproj.io/workflow
are there for other purposes. I am not sure if it's necessary to inherit other labels/metadata as well since they can be obtained easily by owner reference. @alexec WDYT?
You can use a MutatingWebhookConfiguration to add labels to pods that have a Workflow owner reference
// main.go
package main
import (
"context"
"flag"
"fmt"
"github.com/google/martian/log"
"github.com/sirupsen/logrus"
kwhhttp "github.com/slok/kubewebhook/v2/pkg/http"
kwhlogrus "github.com/slok/kubewebhook/v2/pkg/log/logrus"
kwhmodel "github.com/slok/kubewebhook/v2/pkg/model"
kwhmutating "github.com/slok/kubewebhook/v2/pkg/webhook/mutating"
corev1 "k8s.io/api/core/v1"
"k8s.io/apimachinery/pkg/apis/meta/v1"
"net/http"
"os"
"strings"
"time"
)
type config struct {
certFile string
keyFile string
}
func initFlags() *config {
cfg := &config{}
fl := flag.NewFlagSet(os.Args[0], flag.ExitOnError)
fl.StringVar(&cfg.certFile, "tls-cert-file", "", "TLS certificate file")
fl.StringVar(&cfg.keyFile, "tls-key-file", "", "TLS key file")
if err := fl.Parse(os.Args[1:]); err != nil {
err := fmt.Errorf("could not parse flags: %w", err)
log.Errorf(err.Error())
panic(err)
}
log.Infof("Using tls cert file at path: %s", cfg.certFile)
log.Infof("Using tls key file at path: %s", cfg.keyFile)
return cfg
}
func checkArgoWorkflowLabelNotPresent(pod *corev1.Pod) (*kwhmutating.MutatorResult, error) {
if pod.Labels == nil {
return &kwhmutating.MutatorResult{}, nil
}
if _, ok := pod.Labels["argo"]; ok {
return &kwhmutating.MutatorResult{}, fmt.Errorf("forbidden: pod cannot have label argo=workflow as it is not owned by an Argo Workflow")
}
return &kwhmutating.MutatorResult{}, nil
}
func run() error {
logrusLogEntry := logrus.NewEntry(logrus.New())
logrusLogEntry.Logger.SetLevel(logrus.DebugLevel)
logger := kwhlogrus.NewLogrus(logrusLogEntry)
logger.Infof("running...")
cfg := initFlags()
logger.Infof("creating mutator func")
var mf kwhmutating.MutatorFunc = func(ctx context.Context, review *kwhmodel.AdmissionReview, object v1.Object) (*kwhmutating.MutatorResult, error) {
pod, ok := object.(*corev1.Pod)
if !ok {
return &kwhmutating.MutatorResult{}, nil
}
if pod.OwnerReferences == nil || len(pod.OwnerReferences) == 0 {
return checkArgoWorkflowLabelNotPresent(pod)
}
var found = false
for _, reference := range pod.OwnerReferences {
if reference.Kind == "Workflow" && strings.Contains(reference.APIVersion, "argoproj.io/") {
found = true
break
}
}
if !found {
return checkArgoWorkflowLabelNotPresent(pod)
}
if pod.Labels == nil {
pod.Labels = make(map[string]string)
}
currentValue, ok := pod.Labels["argo"]
if ok && currentValue != "workflow" {
return &kwhmutating.MutatorResult{}, fmt.Errorf("invalid label value argo='%s'", currentValue)
}
pod.Labels["argo"] = "workflow"
return &kwhmutating.MutatorResult{MutatedObject: pod}, nil
}
logger.Infof("creating webhook config")
mcfg := kwhmutating.WebhookConfig{
ID: "argo-workflow-labeler",
Mutator: mf,
Logger: logger,
}
logger.Infof("creating webhook")
wh, err := kwhmutating.NewWebhook(mcfg)
if err != nil {
return fmt.Errorf("error creating webhook: %w", err)
}
logger.Infof("creating http handler")
whHandler, err := kwhhttp.HandlerFor(kwhhttp.HandlerConfig{Webhook: wh, Logger: logger})
if err != nil {
return fmt.Errorf("error creating webhook handler: %w", err)
}
// Serve.
logger.Infof("Listening on :8080")
err = http.ListenAndServeTLS(":8080", cfg.certFile, cfg.keyFile, whHandler)
if err != nil {
return fmt.Errorf("error serving webhook: %w", err)
}
return nil
}
func main() {
log.Infof("sleeping a bit...")
time.Sleep(5 * time.Second)
log.Infof("starting app")
err := run()
if err != nil {
log.Errorf("error running app: %v", err)
time.Sleep(time.Second)
os.Exit(1)
}
}
# Dockerfile
FROM golang:1.16 as builder
WORKDIR /workspace
COPY go.mod go.mod
COPY go.sum go.sum
RUN go mod download
COPY main.go main.go
RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 GO111MODULE=on go build -a -o argo-workflow-labeler main.go
FROM gcr.io/distroless/static:nonroot
COPY --from=builder /workspace/argo-workflow-labeler /argo-workflow-labeler
USER nonroot:nonroot
CMD ["/argo-workflow-labeler"]
# Makefile
VERSION ?= latest
IMAGE ?= my-container-registry/argo-workflow-labeler
# Build the docker image
pack:
docker build . -t ${IMAGE}:${VERSION}
# Push the docker image
push: pack
docker push ${IMAGE}:${VERSION}
# Certificate
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: argo-workflow-labeler-tls
namespace: argo
spec:
secretName: argo-workflow-labeler-tls
dnsNames:
- argo-workflow-labeler.argo.svc
- argo-workflow-labeler.argo.svc.cluster.local
commonName: argo-workflow-labeler.argo.svc
issuerRef:
name: selfsigned
---
# Issuer
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
name: selfsigned
namespace: argo
spec:
selfSigned: {}
---
# Service
apiVersion: v1
kind: Service
metadata:
name: argo-workflow-labeler
namespace: argo
spec:
ports:
- port: 443
targetPort: 8080
protocol: TCP
name: webhook
selector:
app: argo-workflow-labeler
---
# ServiceAccount
apiVersion: v1
kind: ServiceAccount
metadata:
name: argo-workflow-labeler
namespace: argo
---
# MutatingWebhookConfiguration
apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
name: argo-workflow-labeler
labels:
app: argo-workflow-labeler
kind: mutator
annotations:
cert-manager.io/inject-ca-from: argo/argo-workflow-labeler-tls
webhooks:
- name: argo-workflow-labeler.example.com
admissionReviewVersions:
- v1
sideEffects: NoneOnDryRun
clientConfig:
service:
port: 443
name: argo-workflow-labeler
namespace: argo
path: "/mutate"
objectSelector:
matchExpressions:
- key: app
operator: NotIn
values: [ argo-workflow-labeler ]
rules:
- operations: [ "CREATE" ]
apiGroups: [ "" ]
apiVersions: [ "v1" ]
resources: [ "pods" ]
---
# Deployment
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
name: argo-workflow-labeler
namespace: argo
spec:
host: argo-workflow-labeler.argo.svc.cluster.local
trafficPolicy:
tls:
mode: DISABLE
exportTo:
- "*"
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: argo-workflow-labeler
namespace: argo
spec:
selector:
matchLabels:
app: argo-workflow-labeler
template:
metadata:
labels:
app: argo-workflow-labeler
control-plane: argo-workflows
annotations:
traffic.sidecar.istio.io/excludeInboundPorts: "8080"
spec:
serviceAccountName: argo-workflow-labeler
containers:
- name: argo-workflow-labeler
image: digitalbackbonecr.azurecr.io/argo-workflow-labeler:latest
resources:
requests:
cpu: 50m
memory: 100Mi
limits:
cpu: 50m
memory: 100Mi
ports:
- containerPort: 8080
name: webhook
args:
- -tls-cert-file=/etc/webhook-certs/tls.crt
- -tls-key-file=/etc/webhook-certs/tls.key
securityContext:
privileged: false
readOnlyRootFilesystem: true
allowPrivilegeEscalation: false
runAsNonRoot: true
runAsUser: 1002
runAsGroup: 1002
volumeMounts:
- mountPath: /etc/webhook-certs
name: webhook-tls
volumes:
- name: webhook-tls
secret:
secretName: argo-workflow-labeler-tls
We ran into this issue and wanted a way for labels on templates to automagically get propogated from the template to the worklowMetadata/podMetadata.
Thought about it or a while and considered a PR to argo-workflows to add this support however I was able use Kyverno to create a mutating policy that would load the related template for a workflow and propogate all labels for us.
Ideally going forward our teams will set podMetadata/workflowMetadata but at least with this solution we can ensure the tags set only on templates today get set on the workflow pods themselves by default.
using controller.workflowDefaults.podMetadata
you can set labels
or annotations
to the Workflow Pod.
controller:
workflowDefaults:
podMetadata:
labels:
foo: "bar"
using
controller.workflowDefaults.podMetadata
you can setlabels
orannotations
to the Workflow Pod.controller: workflowDefaults: podMetadata: labels: foo: "bar"
This is doing exactly what I needed. After adding this now I have labels on all pods. If you want to have them both on the Workflow and on the Pods use a combination of podMetadata
and metadata
. Add this to your values.yaml
:
controller:
workflowDefaults:
metadata:
labels:
foo: "bar"
spec:
podMetadata:
labels:
foo: "bar"
Currently, the pods of a given workflow are labelled with
workflows.argoproj.io/completed
andworkflows.argoproj.io/workflow
. Could we also add the labels of the workflow to that list, something like: