search

kubernetes / ingress-nginx

Ingress NGINX Controller for Kubernetes
https://kubernetes.github.io/ingress-nginx/
Apache License 2.0
17.56k stars 8.27k forks source link

Nginx ingress not honoring sample rate flags for datadog's tracer configuration #7048

Closed audip closed 3 years ago

audip commented 3 years ago

NGINX Ingress controller version: 0.34.1

Kubernetes version (use kubectl version): 1.18.15

Environment:

What happened:

100% of datadog traces were being sent to datadog, so we tried to configure sampling rate to 1% using these two flags: datadog-priority-sampling set to false and datadog-sample-rate to 0.01.

Despite of that change, datadog continues to report 100% trace ingestion and nginx-ingress is picking up the default configuration values instead of the value being set in the configmap (reference: https://github.com/kubernetes/ingress-nginx/blob/controller-v0.34.1/internal/ingress/controller/config/config.go#L814-L815)

What you expected to happen:

Nginx ingress should load the new configuration when configmap has updated values for datadog opentracing module

How to reproduce it:

Install minikube/kind

Install the ingress controller

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/provider/baremetal/deploy.yaml

Setup a configmap to have similar values

apiVersion: v1
kind: ConfigMap
metadata:
  labels:
    helm.sh/chart: ingress-nginx-3.27.0
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 0.45.0
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: controller
  name: ingress-nginx-controller
  namespace: ingress-nginx
data:
  enable-opentracing: "true"
  datadog-collector-host: "127.0.0.1"
  datadog-collector-port: "8130"
  datadog-service-name: "ingress-nginx-controller"
  datadog-priority-sampling: "false"
  datadog-sample-rate: "0.01"

Install datadog agent with configuration

---
kind: ServiceAccount
apiVersion: v1
metadata:
  name: datadog
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: datadog
rules:
  - apiGroups:
    - ""
    resources:
    - services
    - events
    - endpoints
    - pods
    - nodes
    - componentstatuses
    verbs:
    - get
    - list
    - watch
  - apiGroups:
    - ""
    resources:
    - configmaps
    resourceNames:
    - datadogtoken             # Kubernetes event collection state
    - datadog-leader-election  # Leader election token
    verbs:
    - get
    - update
  - apiGroups:  # To create the leader election token
    - ""
    resources:
    - configmaps
    verbs:
    - create
  - nonResourceURLs:
    - "/version"
    - "/healthz"
    - "/metrics"
    verbs:
    - get
  - apiGroups:  # Kubelet connectivity
    - ""
    resources:
    - nodes/metrics
    - nodes/spec
    - nodes/proxy
    - nodes/stats
    verbs:
    - get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: datadog
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: datadog
subjects:
- kind: ServiceAccount
  name: datadog
  namespace: default
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: datadog-agent
spec:
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 25%
  selector:
    matchLabels:
      app: datadog-agent
  template:
    metadata:
      labels:
        app: datadog-agent
      name: datadog-agent
      annotations:
        container.apparmor.security.beta.kubernetes.io/datadog-system-probe: unconfined
        container.seccomp.security.alpha.kubernetes.io/system-probe: localhost/system-probe
    spec:
      serviceAccountName: datadog
      containers:
      - image: datadog/agent:7.25.1
        imagePullPolicy: IfNotPresent
        name: datadog-agent
        ports:
          - containerPort: 8125
            # Custom metrics via DogStatsD - uncomment this section to enable custom metrics collection
            hostPort: 8125
            name: dogstatsdport
            protocol: UDP
          - containerPort: 8130
            # Trace Collection (APM) - uncomment this section to enable APM
            hostPort: 8130
            name: traceport
            protocol: TCP
        env:
          - name: DD_KUBERNETES_KUBELET_HOST
            valueFrom:
              fieldRef:
                fieldPath: status.hostIP
          - name: DD_API_KEY
            value: "<datadog-api-key>"
          - name: KUBERNETES
            value: "true"
          - name: DD_APM_ENABLED
            value: "true"
          - name: DD_APM_NON_LOCAL_TRAFFIC
            value: "true"
          - name: DD_APM_RECEIVER_PORT
            value: "8130"
          - name: DD_DOGSTATSD_NON_LOCAL_TRAFFIC
            value: "true"
        volumeMounts:
          - name: config-kubernetesstate
            mountPath: /etc/datadog-agent/conf.d/kubernetes_state.d
          - name: dockersocket
            mountPath: /var/run/docker.sock
          - name: procdir
            mountPath: /host/proc
            readOnly: true
          - name: cgroups
            mountPath: /host/sys/fs/cgroup
            readOnly: true
          - name: pointerdir
            mountPath: /opt/datadog-agent/run
          - name: debugfs
            mountPath: /sys/kernel/debug
          - name: s6-run
            mountPath: /var/run/s6
        livenessProbe:
          exec:
            command:
            - ./probe.sh
          initialDelaySeconds: 15
          periodSeconds: 5
      - name: datadog-system-probe
        image: datadog/agent:7.25.1
        imagePullPolicy: IfNotPresent
        securityContext:
          capabilities:
            add: ["SYS_ADMIN", "SYS_RESOURCE", "SYS_PTRACE", "NET_ADMIN", "NET_BROADCAST", "IPC_LOCK"]
        command:
          - /opt/datadog-agent/embedded/bin/system-probe
        env:
          - name: DD_SYSTEM_PROBE_ENABLED
            value: "true"
          - name: DD_SYSPROBE_SOCKET
            value: "/var/run/s6/sysprobe.sock"
        volumeMounts:
          - name: procdir
            mountPath: /host/proc
            readOnly: true
          - name: cgroups
            mountPath: /host/sys/fs/cgroup
            readOnly: true
          - name: debugfs
            mountPath: /sys/kernel/debug
          - name: s6-run
            mountPath: /var/run/s6
      volumes:
        - name: config-kubernetesstate
          emptyDir: {}
        - hostPath:
            path: /var/run/docker.sock
          name: dockersocket
        - hostPath:
            path: /proc
          name: procdir
        - hostPath:
            path: /sys/fs/cgroup
          name: cgroups
        - hostPath:
            path: /opt/datadog-agent/run
          name: pointerdir
        - name: s6-run
          emptyDir: {}
        - name: debugfs
          hostPath:
            path: /sys/kernel/debug
      tolerations:
      - operator: Exists
---
apiVersion: v1
kind: Service
metadata:
  name: datadog-agent
spec:
  ports:
  - name: metrics
    port: 8125
    protocol: UDP
    targetPort: dogstatsdport
  - name: traces
    port: 8130
    protocol: TCP
    targetPort: traceport
  selector:
    app: datadog-agent

Install an application that will act as default backend (is just an echo app)

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/docs/examples/http-svc.yaml

Create an ingress (please add any additional annotation required)

  apiVersion: networking.k8s.io/v1beta1
  kind: Ingress
  metadata:
    name: foo-bar
  spec:
    rules:
    - host: foo.bar
      http:
        paths:
        - backend:
            serviceName: http-svc
            servicePort: 80
          path: /
" | kubectl apply -f -

make a request

kubectl exec -it -n ingress-nginx $POD_NAME -- curl -H 'Host: foo.bar' localhost

--->

Anything else we need to know:

Locally, I am able to reproduce it with 0.44.0 version

/kind bug

longwuyuan commented 3 years ago

/remove-kind bug /triage needs-information

datadog-collector-host: datadog-agent.default.svc.cluster.local

  1. Can you show the ConfigMap as is in the cluster state using the command kubectl -n <namespace> get cm <configmapname> -o yaml
  2. What happens if you use fqdn instead of ipaddress for datadog-collector-host
k8s-triage-robot commented 3 years ago

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community. /lifecycle stale

k8s-triage-robot commented 3 years ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

k8s-triage-robot commented 3 years ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

k8s-ci-robot commented 3 years ago

@k8s-triage-robot: Closing this issue.

In response to [this](https://github.com/kubernetes/ingress-nginx/issues/7048#issuecomment-927021199): >The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. > >This bot triages issues and PRs according to the following rules: >- After 90d of inactivity, `lifecycle/stale` is applied >- After 30d of inactivity since `lifecycle/stale` was applied, `lifecycle/rotten` is applied >- After 30d of inactivity since `lifecycle/rotten` was applied, the issue is closed > >You can: >- Reopen this issue or PR with `/reopen` >- Mark this issue or PR as fresh with `/remove-lifecycle rotten` >- Offer to help out with [Issue Triage][1] > >Please send feedback to sig-contributor-experience at [kubernetes/community](https://github.com/kubernetes/community). > >/close > >[1]: https://www.kubernetes.dev/docs/guide/issue-triage/ Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.