The notifications-controller logging level keeps in “info” for argocd, although the config map argocd-cmd-params-cm configurarion is set to "error".

[ferchdav]

The notifications-controller logging level keeps in “info” for argocd, although the config map argocd-cmd-params-cm configurarion is set to "error".

Could you please provide a fix?

We want to avoid the large amount of logs that are stored and cause a great cost.

Here the details:

- Doc link: https://argo-cd.readthedocs.io/en/stable/operator-manual/argocd-cmd-params-cm-yaml/

• argocd-cmd-params-cm configuration:
  • Argo CD Notifications Controller Properties notificationscontroller.log.level: “error” notificationscontroller.log.format: “json” notificationscontroller.selfservice.enabled: "false"

- Log example: { "insertId": "yl1x08uioffimr03", "jsonPayload": { "msg": "Trigger on-sync-status-unknown result: [{[0].6SzWb05EK-0v90hwjyytTbN7S6A [app-sync-status-unknown] false}]", "level": "info", "resource": "argocd/xxxxxx” }, "resource": { "type": "k8s_container", "labels": { "cluster_name": “cluster_name", "namespace_name": "argocd", "project_id": “npe", "location": "us-east3", "container_name": "notifications-controller", "pod_name": "argocd-notifications-controller-7kjsdhfkjshdf-sdfsdf4f" } }, "timestamp": "2024-07-xx”, "severity": "INFO", "labels": { "k8s-pod/pod-template-hash": “45sdfsdfsdf", .... }, "logName": "projects/npe/logs/stderr", "receiveTimestamp": "2024-07-xx” }

- Notifications Current code which I think causes the issue: The container args takes global.logging.level first:

• --loglevel={{ default .Values.global.logging.level .Values.notifications.logLevel }} https://github.com/argoproj/argo-helm/blob/main/charts/argo-cd/templates/argocd-notifications/deployment.yaml#L66 and the global logging level is set to a fix value “info” which determines the final result of the issue. https://github.com/argoproj/argo-helm/blob/main/charts/argo-cd/values.yaml#L71 (edited)

- Slack message: https://cloud-native.slack.com/archives/C01UKS2NKK3/p1721760075212969

[juwon8891]

@ferchdav I can't reproduce this issue. Can you tell me how to reproduce it in detail?

[ferchdav]

Hello @juwon8891

Yes, here the steps.:

1) Get ArgoCD installed on GCP GKE

2) According to documentation, configure the Config Map argocd-cmd-params-cm with the following data:

• Doc link: https://argo-cd.readthedocs.io/en/stable/operator-manual/argocd-cmd-params-cm-yaml/

Argo CD Notifications Controller Properties

notificationscontroller.log.level: “error” notificationscontroller.log.format: “json” notificationscontroller.selfservice.enabled: "false"

• Config Map:

apiVersion: v1 kind: ConfigMap metadata: name: argocd-cmd-params-cm namespace: argocd labels: app.kubernetes.io/component: server app.kubernetes.io/instance: argocd app.kubernetes.io/managed-by: Helm app.kubernetes.io/name: argocd-cmd-params-cm app.kubernetes.io/part-of: argocd app.kubernetes.io/version: v2.11.4 helm.sh/chart: argo-cd-6.9.0 data: applicationsetcontroller.log.format: json applicationsetcontroller.log.level: error applicationsetcontroller.policy: sync controller.log.format: json controller.log.level: error notificationscontroller.log.format: json notificationscontroller.log.level: error notificationscontroller.selfservice.enabled: "false" reposerver.log.format: json reposerver.log.level: error server.log.format: json server.log.level: error

3) Restart the ArgoCD components. kubectl rollout restart sts/argocd-application-controller -n argocd kubectl rollout restart deploy/argocd-applicationset-controller -n argocd kubectl rollout restart deploy/argocd-dex-server -n argocd kubectl rollout restart deploy/argocd-notifications-controller -n argocd kubectl rollout restart deploy/argocd-redis-ha-haproxy -n argocd kubectl rollout restart sts/argocd-redis-ha-server -n argocd kubectl rollout restart deploy/argocd-repo-server -n argocd kubectl rollout restart deploy/argocd-server -n argocd

4) Review the GCP logs where will see logs like the one copied below where you will get the INFO loggging level information due to the default details settings:

{ "insertId": "yl1x08uioffimr03", "jsonPayload": { "msg": "Trigger on-sync-status-unknown result: [{[0].6SzWb05EK-0v90hwjyytTbN7S6A [app-sync-status-unknown] false}]", "level": "info", "resource": "argocd/xxxxxx” }, "resource": { "type": "k8s_container", "labels": { "cluster_name": “cluster_name", "namespace_name": "argocd", "project_id": “npe", "location": "us-east3", "container_name": "notifications-controller", "pod_name": "argocd-notifications-controller-7kjsdhfkjshdf-sdfsdf4f" } }, "timestamp": "2024-07-xx”, "severity": "INFO", "labels": { "k8s-pod/pod-template-hash": “45sdfsdfsdf", .... }, "logName": "projects/npe/logs/stderr", "receiveTimestamp": "2024-07-xx” }

5) I think the current Notifications code here is causing the issue:

The container args takes global.logging.level first:

--loglevel={{ default .Values.global.logging.level .Values.notifications.logLevel }} https://github.com/argoproj/argo-helm/blob/main/charts/argo-cd/templates/argocd-notifications/deployment.yaml#L66

and the global logging level is set to a fix value “info” which determines the final result of the issue. https://github.com/argoproj/argo-helm/blob/main/charts/argo-cd/values.yaml#L71 (edited)

[juwon8891]

Sorry I can't reproduce this issue. What environment does this message appear in? Trigger on-sync-status-unknow result: [{[0].6SzWb05EK-0v90hwjytytTbN7S6A [app-sync-status-unknow]

[ferchdav]

@juwon8891 Have you set your ArgoCD on GCP GKE? If so, you will see that despite having the configmap argocd-cmd-params-cm configured with error, all logs with INFO (default value in https://github.com/argoproj/argo-helm/blob/main/charts/argo-cd/values.yaml#L71) are being saved on the GCP Logging console from the notification controller causing a lot of expense because of unnecessary information. I hope you have all the information you need now. Please tell me if you don't.

[ferchdav]

@juwon8891 PFA logging_notification_controller.png with details. You will see that the notification-controller is sending logs with level error, debug or info despite error was set in the configmap

The logging level configured for the rest of the argocd components are working OK

[juwon8891]

I still can't find the log in the gke environment.. how about actually checking the container log with the kubectl command?

[ferchdav]

@juwon8891 It's seems you are getting no logs in gcp from your argocd instance. Try any of these 2 options:

1) resource.labels.namespace_name="argocd" labels.k8s-pod/app_kubernetes_io/name="argocd-notifications-controller"

or just:

2) resource.labels.namespace_name="argocd"

and let me know please.

[juwon8891]

I can still only get the "error" log. If possible, could you check the container log?

[ferchdav]

@juwon8891

Yes, here the container logs from 1 of the pods where you see level info, warning, etc

and here the configmap conf:

[juwon8891]

Did you restart the notification controller pod after you set up the configmap?

[ferchdav]

Hello @juwon8891 Yes, I've restarted all pieces.