Terraform hook removes all collector processing rule filters when managed with argo-cd

[amitlin]

Describe the bug A clear and concise description of what the bug is. The setup job that creates the collectors removes all processing filter rules when it is ran.

Logs Logs which can be helpful in investigating the issue.

sumologic-collector-su-setup-7sbxd setup
sumologic-collector-su-setup-7sbxd setup Terraform used the selected providers to generate the following execution
sumologic-collector-su-setup-7sbxd setup plan. Resource actions are indicated with the following symbols:
sumologic-collector-su-setup-7sbxd setup   ~ update in-place
sumologic-collector-su-setup-7sbxd setup
sumologic-collector-su-setup-7sbxd setup Terraform will perform the following actions:
sumologic-collector-su-setup-7sbxd setup
sumologic-collector-su-setup-7sbxd setup   # kubernetes_secret.sumologic_collection_secret will be updated in-place
sumologic-collector-su-setup-7sbxd setup   ~ resource "kubernetes_secret" "sumologic_collection_secret" {
sumologic-collector-su-setup-7sbxd setup         id                             = "sumologic-collector/sumologic"
sumologic-collector-su-setup-7sbxd setup       + wait_for_service_account_token = true
sumologic-collector-su-setup-7sbxd setup         # (3 unchanged attributes hidden)
sumologic-collector-su-setup-7sbxd setup
sumologic-collector-su-setup-7sbxd setup       - timeouts {}
sumologic-collector-su-setup-7sbxd setup
sumologic-collector-su-setup-7sbxd setup         # (1 unchanged block hidden)
sumologic-collector-su-setup-7sbxd setup     }
sumologic-collector-su-setup-7sbxd setup
sumologic-collector-su-setup-7sbxd setup   # sumologic_http_source.default_logs_source will be updated in-place
sumologic-collector-su-setup-7sbxd setup   ~ resource "sumologic_http_source" "default_logs_source" {
sumologic-collector-su-setup-7sbxd setup         id                           = "xxxxxxx"
sumologic-collector-su-setup-7sbxd setup         name                         = "logs"
sumologic-collector-su-setup-7sbxd setup         # (9 unchanged attributes hidden)
sumologic-collector-su-setup-7sbxd setup
sumologic-collector-su-setup-7sbxd setup       - filters {
sumologic-collector-su-setup-7sbxd setup           - filter_type = "Exclude" -> null
sumologic-collector-su-setup-7sbxd setup           - name        = "test" -> null
sumologic-collector-su-setup-7sbxd setup           - regexp      = "test" -> null
sumologic-collector-su-setup-7sbxd setup         }
sumologic-collector-su-setup-7sbxd setup
sumologic-collector-su-setup-7sbxd setup         # (1 unchanged block hidden)
sumologic-collector-su-setup-7sbxd setup     }
sumologic-collector-su-setup-7sbxd setup
sumologic-collector-su-setup-7sbxd setup Plan: 0 to add, 2 to change, 0 to destroy.

Command used to install/upgrade Collection Direct command which you used to install/upgrade collection. The collector is managed by Argocd, the action used was a simple argocd sync. The equivalent command will be a simple helm upgrade

Configuration Configuration used for Collection e.g. user-values.yaml for helm. mostly defaults, just the collector name & cluster set.

sumologic:
  accessId: "<accessId>"
  accessKey: "<accessKey>"
  clusterName: "staging"
  collectorName: "staging-collector"

To Reproduce Steps to reproduce the behavior. Create an Argo application that manages a sumo collector chart with the following values:

sumologic:
  accessId: "<accessId>"
  accessKey: "<accessKey>"
  clusterName: "staging"
  collectorName: "staging-collector"

Add any processing filter rule. Sync the argo application.

Expected behavior A clear and concise description of what you expected to happen. Either the ability to set these rules as code through configuration, or the terraform resource should ignore changes to the processing rules.

Environment (please complete the following information):

• Collection version (e.g. helm ls -n sumologic): sumologic-4.5.1
• Kubernetes version (e.g. kubectl version): Server Version: v1.26.12-eks-5e0fdde
• Cloud provider: AWS

Anything else do we need to know Add any other context about the problem here, e.g. issue happens only occasionally.

[sumo-drosiek]

Sorry for the delay :/

Setup job is applying the terraform changes for collector every time, so if you change the collector configuration by hand it will be overwritten by setup job. You may use additional configuration in helm chart to set the processing rules

It can be done the same way we are setting default_date_properties

sumologic:
  collector:
    sources:
      logs:
        default:
          ## Properties can be used to extend default settings, such as processing rules, fields etc
          properties:
            default_date_formats:
              ## Ensures that timestamp key has precedence over timestamp auto discovery
              - format: epoch
                locator: '\"timestamp\":(\\d+)'

Here is documentation for terraform for filters:

https://registry.terraform.io/providers/SumoLogic/sumologic/latest/docs#filters

Please reply if you need more help with the issue