Missing Capability of Grouping logs by namespace

[manikantapallapothu1]

Issue: openshift-logging 6.0.1 Creating logstream per each pod in namespace instead of grouping per namespace the capability openshift-logging 5.x

Environment

• Versions of OpenShift : 4.14.37
• Cluster Logging : 6.0.1
• and any other relevant components: loki 6.0.1

Ask: Ability to add groupBy: namespaceName to get ability of groupng logstreams.

[jcantrill]

Please provide more detail and possibly an example of what you are requesting. I don't understand the issue.

[manikantapallapothu1]

Example ClusterLogForwarder for logging version 5.7

apiVersion: logging.openshift.io/v1
kind: ClusterLogForwarder
metadata:
  name: instance
  namespace: openshift-logging
spec:
  inputs:
    - application:
        namespaces:
          - spoke-logforwarder1
      name: spoke-logforwarder1-logs
  outputs:
    - cloudwatch:
        groupBy: namespaceName # GroupBy defines the strategy for grouping logstreams
        groupPrefix: clusterName
        region: us-east-1
      name: spoke-logforwarder1-team
      secret:
        name: cw-sts-crdentials
      type: cloudwatch
  pipelines:
    - inputRefs:
        - spoke-logforwarder1-logs
      name: spoke-logforwarder1
      outputRefs:
        - spoke-logforwarder1-team
    - inputRefs:
        - application
        - audit
        - infrastructure
      name: app-default
      outputRefs:
        - default

Example ClusterLogForwarder with 6.x:

apiVersion: observability.openshift.io/v1
kind: ClusterLogForwarder
metadata:
  name: collector
  namespace: openshift-logging
spec:
  collector:
    resources:
      limits:
        memory: 3Gi
    tolerations:
      - effect: NoSchedule
        key: node-role.kubernetes.io/infra
        operator: Exists
  filters:
    - name: detectexception
      type: detectMultilineException
    - name: parse-json
      type: parse
  managementState: Managed
  outputs:
    - lokiStack:
        authentication:
          token:
            from: serviceAccount
        target:
          name: logging-loki
          namespace: openshift-logging
      name: default
      tls:
        ca:
          configMapName: openshift-service-ca.crt
          key: service-ca.crt
      type: lokiStack
    - cloudwatch:
        authentication:
          iamRole:
            roleARN:
              key: roleARN
              secretName: cloudwatch-credentials
            token:
              from: serviceAccount
          type: iamRole
        groupName: clustername-application-logs
        region: us-east-1
        tuning:
          deliveryMode: AtLeastOnce
          maxRetryDuration: 60
          maxWrite: 1000
          minRetryDuration: 5
      name: cw-app
      type: cloudwatch
    - cloudwatch:
        authentication:
          iamRole:
            roleARN:
              key: roleARN
              secretName: cloudwatch-credentials
            token:
              from: serviceAccount
          type: iamRole
        groupName: clustername-infrastructure-logs
        region: us-east-1
      name: cw-infra
      type: cloudwatch
    - cloudwatch:
        authentication:
          iamRole:
            roleARN:
              key: roleARN
              secretName: cloudwatch-credentials
            token:
              from: serviceAccount
          type: iamRole
        groupName: clustername-audit-logs
        region: us-east-1
      name: cw-audit
      type: cloudwatch
  pipelines:
    - inputRefs:
        - application
      name: logs-to-cw-app
      outputRefs:
        - cw-app
    - inputRefs:
        - infrastructure
      name: logs-to-cw-infra
      outputRefs:
        - cw-infra
    - inputRefs:
        - audit
      name: logs-to-cw-audit
      outputRefs:
        - cw-audit
    - inputRefs:
        - application
        - infrastructure
        - audit
      name: logs-to-loki
      outputRefs:
        - default
  serviceAccount:
    name: collector

[manikantapallapothu1]

with 5.x version of clusterLogForwarder if I use below settings

        groupBy: namespaceName # GroupBy defines the strategy for grouping logstreams
        groupPrefix: clusterName

I get to see cloudwatch loggroup created per each application namespace.

    - cloudwatch:
        groupBy: logType
        groupPrefix: clusterName

with above ^^ setting, I see one cloudwatchLoggroup and logstream per pod.

just curious if there is a possibility to merge these two capabilities and create one LogGroup per LogType and LogStream per namespace inside that LogGroup. instead of logstream per pod with future releases of logging operator version 6.0.