haitang5445 commented 1 year ago

Describe the bug

VMware Aria Operations for Logs 8.12

old: projects.registry.vmware.com/vrealize_loginsight/fluentd:1.0.0 There is a 'tag' present

new: projects.registry.vmware.com/vrealize_loginsight/fluentd:1.4.0 There is not a 'tag' present

Reproduction steps

fluent.conf


<source>
@id in_tail_container_logs
@type tail
path /var/log/containers/*.log
pos_file /var/log/fluentd-containers.log.pos
tag raw.kubernetes.*
read_from_head true
<parse>
@type multi_format
<pattern>
  format json
  time_key time
  time_format %Y-%m-%dT%H:%M:%S.%NZ
</pattern>
<pattern>
  format /^(?<time>.+) (?<stream>stdout|stderr) [^ ]* (?<log>.*)$/
  time_format %Y-%m-%dT%H:%M:%S.%N%:z
</pattern>
</parse>
</source>

Detect exceptions in the log output and forward them as one log entry.

<match raw.kubernetes.**> @id raw.kubernetes @type detect_exceptions remove_tag_prefix raw message log stream stream multiline_flush_interval 5 max_bytes 500000 max_lines 1000

<filter kubernetes.**> @type record_transformer

environment tanzu_k8s_grid log_type kubernetes

watch false

Enriches records with Kubernetes metadata

<filter kubernetes.**> @id filter_kubernetes_metadata @type kubernetes_metadata watch false

<match **> @type vmware_loginsight scheme https ssl_verify false host port 9543 http_method post serializer json rate_limit_msec 0 raise_on_error true include_tag_key true tag_key tag http_conn_debug false

2. old loginsight-fluent.yml

apiVersion: v1 kind: ServiceAccount metadata: labels: k8s-app: fluentd-loginsight-logging name: fluentd-loginsight-logging namespace: kube-system

kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: fluentd-clusterrole rules:

apiGroups: [""] resources: ["namespaces", "pods"] verbs: ["list", "get", "watch"]

kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: fluentd-clusterrole roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: fluentd-clusterrole subjects:

kind: ServiceAccount name: fluentd-loginsight-logging namespace: kube-system

apiVersion: apps/v1 kind: DaemonSet metadata: name: fluentd-loginsight-logging namespace: kube-system labels: k8s-app: fluentd-loginsight-logging app: fluentd-loginsight-logging version: v1 kubernetes.io/cluster-service: "true" spec: selector: matchLabels: name: fluentd-loginsight-logging template: metadata: labels: name: fluentd-loginsight-logging app: fluentd-loginsight-logging version: v1 kubernetes.io/cluster-service: "true" spec: serviceAccount: fluentd-loginsight-logging serviceAccountName: fluentd-loginsight-logging tolerations:

key: node-role.kubernetes.io/master effect: NoSchedule containers:
name: fluentd-loginsight image: projects.registry.vmware.com/vrealize_loginsight/fluentd:1.0.0 command: ["fluentd", "-c", "/etc/fluentd/fluent.conf", "-p", "/fluentd/plugins"] env:
- name: FLUENTD_ARGS value: --no-supervisor -q resources: limits: memory: 500Mi requests: cpu: 100m memory: 200Mi volumeMounts:
- name: varlog mountPath: /var/log readOnly: false
- name: config-volume mountPath: /etc/fluentd readOnly: true volumes:
name: varlog hostPath: path: /var/log
name: config-volume configMap: name: loginsight-fluentd-config
```
3. new loginsight-fluent.yml
```
apiVersion: v1 kind: ServiceAccount metadata: labels: k8s-app: fluentd-loginsight-logging name: fluentd-loginsight-logging namespace: kube-system

kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: fluentd-clusterrole rules:

apiGroups: [""] resources: ["namespaces", "pods"] verbs: ["list", "get", "watch"]

kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: fluentd-clusterrole roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: fluentd-clusterrole subjects:

kind: ServiceAccount name: fluentd-loginsight-logging namespace: kube-system

apiVersion: apps/v1 kind: DaemonSet metadata: name: fluentd-loginsight-logging namespace: kube-system labels: k8s-app: fluentd-loginsight-logging app: fluentd-loginsight-logging version: v1 kubernetes.io/cluster-service: "true" spec: selector: matchLabels: name: fluentd-loginsight-logging template: metadata: labels: name: fluentd-loginsight-logging app: fluentd-loginsight-logging version: v1 kubernetes.io/cluster-service: "true" spec: serviceAccount: fluentd-loginsight-logging serviceAccountName: fluentd-loginsight-logging tolerations:

key: node-role.kubernetes.io/master effect: NoSchedule containers:
name: fluentd-loginsight image: projects.registry.vmware.com/vrealize_loginsight/fluentd:1.4.0 command: ["fluentd", "-c", "/etc/fluentd/fluent.conf", "-p", "/fluentd/plugins"] env:
- name: FLUENTD_ARGS value: --no-supervisor -q resources: limits: memory: 500Mi requests: cpu: 100m memory: 200Mi volumeMounts:
- name: varlog mountPath: /var/log readOnly: false
- name: config-volume mountPath: /etc/fluentd readOnly: true volumes:
name: varlog hostPath: path: /var/log

name: config-volume configMap: name: loginsight-fluentd-config


4.  Using the same fluent. conf, deploy two versions of vrealize_loginsight/fluentd separately
...

Expected behavior

1. Use old loginsight-fluent.yml

2. Use new loginsight-fluent.yml

There is not a 'tag' present

Additional context

No response

mohitevishal commented 1 year ago

@haitang5445 With introduction of Buffer support, the tag had to be removed. You can always inject tag into the logs using record_modifier or record_transformer plugins. Please check https://github.com/repeatedly/fluent-plugin-record-modifier Or if you need the tailed path from in_tail plugin, you can use path_key https://docs.fluentd.org/input/tail#path_key

haitang5445 commented 1 year ago

TKS !!!

new fluent.conf

# Prevent fluentd from handling records containing its own logs to handle cycles.
# [warn]: define <match fluent.**> to capture fluentd logs in top level is deprecated. Use <label @FLUENT_LOG> instead
<label @FLUENT_LOG>
  <match fluent.**>
    @type null
  </match>
</label>

<source>
  @id in_tail_container_logs
  @type tail
  path /var/log/containers/*.log
  pos_file /var/log/fluentd-containers.log.pos
  tag raw.kubernetes.*
  read_from_head true
  <parse>
    @type multi_format
    <pattern>
      format json
      time_key time
      time_format %Y-%m-%dT%H:%M:%S.%NZ
    </pattern>
    <pattern>
      format /^(?<time>.+) (?<stream>stdout|stderr) [^ ]* (?<log>.*)$/
      time_format %Y-%m-%dT%H:%M:%S.%N%:z
    </pattern>
  </parse>
</source>

<source>
  @type tail
  path /var/log/messages,/var/log/secure,/var/log/cron
  pos_file /var/log/fluentd-system.log.pos
  read_from_head true
  tag systemlog.*
  format none
  time_format %d/%b/%Y:%H:%M:%S %z
</source>

# Detect exceptions in the log output and forward them as one log entry.
<match raw.kubernetes.**>
  @id raw.kubernetes
  @type detect_exceptions
  remove_tag_prefix raw
  message log
  stream stream
  multiline_flush_interval 5
  max_bytes 500000
  max_lines 1000
</match>

<filter kubernetes.**>
  @type record_transformer
  <record>
  environment tanzu_k8s_grid
  log_type kubernetes
  tag kubernetes
  </record>
#  watch false   # [warn]: parameter 'watch' in <filter kubernetes.**>
</filter>

# Enriches records with Kubernetes metadata
<filter kubernetes.**>
  @id filter_kubernetes_metadata
  @type kubernetes_metadata
#  watch false  # [warn]: parameter 'watch' in <filter kubernetes.**>
</filter>

<filter systemlog.**>
  @type record_transformer
  <record>
  log_type systemlog
  tag ${tag}
  </record>
</filter>

<match **>
  @type vmware_loginsight
  scheme https
  ssl_verify false
  host  x.x.x.x
  port 9543
  http_method post
  serializer json
  rate_limit_msec 0
  raise_on_error true
 # include_tag_key true   # [warn]: parameter 'include_tag_key' in <match **>
 # tag_key tag            # [warn]: parameter 'tag_key' in <match **>
  http_conn_debug false
  <buffer>
    chunk_limit_records 300
    flush_interval 3s
    retry_max_times 3
  </buffer>
</match>

vmware / fluent-plugin-vmware-loginsight

Upgrade the loginsight fluent log plugin to 1.4, but the logs do not have "tag" tags #29