search

fluent / fluentd

Fluentd: Unified Logging Layer (project under CNCF)
https://www.fluentd.org
Apache License 2.0
12.81k stars 1.34k forks source link

Json parser can not apply `null_value_pattern` to nested values #4222

Open andrew-pickin-epi opened 1 year ago

andrew-pickin-epi commented 1 year ago

Describe the bug

Json parser not respecting null_value_pattern for nested values

To Reproduce

https://github.com/fluent/fluentd/issues/4222#issuecomment-1627874181

old ``` {msg: "foo", "ConfigMap/logging/fluentd-config":"unchanged","ConfigMap/logging/index-template":"unchanged"} ``` `null_value_pattern unchanged` should produce: ``` {msg: "foo"} ```

Expected behavior

Field with value unchanged should be discarded.

Your Environment

- Fluentd version: v1.15.3-debian
- TD Agent version: n/a
- Operating system:
- Kernel version: AWS EKS

Your Configuration

<filter kubernetes.log.flux-system.**>
  @id parser_flux
  @type parser
  key_name "log"
  reserve_data true
  remove_key_name_field true
  inject_key_prefix "log_"
  emit_invalid_record_to_error false
  <parse>
    @type "json"
    null_value_pattern unchanged
  </parse>
</filter>

### Your Error Log

```shell
No errors

Additional context

I'm trying to discard a large number of fields (list is dynamic) with anything of value unchanged. alternative approaches welcome.

daipom commented 1 year ago

null_value_pattern unchanged should produce:

{msg: "foo"}

Expected behavior

Field with value unchanged should be discarded.

No. The expected result is as follows.

{"msg":"foo","ConfigMap/logging/fluentd-config":null,"ConfigMap/logging/index-template":null}

We use null_value_pattern to recognize some patterns as NULL value. It is not to discard keys.

andrew-pickin-epi commented 1 year ago

{"msg":"foo","ConfigMap/logging/fluentd-config":null,"ConfigMap/logging/index-template":null}

Ok, this isn't my desired result, but this doesn't happen either.

daipom commented 1 year ago

What happens actually? Please share the actual result.

andrew-pickin-epi commented 1 year ago

Sorry, hadn't realised I hadn't posted that. Output is unchanged (literally):

{msg: "foo", "ConfigMap/logging/fluentd-config":"unchanged","ConfigMap/logging/index-template":"unchanged"}

As I said, what I'd like is {msg: "foo}" and suggestions as to how this can be achieved would be very welcome.

andrew-pickin-epi commented 1 year ago

Actually, this needs more qualification. input

{"error":null,"level":"info","msg":"Main container completed","time":"2023-06-30T16:43:53.152Z"}

and config

  @type parser
  key_name "log"
  reserve_time true
  reserve_data true
  remove_key_name_field true
  inject_key_prefix "log_"
  emit_invalid_record_to_error false
  <parse>
    @type "json"
  </parse>
Gives: key value
log_level info
log_msg Main container completed

Note: no log_error, so a null value doesn't create a field.

So {"msg":"foo","ConfigMap/logging/fluentd-config":null,"ConfigMap/logging/index-template":null}

With the above config with null_value_pattern unchanged, would expect log_msg only.

daipom commented 1 year ago

Note: no log_error, so a null value doesn't create a field.

I think it is not the specification of Parser plugin. Some other settings may discard the key o the value null, such as Formatter of Output plugin.

I can't reproduce it.

config:

<source>
  @type tail
  tag test
  path /path/to/test.log
  read_from_head true
  <parse>
    @type none
  </parse>
</source>

<filter test.**>
  @type parser
  key_name message
  reserve_data true
  remove_key_name_field true
  inject_key_prefix log_
  emit_invalid_record_to_error false
  <parse>
    @type json
  </parse>
</filter>

<match test.**>
  @type stdout
</match>

/path/to/test.log:

{"error":null,"level":"info","msg":"Main container completed","time":"2023-06-30T16:43:53.152Z"}

Result:

2023-07-01 11:26:45 +0900 [info]: #0 starting fluentd worker pid=32125 ppid=32121 worker=0
2023-07-01 11:26:45 +0900 [info]: #0 following tail of /path/to/test.log
1970-01-01 09:33:43.000000152 +0900 test: {"log_error":null,"log_level":"info","log_msg":"Main container completed"}
2023-07-01 11:26:45 +0900 [info]: #0 fluentd worker is now running worker=0
daipom commented 1 year ago

In addition, the following setting with null_value_pattern works as expected.

<source>
  @type tail
  tag test
  path /path/to/test.log
  read_from_head true
  <parse>
    @type none
  </parse>
</source>

<filter test.**>
  @type parser
  key_name message
  reserve_data true
  remove_key_name_field true
  inject_key_prefix log_
  emit_invalid_record_to_error false
  <parse>
    @type json
    null_value_pattern unchanged
  </parse>
</filter>

<match test.**>
  @type stdout
</match>

/path/to/test.log:

{"error":null,"level":"info","msg":"unchanged","time":"2023-06-30T16:43:53.152Z"}

Result:

2023-07-01 11:35:20 +0900 [info]: #0 starting fluentd worker pid=32806 ppid=32802 worker=0
2023-07-01 11:35:20 +0900 [info]: #0 following tail of /path/to/test.log
1970-01-01 09:33:43.000000152 +0900 test: {"log_error":null,"log_level":"info","log_msg":null}
2023-07-01 11:35:20 +0900 [info]: #0 fluentd worker is now running worker=0
daipom commented 1 year ago

Please check this behavior in your environment, and please find out what is causing that behavior. Since you are able to discard nulls (I'm not sure which setting is working though...), perhaps you can fix some configs and it will work.

If you find other settings that seem to be the cause, you can ask us again! I don't think this is a bug, so perhaps you should ask your question here:

andrew-pickin-epi commented 1 year ago

Your example isn't quite the same, I have multiple instances I want to be null and more complex field names. Irrespective of the format output, the json is being being parsed and yet I have multiple field that aren't being nulled .

daipom commented 1 year ago

I just wanted to say I think this is not a bug of Parser, and you can confirm it by this example.

andrew-pickin-epi commented 1 year ago

I understand it works in your example, but the same config doesn't work consistently. I'm suggesting your example is too simplistic. My config:

  <parse>
    @type "json"
    null_value_pattern unchanged
  </parse>

Still results in fields with values unchanged:

{"log_msg":"foo","log_ConfigMap/logging/fluentd-config":"unchanged","log_ConfigMap/logging/index-template":"unchanged"}

Even thought it's going through the json parser.

daipom commented 1 year ago

Could you summarize how to reproduce that in this way: https://github.com/fluent/fluentd/issues/4222#issuecomment-1615381247? It is important to summarize and share the general reproduction way.

andrew-pickin-epi commented 1 year ago

This is my real-world example.

Log from pod, specifically fluxcd/flux kustomize controller:

{"level":"info","ts":"2023-06-29T14:48:48.229Z","msg":"server-side apply completed","controller":"kustomization","controllerGroup":"kustomize.toolkit.fluxcd.io","controllerKind":"Kustomization","Kustomization":{"name":"eks-cluster-konf","namespace":"flux-system"},"namespace":"flux-system","name":"eks-cluster-konf","reconcileID":"06ad0250-3519-4e0a-8889-ffe3d48dd2cf","output":{"ConfigMap/cbd-production/cbd-api-k6bk5m8d82":"unchanged","ConfigMap/cbd-production/cbd-api-logback-config-h97k79tk9d":"unchanged","ConfigMap/cbd-production/cbd-app-hd2b762mk2":"unchanged","ConfigMap/cbd-production/cbd-podium-config-mt9m5dhkk7":"unchanged","ConfigMap/cbd-production/cbd-podium-fuseki-config":"unchanged","ConfigMap/cbd-production/cbd-podium-logback-config-g5fk2fb2kk":"unchanged","ConfigMap/cbd-production/cbd-podium-redis-config":"unchanged","ConfigMap/cbd-production/data-consumer-config-td9bm259b5":"unchanged","ConfigMap/cbd-testing/cbd-api-k6bk5m8d82":"unchanged","ConfigMap/cbd-testing/cbd-api-logback-config-h97k79tk9d":"unchanged","ConfigMap/cbd-testing/cbd-app-hd2b762mk2":"unchanged","ConfigMap/cbd-testing/cbd-podium-config-d625d942gc":"unchanged","ConfigMap/cbd-testing/cbd-podium-fuseki-config":"unchanged","ConfigMap/cbd-testing/cbd-podium-logback-config-g5fk2fb2kk":"unchanged","ConfigMap/cbd-testing/cbd-podium-redis-config":"unchanged","ConfigMap/cbd-testing/data-consumer-config-48m8dct24b":"unchanged","ConfigMap/registry-dev/build-script":"unchanged","ConfigMap/registry-dev/registry-config":"unchanged","ConfigMap/registry-dev/registry-scripts":"unchanged","ConfigMap/registry-dev/tdb-config":"unchanged","ConfigMap/registry-prod/build-script":"unchanged","ConfigMap/registry-prod/registry-config":"unchanged","ConfigMap/registry-prod/registry-scripts":"unchanged","ConfigMap/registry-prod/tdb-config":"unchanged","ConfigMap/registry-staging/build-script":"unchanged","ConfigMap/registry-staging/registry-config":"unchanged","ConfigMap/registry-staging/registry-scripts":"unchanged","ConfigMap/registry-staging/tdb-config":"unchanged","ConfigMap/rp-production/data-consumer-config-bd9tft494k":"unchanged","ConfigMap/rp-production/rp-api-996f76g9m9":"unchanged","ConfigMap/rp-production/rp-api-logback-config-h97k79tk9d":"unchanged","ConfigMap/rp-production/rp-app-8mgt7mdkg4":"unchanged","ConfigMap/rp-production/rp-podium-config-t5457b6kdk":"unchanged","ConfigMap/rp-production/rp-podium-fuseki-config":"unchanged","ConfigMap/rp-production/rp-podium-logback-config-g5fk2fb2kk":"unchanged","ConfigMap/rp-production/rp-podium-redis-config":"unchanged","ConfigMap/rp-testing/data-consumer-config-k2h5hbtf92":"unchanged","ConfigMap/rp-testing/rp-api-996f76g9m9":"unchanged","ConfigMap/rp-testing/rp-api-logback-config-h97k79tk9d":"unchanged","ConfigMap/rp-testing/rp-app-8mgt7mdkg4":"unchanged","ConfigMap/rp-testing/rp-podium-config-844888k8tt":"unchanged","ConfigMap/rp-testing/rp-podium-fuseki-config":"unchanged","ConfigMap/rp-testing/rp-podium-logback-config-g5fk2fb2kk":"unchanged","ConfigMap/rp-testing/rp-podium-redis-config":"unchanged","ConfigMap/rp-uat/data-consumer-config-7dbm4dt49f":"unchanged","ConfigMap/rp-uat/rp-api-996f76g9m9":"unchanged","ConfigMap/rp-uat/rp-api-logback-config-h97k79tk9d":"unchanged","ConfigMap/rp-uat/rp-app-8mgt7mdkg4":"unchanged","ConfigMap/rp-uat/rp-podium-config-69kh6f6749":"unchanged","ConfigMap/rp-uat/rp-podium-fuseki-config":"unchanged","ConfigMap/rp-uat/rp-podium-logback-config-g5fk2fb2kk":"unchanged","ConfigMap/rp-uat/rp-podium-redis-config":"unchanged","ConfigMap/rpa-dev/data-consumer-config-4f688ht5t2":"unchanged","ConfigMap/rpa-dev/rpa-api-dcgb7khchd":"unchanged","ConfigMap/rpa-dev/rpa-api-logback-config-h97k79tk9d":"unchanged","ConfigMap/rpa-dev/rpa-app-tm4b7dbk2g":"unchanged","ConfigMap/rpa-dev/rpa-podium-config-g994f8242c":"unchanged","ConfigMap/rpa-dev/rpa-podium-fuseki-config":"unchanged","ConfigMap/rpa-dev/rpa-podium-logback-config-g5fk2fb2kk":"unchanged","ConfigMap/rpa-dev/rpa-podium-redis-config":"unchanged","ConfigMap/uv-dev/build-script":"unchanged","ConfigMap/uv-dev/consumer-config-f56tf4mt28":"unchanged","ConfigMap/uv-dev/podium-config-fhhggtg6fc":"unchanged","ConfigMap/uv-dev/podium-fuseki-config":"unchanged","ConfigMap/uv-dev/podium-logback-config-g5fk2fb2kk":"unchanged","ConfigMap/uv-dev/podium-redis-config":"unchanged","ConfigMap/uv-dev/tdb-config":"unchanged","ConfigMap/uv-dev/tdb-stats":"unchanged","ConfigMap/uv-dev/uv-api-mg4dbb957t":"unchanged","ConfigMap/uv-dev/uv-context-file-dtc67ck577":"unchanged","CronJob/uv-dev/podium-backup":"unchanged","CronJob/uv-dev/podium-redis-pruning":"unchanged","CronWorkflow/registry-staging/registry-image-build":"unchanged","Deployment/cbd-production/cbd-api":"unchanged","Deployment/cbd-production/cbd-app":"unchanged","Deployment/cbd-production/cbd-podium-manager":"unchanged","Deployment/cbd-production/cbd-podium-publisher":"unchanged","Deployment/cbd-production/fuseki-consumer":"unchanged","Deployment/cbd-testing/cbd-api":"unchanged","Deployment/cbd-testing/cbd-app":"unchanged","Deployment/cbd-testing/cbd-podium-manager":"unchanged","Deployment/cbd-testing/cbd-podium-publisher":"unchanged","Deployment/cbd-testing/fuseki-consumer":"unchanged","Deployment/rp-production/fuseki-consumer":"unchanged","Deployment/rp-production/rp-api":"unchanged","Deployment/rp-production/rp-app":"unchanged","Deployment/rp-production/rp-podium-manager":"unchanged","Deployment/rp-production/rp-podium-publisher":"unchanged","Deployment/rp-testing/fuseki-consumer":"unchanged","Deployment/rp-testing/rp-api":"unchanged","Deployment/rp-testing/rp-app":"unchanged","Deployment/rp-testing/rp-podium-manager":"unchanged","Deployment/rp-testing/rp-podium-publisher":"unchanged","Deployment/rp-uat/fuseki-consumer":"unchanged","Deployment/rp-uat/rp-api":"unchanged","Deployment/rp-uat/rp-app":"unchanged","Deployment/rp-uat/rp-podium-manager":"unchanged","Deployment/rp-uat/rp-podium-publisher":"unchanged","Deployment/rpa-dev/fuseki-consumer":"unchanged","Deployment/rpa-dev/rpa-api":"unchanged","Deployment/rpa-dev/rpa-app":"unchanged","Deployment/rpa-dev/rpa-podium-manager":"unchanged","Deployment/rpa-dev/rpa-podium-publisher":"unchanged","Deployment/uv-dev/fuseki-json-consumer":"unchanged","Deployment/uv-dev/podium-manager":"unchanged","Deployment/uv-dev/podium-publisher":"unchanged","Deployment/uv-dev/uv-api":"unchanged","ImagePolicy/cbd-production/cbd-api":"unchanged","ImagePolicy/cbd-production/cbd-app":"unchanged","ImagePolicy/cbd-testing/cbd-api":"unchanged","ImagePolicy/cbd-testing/cbd-app":"unchanged","ImagePolicy/flux-system/orca-fuseki-consumer":"unchanged","ImagePolicy/registry-dev/fsa-registry-config":"unchanged","ImagePolicy/registry-prod/fsa-registry-config":"unchanged","ImagePolicy/registry-staging/fsa-registry-config":"unchanged","ImagePolicy/rp-production/rp-api":"unchanged","ImagePolicy/rp-production/rp-app":"unchanged","ImagePolicy/rp-testing/rp-api":"unchanged","ImagePolicy/rp-testing/rp-app":"unchanged","ImagePolicy/rp-uat/rp-api":"unchanged","ImagePolicy/rp-uat/rp-app":"unchanged","ImagePolicy/rpa-dev/rpa-api":"unchanged","ImagePolicy/rpa-dev/rpa-app":"unchanged","ImagePolicy/uv-dev/fuseki-json-consumer":"unchanged","ImagePolicy/uv-dev/uv-api":"unchanged","ImageRepository/cbd-production/cbd-api":"unchanged","ImageRepository/cbd-production/cbd-app":"unchanged","ImageRepository/cbd-testing/cbd-api":"unchanged","ImageRepository/cbd-testing/cbd-app":"unchanged","ImageRepository/flux-system/orca-fuseki-consumer":"unchanged","ImageRepository/registry-dev/fsa-registry-config":"unchanged","ImageRepository/registry-prod/fsa-registry-config":"unchanged","ImageRepository/registry-staging/fsa-registry-config":"unchanged","ImageRepository/rp-production/rp-api":"unchanged","ImageRepository/rp-production/rp-app":"unchanged","ImageRepository/rp-testing/rp-api":"unchanged","ImageRepository/rp-testing/rp-app":"unchanged","ImageRepository/rp-uat/rp-api":"unchanged","ImageRepository/rp-uat/rp-app":"unchanged","ImageRepository/rpa-dev/rpa-api":"unchanged","ImageRepository/rpa-dev/rpa-app":"unchanged","ImageRepository/uv-dev/fuseki-json-consumer":"unchanged","ImageRepository/uv-dev/uv-api":"unchanged","ImageUpdateAutomation/cbd-production/fsa-eks-cluster":"unchanged","ImageUpdateAutomation/cbd-testing/fsa-eks-cluster":"unchanged","ImageUpdateAutomation/registry-dev/fsa-eks-cluster":"unchanged","ImageUpdateAutomation/registry-prod/fsa-eks-cluster":"unchanged","ImageUpdateAutomation/registry-staging/fsa-eks-cluster":"unchanged","ImageUpdateAutomation/rp-production/fsa-eks-cluster":"unchanged","ImageUpdateAutomation/rp-testing/fsa-eks-cluster":"unchanged","ImageUpdateAutomation/rp-uat/fsa-eks-cluster":"unchanged","ImageUpdateAutomation/rpa-dev/fsa-eks-cluster":"unchanged","ImageUpdateAutomation/uv-dev/fsa-eks-cluster":"unchanged","Ingress/cbd-production/cbd-api":"unchanged","Ingress/cbd-production/cbd-app":"unchanged","Ingress/cbd-testing/cbd-api":"unchanged","Ingress/cbd-testing/cbd-app":"unchanged","Ingress/registry-dev/registry":"unchanged","Ingress/registry-prod/registry":"unchanged","Ingress/registry-staging/registry":"unchanged","Ingress/rn-prod/rn":"unchanged","Ingress/rp-production/rp-api":"unchanged","Ingress/rp-production/rp-app":"unchanged","Ingress/rp-testing/rp-api":"unchanged","Ingress/rp-testing/rp-app":"unchanged","Ingress/rp-uat/rp-api":"unchanged","Ingress/rp-uat/rp-app":"unchanged","Ingress/rpa-dev/rpa-api":"unchanged","Ingress/rpa-dev/rpa-app":"unchanged","Ingress/uv-dev/uv-api":"unchanged","PersistentVolume/efs-pv-registry-dev":"unchanged","PersistentVolume/efs-pv-uv-dev":"unchanged","PersistentVolumeClaim/registry-dev/efs-claim":"unchanged","PersistentVolumeClaim/uv-dev/efs-claim":"unchanged","PodDisruptionBudget/cbd-production/cbd-api-pdb":"unchanged","PodDisruptionBudget/cbd-production/cbd-app-pdb":"unchanged","PodDisruptionBudget/cbd-production/cbd-podium-worker-pdb":"unchanged","PodDisruptionBudget/registry-prod/registry-pdb":"unchanged","PodDisruptionBudget/rn-prod/rn-pdb":"unchanged","PodDisruptionBudget/rp-production/rp-api-pdb":"unchanged","PodDisruptionBudget/rp-production/rp-app-pdb":"unchanged","PodDisruptionBudget/rp-production/rp-podium-worker-pdb":"unchanged","Role/registry-prod/release-role":"unchanged","RoleBinding/registry-prod/release-rolebinding":"unchanged","SealedSecret/uv-dev/batch-api-secrets":"unchanged","Service/cbd-production/cbd-api":"unchanged","Service/cbd-production/cbd-app":"unchanged","Service/cbd-production/cbd-podium-manager":"unchanged","Service/cbd-production/cbd-podium-publisher":"unchanged","Service/cbd-production/cbd-podium-redis":"unchanged","Service/cbd-production/cbd-podium-worker":"unchanged","Service/cbd-testing/cbd-api":"unchanged","Service/cbd-testing/cbd-app":"unchanged","Service/cbd-testing/cbd-podium-manager":"unchanged","Service/cbd-testing/cbd-podium-publisher":"unchanged","Service/cbd-testing/cbd-podium-redis":"unchanged","Service/cbd-testing/cbd-podium-worker":"unchanged","Service/registry-dev/registry":"unchanged","Service/registry-prod/registry":"unchanged","Service/registry-staging/registry":"unchanged","Service/rn-prod/rn":"unchanged","Service/rp-production/rp-api":"unchanged","Service/rp-production/rp-app":"unchanged","Service/rp-production/rp-podium-manager":"unchanged","Service/rp-production/rp-podium-publisher":"unchanged","Service/rp-production/rp-podium-redis":"unchanged","Service/rp-production/rp-podium-worker":"unchanged","Service/rp-testing/rp-api":"unchanged","Service/rp-testing/rp-app":"unchanged","Service/rp-testing/rp-podium-manager":"unchanged","Service/rp-testing/rp-podium-publisher":"unchanged","Service/rp-testing/rp-podium-redis":"unchanged","Service/rp-testing/rp-podium-worker":"unchanged","Service/rp-uat/rp-api":"unchanged","Service/rp-uat/rp-app":"unchanged","Service/rp-uat/rp-podium-manager":"unchanged","Service/rp-uat/rp-podium-publisher":"unchanged","Service/rp-uat/rp-podium-redis":"unchanged","Service/rp-uat/rp-podium-worker":"unchanged","Service/rpa-dev/rpa-api":"unchanged","Service/rpa-dev/rpa-app":"unchanged","Service/rpa-dev/rpa-podium-manager":"unchanged","Service/rpa-dev/rpa-podium-publisher":"unchanged","Service/rpa-dev/rpa-podium-redis":"unchanged","Service/rpa-dev/rpa-podium-worker":"unchanged","Service/uv-dev/fuseki-json-consumer":"unchanged","Service/uv-dev/podium-manager":"unchanged","Service/uv-dev/podium-publisher":"unchanged","Service/uv-dev/podium-redis":"unchanged","Service/uv-dev/podium-worker":"unchanged","Service/uv-dev/uv-api":"unchanged","Service/uv-dev/uv-rds":"unchanged","ServiceMonitor/cbd-production/cbd-api":"unchanged","ServiceMonitor/cbd-production/cbd-app":"unchanged","ServiceMonitor/cbd-production/cbd-podium-manager":"unchanged","ServiceMonitor/cbd-production/cbd-podium-publisher":"unchanged","ServiceMonitor/cbd-production/cbd-podium-worker":"unchanged","ServiceMonitor/cbd-testing/cbd-api":"unchanged","ServiceMonitor/cbd-testing/cbd-app":"unchanged","ServiceMonitor/cbd-testing/cbd-podium-manager":"unchanged","ServiceMonitor/cbd-testing/cbd-podium-publisher":"unchanged","ServiceMonitor/cbd-testing/cbd-podium-worker":"unchanged","ServiceMonitor/rn-prod/rn":"unchanged","ServiceMonitor/rp-production/rp-api":"unchanged","ServiceMonitor/rp-production/rp-app":"unchanged","ServiceMonitor/rp-production/rp-podium-manager":"unchanged","ServiceMonitor/rp-production/rp-podium-publisher":"unchanged","ServiceMonitor/rp-production/rp-podium-worker":"unchanged","ServiceMonitor/rp-testing/rp-api":"unchanged","ServiceMonitor/rp-testing/rp-app":"unchanged","ServiceMonitor/rp-testing/rp-podium-manager":"unchanged","ServiceMonitor/rp-testing/rp-podium-publisher":"unchanged","ServiceMonitor/rp-testing/rp-podium-worker":"unchanged","ServiceMonitor/rp-uat/rp-api":"unchanged","ServiceMonitor/rp-uat/rp-app":"unchanged","ServiceMonitor/rp-uat/rp-podium-manager":"unchanged","ServiceMonitor/rp-uat/rp-podium-publisher":"unchanged","ServiceMonitor/rp-uat/rp-podium-worker":"unchanged","ServiceMonitor/rpa-dev/rpa-api":"unchanged","ServiceMonitor/rpa-dev/rpa-app":"unchanged","ServiceMonitor/rpa-dev/rpa-podium-manager":"unchanged","ServiceMonitor/rpa-dev/rpa-podium-publisher":"unchanged","ServiceMonitor/rpa-dev/rpa-podium-worker":"unchanged","ServiceMonitor/uv-dev/fuseki-json-consumer":"unchanged","ServiceMonitor/uv-dev/podium-manager":"unchanged","ServiceMonitor/uv-dev/podium-publisher":"unchanged","ServiceMonitor/uv-dev/podium-worker":"unchanged","ServiceMonitor/uv-dev/uv-api":"unchanged","StatefulSet/cbd-production/cbd-podium-redis":"unchanged","StatefulSet/cbd-production/cbd-podium-worker":"unchanged","StatefulSet/cbd-testing/cbd-podium-redis":"unchanged","StatefulSet/cbd-testing/cbd-podium-worker":"unchanged","StatefulSet/registry-dev/registry":"unchanged","StatefulSet/registry-prod/registry":"unchanged","StatefulSet/registry-staging/registry":"unchanged","StatefulSet/rn-prod/rn":"unchanged","StatefulSet/rp-production/rp-podium-redis":"unchanged","StatefulSet/rp-production/rp-podium-worker":"unchanged","StatefulSet/rp-testing/rp-podium-redis":"unchanged","StatefulSet/rp-testing/rp-podium-worker":"unchanged","StatefulSet/rp-uat/rp-podium-redis":"unchanged","StatefulSet/rp-uat/rp-podium-worker":"unchanged","StatefulSet/rpa-dev/rpa-podium-redis":"unchanged","StatefulSet/rpa-dev/rpa-podium-worker":"unchanged","StatefulSet/uv-dev/podium-redis":"unchanged","StatefulSet/uv-dev/podium-worker":"unchanged","WorkflowTemplate/registry-dev/publish-prep":"unchanged","WorkflowTemplate/registry-dev/registry-image-build":"unchanged","WorkflowTemplate/registry-dev/registry-restart":"unchanged","WorkflowTemplate/registry-prod/publish-prep":"unchanged","WorkflowTemplate/registry-prod/registry-image-build":"unchanged","WorkflowTemplate/registry-prod/registry-restart":"unchanged","WorkflowTemplate/registry-staging/publish-prep":"unchanged","WorkflowTemplate/registry-staging/registry-image-build":"unchanged","WorkflowTemplate/registry-staging/registry-restart":"unchanged"},"revision":"master/a41efdbc4a08041e2ffc0bd092138163477e8a95"}

The relevant config

<filter kubernetes.log.json.**>
  @id raw_json
  @type record_transformer
  <record>
    raw ${record["log"]}
  </record>
</filter>

<filter kubernetes.log.json.**>
  @id parser_argo
  @type parser
  key_name "log"
  reserve_time true
  reserve_data true
  remove_key_name_field true
  inject_key_prefix "log_"
  emit_invalid_record_to_error false
  <parse>
    @type "json"
    null_value_pattern "unchanged"
  </parse>
</filter>
Which after the output to Elasticsearch gives: field value
...
log_output.WorkflowTemplate/registry-staging/publish-prep.keyword unchanged
log_output.WorkflowTemplate/registry-staging/registry-image-build unchanged
log_output.WorkflowTemplate/registry-staging/registry-image-build.keyword unchanged
log_output.WorkflowTemplate/registry-staging/registry-restart unchanged
log_output.WorkflowTemplate/registry-staging/registry-restart.keyword unchanged
...
raw {"level":"info","ts":"2023-06-29T14:48:48.229Z","msg":"server-side apply completed","controller":"kustomization","controllerGroup":"kustomize.toolkit.fluxcd.io","controllerKind":"Kustomization","Kustomization":{"name":"eks-cluster-konf","namespace":"flux-system"},"namespace":"flux-system","name":"eks-cluster-konf","reconcileID":"06ad0250-3519-4e0a-8889-ffe3d48dd2cf","output":{"ConfigMap/cbd-production/cbd-api-k6bk5m8d82":"unchanged","ConfigMap/cbd-production/cbd-api-logback-config-h97k79tk9d":"unchanged","ConfigMap/cbd-production/cbd-app-hd2b762mk2":"unchanged","ConfigMap/cbd-production/cbd-podium-config-mt9m5dhkk7":"unchanged","ConfigMap/cbd-production/cbd-podium-fuseki-config":"unchanged","ConfigMap/cbd-production/cbd-podium-logback-config-g5fk2fb2kk":"unchanged","ConfigMap/cbd-production/cbd-podium-redis-config":"unchanged",...

Note. The raw field is indented to record the key field log prior to parsing. Here, it is complete and agrees with the input (although truncated here for brevity). Field log is then successfully parsed as json, including all of the log_output.* fields, alas their contents unchanged, again truncated for brevity. Thus I conclude

andrew-pickin-epi commented 1 year ago

I see this is waiting on me. What more is required?

daipom commented 1 year ago

Sorry! Thanks for your report! I will try to reproduce it!

daipom commented 1 year ago

@andrew-pickin-epi Thanks for sharing the detailed info for reproducing. I can reproduce this.

<source>
  @type tail
  tag test
  path /path/to/sample.json
  read_from_head true
  <parse>
    @type none
  </parse>
</source>

<filter test.**>
  @id parser_argo
  @type parser
  key_name message
  reserve_time true
  reserve_data true
  remove_key_name_field true
  inject_key_prefix log_
  emit_invalid_record_to_error false
  <parse>
    @type json
    null_value_pattern unchanged
  </parse>
</filter>

<match test.**>
  @type stdout
</match>

/path/to/sample.json (input)

{"level": "info","ts": "2023-06-29T14:48:48.229Z","msg": "server-side apply completed","controller": "kustomization","controllerGroup": "kustomize.toolkit.fluxcd.io","controllerKind": "Kustomization","Kustomization": {"name": "eks-cluster-konf","namespace": "flux-system"},"namespace": "flux-system","name": "eks-cluster-konf","reconcileID": "06ad0250-3519-4e0a-8889-ffe3d48dd2cf","output": {"ConfigMap/cbd-production/cbd-api-k6bk5m8d82": "unchanged","Deployment/cbd-production/cbd-api": "unchanged","ImagePolicy/cbd-production/cbd-api": "unchanged","Service/cbd-production/cbd-api": "unchanged","WorkflowTemplate/registry-dev/publish-prep": "unchanged"},"revision": "master/a41efdbc4a08041e2ffc0bd092138163477e8a95"}

Input JSON formatted

{
    "level": "info",
    "ts": "2023-06-29T14:48:48.229Z",
    "msg": "server-side apply completed",
    "controller": "kustomization",
    "controllerGroup": "kustomize.toolkit.fluxcd.io",
    "controllerKind": "Kustomization",
    "Kustomization": {
        "name": "eks-cluster-konf",
        "namespace": "flux-system"
    },
    "namespace": "flux-system",
    "name": "eks-cluster-konf",
    "reconcileID": "06ad0250-3519-4e0a-8889-ffe3d48dd2cf",
    "output": {
        "ConfigMap/cbd-production/cbd-api-k6bk5m8d82": "unchanged",
        "Deployment/cbd-production/cbd-api": "unchanged",
        "ImagePolicy/cbd-production/cbd-api": "unchanged",
        "Service/cbd-production/cbd-api": "unchanged",
        "WorkflowTemplate/registry-dev/publish-prep": "unchanged"
    },
    "revision": "master/a41efdbc4a08041e2ffc0bd092138163477e8a95"
}

Result:

2023-07-10 09:13:38 +0900 [info]: #0 fluentd worker is now running worker=0
2023-07-10 09:13:54.074131600 +0900 test: {"log_level":"info","log_ts":"2023-06-29T14:48:48.229Z","log_msg":"server-side apply completed","log_controller":"kustomization","log_controllerGroup":"kustomize.toolkit.fluxcd.io","log_controllerKind":"Kustomization","log_Kustomization":{"name":"eks-cluster-konf","namespace":"flux-system"},"log_namespace":"flux-system","log_name":"eks-cluster-konf","log_reconcileID":"06ad0250-3519-4e0a-8889-ffe3d48dd2cf","log_output":{"ConfigMap/cbd-production/cbd-api-k6bk5m8d82":"unchanged","Deployment/cbd-production/cbd-api":"unchanged","ImagePolicy/cbd-production/cbd-api":"unchanged","Service/cbd-production/cbd-api":"unchanged","WorkflowTemplate/registry-dev/publish-prep":"unchanged"},"log_revision":"master/a41efdbc4a08041e2ffc0bd092138163477e8a95"}

Result(formatted):

{
    "log_level": "info",
    "log_ts": "2023-06-29T14:48:48.229Z",
    "log_msg": "server-side apply completed",
    "log_controller": "kustomization",
    "log_controllerGroup": "kustomize.toolkit.fluxcd.io",
    "log_controllerKind": "Kustomization",
    "log_Kustomization": {
        "name": "eks-cluster-konf",
        "namespace": "flux-system"
    },
    "log_namespace": "flux-system",
    "log_name": "eks-cluster-konf",
    "log_reconcileID": "06ad0250-3519-4e0a-8889-ffe3d48dd2cf",
    "log_output": {
        "ConfigMap/cbd-production/cbd-api-k6bk5m8d82": "unchanged",
        "Deployment/cbd-production/cbd-api": "unchanged",
        "ImagePolicy/cbd-production/cbd-api": "unchanged",
        "Service/cbd-production/cbd-api": "unchanged",
        "WorkflowTemplate/registry-dev/publish-prep": "unchanged"
    },
    "log_revision": "master/a41efdbc4a08041e2ffc0bd092138163477e8a95"
}
daipom commented 1 year ago

This is caused by the nested JSON. null_value_pattern is not applied to the nested JSON values.

This is simply because the JSON parser does not have enough features. (It looks more like a lack of features than a bug.)

daipom commented 1 year ago

You want to discard certain values for nested JSON, right? The current JSON parser can not do that, so we need to think of another way. I can't think of a good way to do it with the normal plugins right away... It would be possible to create such a plugin.

I want to focus this issue on the lack of features of the JSON parser, for the maintenance of Fluentd. For more discussions about how to handle nested JSON, could you please ask about it in the Q&A?