Support nested log_key in S3 output plugin

[adrijshikhar]

Is your feature request related to a problem? Please describe.

I have a setup where I am streaming to coralogix and S3 simultaneously from an AWS Fargate Container. The logs sent to coralogix are for internal usage and debugging among the team and the S3 logs are for the user observability. I have added certain attributes to the log message using a filter to for easy debugging in coralogix. These fields are internal fields which are not to be shown the end user. In coralogix, you need to nest under a specific key called json for it to understand log. I am using filter to put the raw log message under the json key. I have to add the attributes keys in this json key only. In S3 output plugin, there is an option to provide the log_key which sends only the matching key.

The problem here is that the log_key only works for the first level of nesting. This is my final message looks like after the filters.

{
    "json": {
        "log": "{\"timestamp\": \"2024-03-13 13:13:53.763000+00:00\", \"message\": \"INFO: This is a sample log message\"}",
        "some_key_1_for_internal_use": "some_value_1",
        "some_key_2_for_internal_use": "some_value_2"
    },
    "applicationName": "app_name_for_coralogix",
    "subsystemName": "subsystem_name_for_coralogix"
}

Describe the solution you'd like I wanted to provide the log_key as json.log but the s3 plugin is only honouring json and sends all data inside it to S3. I would like to remove some_key_1_for_internal_use and some_key_2_for_internal_use before sending to S3 so that these fields are not visible to the end user.

Describe alternatives you've considered It might be possible that I have configured the filters or output in a wrong way. If someone can help me with better solution to my problem, thank you in advance. If this is a feature that is not present then I would be happy to contribute here.

Additional context this is my filter

[FILTER]
    Name            modify
    Match           *
    Add             some_key_1_for_internal_use     some_value_1
    Add             some_key_2_for_internal_use     some_value_2

[FILTER]
    Name            nest
    Match           *
    Operation       nest
    Wildcard        *
    Nest_under      json

[FILTER]
    Name            modify
    Match           *
    Add             applicationName                 app_name_for_coralogix
    Add             subsystemName                   subsystem_name_for_coralogix

this is my s3 output

[OUTPUT]
    name                s3
    match               *
    bucket              ${S3_BUCKET}
    region              ${S3_REGION}
    upload_timeout      5s
    s3_key_format       some_format
    upload_chunk_size   30M
    use_put_object      off
    log_key             json.log
    retry_limit         5

[adrijshikhar]

Hi any update on this?

[github-actions[bot]]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[github-actions[bot]]

This issue was closed because it has been stalled for 5 days with no activity.

[adrijshikhar]

Hi any update on this?