Higher Memory Usage with log rotation than without: Is this expected?

[PettitWesley]

I got a report from one of my customers which is interesting:

Currently, we use Fluent Bit to tail log files that are rotated by our app and are then redirected to S3 and Cloudwatch. The way the rotation works is that every 60 seconds, the driver then renames the active file to something like "rotated-file.1" and creates a new file with name "rotated-file-to-watch" that FluentBit is set to watch (real file names redacted).

When running a long test job, Prometheus is detecting that maximum memory usage is ~400 MB when this rotation is turned on compared to ~175 MB when rotation is turned off, with nothing else changed. This increase in memory usage is unusual/surprising.

Is this expected? Is it normal- does it match other user results? Is there any way to tune the config for this case?

The specific FluentBit configuration (slightly redacted) is below:

[SERVICE]
    # Log level can be set to info when finished.
    log_level debug
    # Flush input records to output plugins every N seconds.
    flush 10
    # File path to store buffer.
    storage.path /tmp/fluent-bit/output-buffer

[INPUT]
    # Reads file specified in path.
    name tail
    tag stdout
    path REDACTED
    # Stores last read location in file in DB file. Equivalent to "pos_file" in FluentD.
    db /tmp/fluent-bit/stdout.db
    # Only allows fluent-bit to read DB file, but increases performance.
    db.locking true
    # Read from head of file for new files.
    read_from_head true
    # Refresh list of files watched every N seconds.
    refresh_interval 10
    # Limits memory allocated for each file.
    buffer_max_size 8M
    # Limits memory use by input plugin before pausing until next flush.
    mem_buf_limit 8M
    # If a line is extremely long and plugin reaches mem_buf_limit before end of line, skip line.
    skip_long_lines on
    # Sets buffer to store using files instead of memory.
    storage.type filesystem

[INPUT]
    name tail
    tag stderr
    path REDACTED
    db /tmp/fluent-bit/stderr.db
    db.locking true
    read_from_head true
    refresh_interval 10
    buffer_max_size 8M
    mem_buf_limit 8M
    skip_long_lines on
    storage.type filesystem

[INPUT]
    name tail
    tag event-log
    path REDACTED
    db /tmp/fluent-bit/event-log.db
    db.locking true
    read_from_head true
    refresh_interval 10
    buffer_max_size 8M
    mem_buf_limit 8M
    skip_long_lines on
    storage.type filesystem

[OUTPUT]
    name s3
    match app-status
    region ${AWS_REGION}
    bucket $BUCKET}
    s3_key_format REDACTED
    use_put_object true
    # 6 seconds is minimum for upload_timeout. Smaller values will upload more frequently.
    upload_timeout 6s
    total_file_size 1M
    store_dir /tmp/fluent-bit/s3-output-buffer
    retry_limit 5
    log_key exec
    static_file_path true

[OUTPUT]
    # S3 Output plugin reads from 'match' input and writes to specified S3 bucket.
    name s3
    match stdout 
    region ${AWS_REGION}
    bucket ${BUCKET}
    # $UUID generates a random string and is required when 'use_put_object' is true. 
    s3_key_format REDACTED
    # Use PutObject API instead of Multipart API.
    use_put_object true
    # Max size of each file uploaded to S3. Must be within 1M ~ 50M when 'use_put_object' is true.
    total_file_size 32M
    # Uploads every N seconds.
    upload_timeout 120s
    # Compresses files as gzip.
    compression gzip
    # Buffer for S3 Output plugin is stored here.
    store_dir /tmp/fluent-bit/s3-output-buffer
    # Discards data after 5 attempts. Equivalent to 'retry_max_times' in FluentD.
    retry_limit 5
    log_key log
    preserve_data_ordering true

[OUTPUT]
    name s3
    match stderr
    region ${AWS_REGION} 
    bucket ${BUCKET}
    s3_key_format REDACTED
    use_put_object true
    total_file_size 32M
    upload_timeout 120s
    compression gzip
    store_dir /tmp/fluent-bit/s3-output-buffer
    retry_limit 5
    log_key log
    preserve_data_ordering true

[OUTPUT]
    name s3
    match event-log
    region ${AWS_REGION}
    bucket ${BUCKET}
    s3_key_format REDACTED
    use_put_object true
    total_file_size 32M
    upload_timeout 30s
    store_dir /tmp/fluent-bit/s3-output-buffer
    retry_limit 5
    log_key log
    preserve_data_ordering true

[OUTPUT]
    # Cloudwatch output plugin reads from 'match' input and writes to specified log group.
    name cloudwatch_logs
    match stdout
    log_group_name ${CW_LOG_GROUP_NAME}
    log_stream_name ${CW_LOG_STREAM_STDOUT}
    auto_create_group true
    region ${AWS_REGION}
    # Only writes value of key 'log'. This excludes other keys and values like timestamp.
    log_key log
    retry_limit 5

[OUTPUT]
    name cloudwatch_logs
    match stderr
    log_group_name ${CW_LOG_GROUP_NAME}
    log_stream_name ${CW_LOG_STREAM_STDERR}
    auto_create_group true
    region ${AWS_REGION}
    log_key log
    retry_limit 5

[github-actions[bot]]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[dalcouffe]

I am seeing a similar issue running fluent bit with Kubernetes. If there are container logs that are rotated frequently by kubelet then the instances of fluent-bit on those hosts will grow in memory usage until they OOM. Seems likely there is a memory leak around how the log rotation is handled.

[JeffLuoo]

+1 observing the similar issue

[mcauto]

I have the similar issue. :(

It seems to have a memory leak. The settings I am using are as follows.

INPUT: http FILTER: expect, rewrite_tag, record_modifier OUTPUT: http, kafka

[github-actions[bot]]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[nithin-kumar]

@PettitWesley We have identified a possible cause for this issue. https://github.com/fluent/fluent-bit/blob/master/plugins/out_cloudwatch_logs/cloudwatch_api.c#L1017 Every time get_or_create_log_streams is invoked, it checks whether any streams are expired. If the streams are not expired then it creates a new stream. And now the default expiration time for a stream is set as 4 hours. (https://github.com/fluent/fluent-bit/blob/master/plugins/out_cloudwatch_logs/cloudwatch_api.c#L1066) If the log rotation time is lower, the created log streams are not freed up by the system until 4 hours are reached, potentially leaking the memory for a long-running job. As a quick fix

1. We can reduce the default expiration time
2. Match the log rotation time with the stream expiration time.

[PettitWesley]

@nithin-kumar good thought. I will fix this in cloudwatch_logs. The issue description says rotation happens every 1 minute, which is fast. I bet there could be many other places in the code which have similar behavior in which memory accumulates proportional to the number of tags processed over time that we/anyone could find/profile @JeffLuoo if we had time...