axoflow / axosyslog

AxoSyslog - the scalable security data processor
https://axoflow.com
Other
45 stars 14 forks source link

`s3()` destination high memory usage caused by reload #317

Open alltilla opened 6 days ago

alltilla commented 6 days ago

AxoSyslog

Version of AxoSyslog

4.8.1

Issue

Failure

The memory usage of the syslog-ng process increases with every syslog-ng-ctl reload indefinitely. It seems like that this is not a memory leak, but the botocore python library is caching some session related data.

The following python snippet also reproduces something similar, but the memory usage stops after 5-10 "reloads":

from boto3 import Session, client

while True:
    input(": ")

    # This does not "leak"
    # c = client("s3")
    # c.close()

    s = Session()
    c = s.client("s3")
    c.close()

The caching happens around the Session object, if we create the client directly, the memory usage problem does not happen.

This might be related: https://github.com/boto/boto3/issues/3614