netenglabs / suzieq

Using network observability to operate and design healthier networks
https://www.stardustsystems.net/
Apache License 2.0
791 stars 106 forks source link

sq-poller: remove RELATIVE_BASE option to avoid leak #856

Closed claudiolor closed 1 year ago

claudiolor commented 1 year ago

Description

The dateparser library cache for each combination of the provided settings a copy of the dictionary used for the regex to extract the data from the string. When different combination of settings is provided at every function call (i.e. when we always provide a different value of RELATIVE_BASE) dateparser will cache a dictionary at every function call, causing the memory to grow really quickly. This PR introduces a workaround to avoid the usage of the RELATIVE_BASE option, i.e. computing the difference between now and the relative base and subtracting it to the result produced by dateparser. The following graph shows the difference of memory usage between 0.20.0 and the same version with this patch applied:

Figure_1

Type of change

Please delete options that are not relevant.