sipcapture / heplify

Portable and Lightweight HEP Capture Agent for HOMER
https://sipcapture.org
GNU Affero General Public License v3.0
179 stars 64 forks source link

high memory consumption after upgrading from v1.62 to v1.66.2 #284

Closed luis-alen closed 2 months ago

luis-alen commented 2 months ago

Heplify's memory footprint drastically increased after upgrading from v1.62 to v1.66.2 (https://github.com/sipcapture/heplify/compare/1.62...v1.66.2).

Is that expected?

We upgraded on the 2nd of April and, on the 3rd, when traffic started, we started noticing a very high memory consumption.

image

lmangani commented 2 months ago

Thanks for the report, most definitely not expected. Could you summarize your configuration? Anything special about it? Note: If you have any chance to downgrade and find the version where this issue happens we can fix it faster.

adubovikov commented 2 months ago

@luis-alen hard to say - there are a lot of changes - lua, decoders, prometheus, sipdefragmentation. Can you downgrade to the 1.63 and check if you will have same memory issue ? It will help to isolate the issue

luis-alen commented 2 months ago

@lmangani @adubovikov where can I get the old images? We had v1.62 on our own repo and I couldn't find them here. The oldest tagged one is v1.65.2. I tried a few untagged ones published over 1y ago but they also contain v1.65.2.

lmangani commented 2 months ago

You can just drop a binary from the releases - they're all static: https://github.com/sipcapture/heplify/releases

luis-alen commented 2 months ago

Quick update: last night we deployed v1.66.2 to environments that are way less busy than the one that made me raise this issue and, on those, the memory footprint actually decreased significantly 🤔

adubovikov commented 2 months ago

again, lets isolate the bad version :-/

luis-alen commented 2 months ago

Surprisingly the memory footprint has decreased over the last 7 days. I apologize. It seems there's nothing wrong with heplify although I'm unable to find evidence on our end that could cause such memory consumption (the number of calls and traffic patterns has not changed).

adubovikov commented 2 months ago

Super! We close the issue