search

vmware / go-ipfix

An ipfix library in Go
Other
39 stars 27 forks source link

Performance impact because of decoded message struct #266

Open srikartati opened 2 years ago

srikartati commented 2 years ago

Describe the bug The message struct contains set pointer. And set pointer contains pointers to records. A single record contains pointers to multiple information elements.

Using pointers in these different levels of methods puts pressure on GC. We need to revisit the decision of using interface to support both template and data entities. This leads to the use of a pointer. Instead, move to the direct structs. In addition for the decoded messages, allocate contiguous memory and fill the required data in it.

To Reproduce Memory consumption with antrea and ipfix performance tests. Fragmentation because of GC stress.

Expected behavior We should cut down the difference between sys and heap usage

Additional context Try with a PoC and improve upon it.

srikartati commented 2 years ago

@zyiou @wsquan171 For closer look.

srikartati commented 2 years ago

Hi @dreamtalen @heanlan Hope you are doing well. Just wondering about the resolution of this performance issue where high memory is consumed.

One thing I came across is option templates in IPFIX. It is generally used for sending metadata for different info in flow records. For example, our flow record has complete metadata along with five tuple. However, metadata is sent multiple times unnecessarily. Instead Flow Exporter could send it as a separate record with IP of endpoint and corresponding pod name using an option template. Using option templates for metadata seems to be industry standard.

https://docs.paloaltonetworks.com/prisma/prisma-sd-wan/prisma-sd-wan-admin/prisma-sd-wan-sites-and-devices/use-external-services-for-monitoring/prisma-sd-wan-ipfix/ipfix-template-record-options

https://kb.vmware.com/s/article/79184

The benefit of doing this is reducing the size of the record that we store in in-memory map at flow aggregator. Smaller record will require smaller allocation of memory leading lower fragmentation. We may need to have multiple maps.

Just wanted to share this for a while. Cheers!