search

phaag / nfsen

Legacy NfSen code
Other
23 stars 11 forks source link

Slow performance when querying custom profiles #38

Open mkoelewijn opened 2 months ago

mkoelewijn commented 2 months ago

We have upgraded to a new server for NfSen with the following hardware specifications:

CPU: 2x Intel(R) Xeon(R) Gold 5420+ @ 2.0GHz Memory: 128GB Micron DDR5 SDRAM Storage: NVMe PE8010 RI U.2

We are running the following software versions:

Our configuration is as follows:

NfSen:

Nfdump:

When querying the LIVE profile, performance is significantly improved—approximately 5 times faster than our previous NfSen setup. For example:

Summary: total flows: 280850300, total bytes: 84.4 T, total packets: 113.1 G, avg bps: 162.3 M, avg pps: 27190, avg bpp: 746 Time window: 2024-07-05 01:49:16 - 2024-08-22 05:05:12 Total flows processed: 280850300, passed: 280850300, Blocks skipped: 0, Bytes read: 41209733320 Sys: 16.9656s User: 192.8634s Wall: 159.2610s flows/second: 1763459.2 Runtime: 160.1039s

Note: This query does not trigger any "Too many exporters (>256)" errors.

The issue arises when we create a new profile with different channels; the query speed slows down significantly (taking as long as it did on the old server, or even longer). For example, a Stat TopN 10 query:

Summary: total flows: 6113, total bytes: 3.3 G, total packets: 7.0 M, avg bps: 655285, avg pps: 177, avg bpp: 461 Time window: 2024-08-22 01:03:23 - 2024-08-22 12:04:50 Total flows processed: 6113, passed: 6113, Blocks skipped: 0, Bytes read: 126371472 Sys: 0.5604s User: 81.1822s Wall: 81.7372s flows/second: 74.8 Runtime: 81.7373s

Note: This query does trigger the "Too many exporters (>256)" errors.

What could be causing the slow performance when querying profiles other than LIVE?

Could the "Too many exporters (>256)" errors be contributing to this issue?

Do you have any recommendations to improve the speed and fully leverage the hardware capabilities?

phaag commented 2 months ago

That is pretty difficult to tell. I cannot debug your installation. The "Too many exporters (>256)" does not really harm. You may try the latest master which does improve performance over #4f409e9. If possible, try to debug the performance bottleneck. Furthermore, there are certain things which cannot be processed in parallel - at least not to my knowledge - such as building up the top N hash. If you could try with the latest master, let me know the results.