search

inex / IXP-Manager

Full stack web application powering peering at over 200 Internet Exchange Points (IXPs) globally.
https://www.ixpmanager.org/
GNU General Public License v2.0
365 stars 160 forks source link

sflow-to-rrd-handler: spikes when API key disappears #829

Open nickhilliard opened 1 year ago

nickhilliard commented 1 year ago

If the API key disappears, then sflow-to-rrd-handler will not zero out the matrix on each write, causing the graph output to increase monotonically.

Probably the process needs to trap authorization problems in reload_mactable() and abort if authorization is unsuccessful, as that is a terminal error that cannot be resolved without operator intervention.

agbcix commented 1 year ago

We are experiencing a similar, but maybe more generic issue here. [Or do you consider this a different issue?]

Whenever the API is unavailable (database, webserver, ...), sflow-to-rrd-handler will block in reload_mactable() and eventually produce spikes.

It'd be cool to decouple reload_mactable() from the main loop, e.g. short timeout and improved error handling.

barryo commented 1 year ago

PRs welcome @agbcix if you have perl skillz. See this page on contributing.

agbcix commented 1 year ago

I looked into this more deeply now. It seems like reload_mactable() is dealing well with the timeout etc.

But: Upon any error (timeout, authentication issue, ...) the $newmactable will be undefined. Only if $newmactable is defined, we are running matrix_init() which is actually flushing the in-memory data.

Proposed fix (no PR yet): run $matrix = matrix_init($mactable, $infraid); unconditionally.