search

inex / IXP-Manager

Full stack web application powering peering at over 200 Internet Exchange Points (IXPs) globally.
https://www.ixpmanager.org/
GNU General Public License v2.0
363 stars 159 forks source link

Aggregate MRTG graphs are including the reseller uplink interfaces as well as the peering interfaces, creating incorrect graphs/stats. #855

Closed ichilton closed 1 year ago

ichilton commented 1 year ago
ISSUE TYPE

Bug Report

OS
VERSION
define( 'APPLICATION_VERSION', '6.3.0' );
define( 'APPLICATION_VERDATE', '2022110200' );

(latest on release-v6 branch)

ENVIRONMENT
PHP 8.0.14 (cli) (built: Dec 20 2021 21:22:57) ( NTS )
Copyright (c) The PHP Group
Zend Engine v4.0.14, Copyright (c) Zend Technologies
    with Zend OPcache v8.0.14, Copyright (c), by Zend Technologies
ichilton@portal:~$ dpkg -l | grep php
ii  libapache2-mod-php8.0          8.0.14-1+ubuntu20.04.1+deb.sury.org+1                               amd64        server-side, HTML-embedded scripting language (Apache 2 module)
ii  php-common                     2:91+ubuntu20.04.1+deb.sury.org+1                                   all          Common files for PHP packages
ii  php-composer-ca-bundle         1.2.6-1                                                             all          utility library to find a path to the system CA bundle
ii  php-composer-semver            1.5.1-1                                                             all          utilities, version constraint parsing and validation
ii  php-composer-spdx-licenses     1.5.3-1                                                             all          SPDX licenses list and validation library
ii  php-composer-xdebug-handler    1.4.0-1                                                             all          Restarts a process without Xdebug
ii  php-json-schema                5.2.9-1                                                             all          implementation of JSON schema
ii  php-psr-container              1.0.0-2                                                             all          Common Container Interface (PHP FIG PSR-11)
ii  php-psr-log                    1.1.2-1                                                             all          common interface for logging libraries
ii  php-symfony-console            4.3.8+dfsg-1ubuntu1                                                 all          run tasks from the command line
ii  php-symfony-filesystem         4.3.8+dfsg-1ubuntu1                                                 all          basic filesystem utilities
ii  php-symfony-finder             4.3.8+dfsg-1ubuntu1                                                 all          find files and directories
ii  php-symfony-process            4.3.8+dfsg-1ubuntu1                                                 all          execute commands in sub-processes
ii  php-symfony-service-contracts  1.1.8-1                                                             all          Generic abstractions related to writing services
ii  php8.0                         8.0.14-1+ubuntu20.04.1+deb.sury.org+1                               all          server-side, HTML-embedded scripting language (metapackage)
ii  php8.0-bcmath                  8.0.14-1+ubuntu20.04.1+deb.sury.org+1                               amd64        Bcmath module for PHP
ii  php8.0-cgi                     8.0.14-1+ubuntu20.04.1+deb.sury.org+1                               amd64        server-side, HTML-embedded scripting language (CGI binary)
ii  php8.0-cli                     8.0.14-1+ubuntu20.04.1+deb.sury.org+1                               amd64        command-line interpreter for the PHP scripting language
ii  php8.0-common                  8.0.14-1+ubuntu20.04.1+deb.sury.org+1                               amd64        documentation, examples and common module for PHP
ii  php8.0-curl                    8.0.14-1+ubuntu20.04.1+deb.sury.org+1                               amd64        CURL module for PHP
ii  php8.0-ds                      1.4.0-1+ubuntu20.04.1+deb.sury.org+1                                amd64        PHP extension providing efficient data structures for PHP 7
ii  php8.0-fpm                     8.0.14-1+ubuntu20.04.1+deb.sury.org+1                               amd64        server-side, HTML-embedded scripting language (FPM-CGI binary)
ii  php8.0-gd                      8.0.14-1+ubuntu20.04.1+deb.sury.org+1                               amd64        GD module for PHP
ii  php8.0-igbinary                3.2.6+2.0.8-2+ubuntu20.04.1+deb.sury.org+3                          amd64        igbinary PHP serializer
ii  php8.0-intl                    8.0.14-1+ubuntu20.04.1+deb.sury.org+1                               amd64        Internationalisation module for PHP
ii  php8.0-mbstring                8.0.14-1+ubuntu20.04.1+deb.sury.org+1                               amd64        MBSTRING module for PHP
ii  php8.0-memcache                8.0+4.0.5.2+3.0.9~20170802.e702b5f9+-3+ubuntu20.04.1+deb.sury.org+3 amd64        memcache extension module for PHP
ii  php8.0-memcached               3.1.5+2.2.0-10+ubuntu20.04.1+deb.sury.org+3                         amd64        memcached extension module for PHP, uses libmemcached
ii  php8.0-msgpack                 2.2.0~rc1+2.1.2+0.5.7-2+ubuntu20.04.1+deb.sury.org+3                amd64        PHP extension for interfacing with MessagePack
ii  php8.0-mysql                   8.0.14-1+ubuntu20.04.1+deb.sury.org+1                               amd64        MySQL module for PHP
ii  php8.0-opcache                 8.0.14-1+ubuntu20.04.1+deb.sury.org+1                               amd64        Zend OpCache module for PHP
ii  php8.0-readline                8.0.14-1+ubuntu20.04.1+deb.sury.org+1                               amd64        readline module for PHP
ii  php8.0-rrd                     2.0.3+1.1.3-2+ubuntu20.04.1+deb.sury.org+3                          amd64        PHP bindings to rrd tool system
ii  php8.0-snmp                    8.0.14-1+ubuntu20.04.1+deb.sury.org+1                               amd64        SNMP module for PHP
ii  php8.0-xml                     8.0.14-1+ubuntu20.04.1+deb.sury.org+1                               amd64        DOM, SimpleXML, XML, and XSL module for PHP
ii  php8.0-yaml                    2.2.2+2.1.0+2.0.4+1.3.2-1+ubuntu20.04.1+deb.sury.org+3              amd64        YAML-1.1 parser and emitter for PHP
ii  php8.0-zip                     8.0.14-1+ubuntu20.04.1+deb.sury.org+1                               amd64        Zip module for PHP
CONFIGURATION

Don't want to put that on GitHub.

SUMMARY

We are in the process of moving from physical loopback cables to EOS layer 2 subinterfaces to provide reseller ports.

We have just his morning moved our first which is doing a significant amount of traffic (~50G) as it's actually an ASN share rather than reseller.

We are now seeing that our traffic graphs have ~40G difference between IN and OUT traffic.

I had a hunch what this would be and have confirmed it - the MRTG aggregate graphs are graphing the reseller uplink interfaces as well as the peering interfaces.

i.e it's graphing both the traffic on the physical port as well as the subinterfaces.

I have verified this by manually editing mrtg.cfg and removing each occurrence of that physical port and the IN and OUT have now converged to within a few hundred Mbps, which is about right for the other resellers.

It should just be including the 'peering' ports, not those set to 'reseller'.

IMPORTANCE

HIGH - this is affecting our production traffic graphs.

nickhilliard commented 1 year ago

is the reseller NNI physical interface set to type: Reseller?

barryo commented 1 year ago

@ichilton - I'll need blow away some cobwebs to look at that code - hasn't been changed in years and it seems slightly odd that this hasn't come up before. E.g. if this is the cause, graphing the reseller port would be done in both fan out mode and veth hairpinning mode, I think.

(edited to remove reference to disabling polling - that's per-switch, not per-interface)

ichilton commented 1 year ago

is the reseller NNI physical interface set to type: Reseller?

Yep!

...and the two sub-interfaces set to 'Peering'.

ichilton commented 1 year ago

@ichilton - I'll need blow away some cobwebs to look at that code - hasn't been changed in years and it seems slightly odd that this hasn't come up before.

As I say, we only noticed it because this 'reseller' is doing more than a trivial amount of traffic (40Gbps in this case).

Happy to send screenshots or screen share demo on a zoom call if you think we might have something slightly different to you / wrong, rather than the code?

...but the parent interfaces are definitely set to type 'Reseller' and are appearing in the MRTG aggregated graphs.

Thanks,

Ian

barryo commented 1 year ago

Fixed via inex/IXP-Manager@e40be75de9deec22978df766dee88b912242316b