search

inex / IXP-Manager

Full stack web application powering peering at over 200 Internet Exchange Points (IXPs) globally.
https://www.ixpmanager.org/
GNU General Public License v2.0
380 stars 164 forks source link

SNMP polling for some switches fails #910

Open rfc1036 opened 1 month ago

rfc1036 commented 1 month ago
ISSUE TYPE

Bug Report

OS

Ubuntu 24.04 LTS

VERSION
define( 'APPLICATION_VERSION', '6.4.1' );
define( 'APPLICATION_VERDATE', '2024060900' );
ENVIRONMENT
PHP 8.0.30 (cli) (built: Aug  2 2024 16:27:12) ( NTS )
Copyright (c) The PHP Group
Zend Engine v4.0.30, Copyright (c) Zend Technologies
    with Zend OPcache v8.0.30, Copyright (c), by Zend Technologies

ii  php-common                                       2:94+ubuntu24.04.1+deb.sury.org+1             all          Common files for PHP packages
ii  php-pear                                         1:1.10.13+submodules+notgz+2022032202-2build1 all          PEAR Base System
ii  php8.0-bcmath                                    1:8.0.30-8+ubuntu24.04.1+deb.sury.org+1       amd64        Bcmath module for PHP
ii  php8.0-cli                                       1:8.0.30-8+ubuntu24.04.1+deb.sury.org+1       amd64        command-line interpreter for the PHP scripting language
ii  php8.0-common                                    1:8.0.30-8+ubuntu24.04.1+deb.sury.org+1       amd64        documentation, examples and common module for PHP
ii  php8.0-curl                                      1:8.0.30-8+ubuntu24.04.1+deb.sury.org+1       amd64        CURL module for PHP
ii  php8.0-fpm                                       1:8.0.30-8+ubuntu24.04.1+deb.sury.org+1       amd64        server-side, HTML-embedded scripting language (FPM-CGI binary)
ii  php8.0-gd                                        1:8.0.30-8+ubuntu24.04.1+deb.sury.org+1       amd64        GD module for PHP
ii  php8.0-igbinary                                  3.2.15-1+ubuntu24.04.1+deb.sury.org+2         amd64        igbinary PHP serializer
ii  php8.0-intl                                      1:8.0.30-8+ubuntu24.04.1+deb.sury.org+1       amd64        Internationalisation module for PHP
ii  php8.0-mbstring                                  1:8.0.30-8+ubuntu24.04.1+deb.sury.org+1       amd64        MBSTRING module for PHP
ii  php8.0-mcrypt                                    3:1.0.7-1+ubuntu24.04.1+deb.sury.org+2        amd64        PHP bindings for the libmcrypt library
ii  php8.0-memcache                                  8.2-2+ubuntu24.04.1+deb.sury.org+1            amd64        memcache extension module for PHP
ii  php8.0-memcached                                 3.2.0++-1+ubuntu24.04.1+deb.sury.org+2        amd64        memcached extension module for PHP, uses libmemcached
ii  php8.0-msgpack                                   1:2.2.0-4+ubuntu24.04.1+deb.sury.org+1        amd64        PHP extension for interfacing with MessagePack
ii  php8.0-mysql                                     1:8.0.30-8+ubuntu24.04.1+deb.sury.org+1       amd64        MySQL module for PHP
ii  php8.0-opcache                                   1:8.0.30-8+ubuntu24.04.1+deb.sury.org+1       amd64        Zend OpCache module for PHP
ii  php8.0-readline                                  1:8.0.30-8+ubuntu24.04.1+deb.sury.org+1       amd64        readline module for PHP
ii  php8.0-rrd                                       2.0.3++-1+ubuntu24.04.1+deb.sury.org+2        amd64        PHP bindings to rrd tool system
ii  php8.0-snmp                                      1:8.0.30-8+ubuntu24.04.1+deb.sury.org+1       amd64        SNMP module for PHP
ii  php8.0-xml                                       1:8.0.30-8+ubuntu24.04.1+deb.sury.org+1       amd64        DOM, SimpleXML, XML, and XSL module for PHP
ii  php8.0-yaml                                      2.2.3-2+ubuntu24.04.1+deb.sury.org+2          amd64        YAML-1.1 parser and emitter for PHP
ii  php8.0-zip                                       1:8.0.30-8+ubuntu24.04.1+deb.sury.org+1       amd64        Zip module for PHP
CONFIGURATION
APP_DEBUG=false
APP_URL="https://manager.minap.it"
APP_TIMEZONE="Europe/Rome"
APP_LOG="daily"
APP_LOG_LEVEL=debug
IXP_RESELLER_ENABLED=false
IXP_AS112_UI_ACTIVE=false
IXP_FE_FRONTEND_DISABLED_LOGO=false
IXP_FE_FRONTEND_DISABLED_PEERING_MATRIX=true
GRAPHER_BACKENDS="mrtg"
GRAPHER_CACHE_ENABLED=true
GRAPHER_BACKEND_MRTG_DBTYPE="rrd"
GRAPHER_BACKEND_MRTG_WORKDIR="/var/lib/mrtg/"
GRAPHER_BACKEND_MRTG_LOGDIR="/var/lib/mrtg/"
IXP_API_JSONEXPORTSCHEMA_PUBLIC=true
VIEW_SKIN="minap"
VIEW_SMARTY_SKIN="minap"
CACHE_DRIVER=memcached
SESSION_DRIVER=memcached
DOCTRINE_PROXY_AUTOGENERATE=true
DOCTRINE_CACHE=memcached
DOCTRINE_CACHE_NAMESPACE=IXPMANAGERNAMESPACE
IXP_IRRDB_BGPQ3_PATH=/usr/bin/bgpq3
AUTH_PEERINGDB_ENABLED=true
PEERINGDB_OAUTH_CLIENT_ID="..."
PEERINGDB_OAUTH_CLIENT_SECRET="..."
PEERINGDB_OAUTH_REDIRECT="https://manager.minap.it/auth/login/peeringdb/callback"
IXP_FE_FRONTEND_DISABLED_CONSOLE=true
IXP_FE_FRONTEND_DISABLED_CUSTKIT=true
IXP_FE_FRONTEND_DISABLED_LOOKING_GLASS=false
IXP_FE_FRONTEND_DISABLED_FILTERED_PREFIXES=true
IXP_FE_LAYER2_ADDRESSES_CUST_CAN_EDIT=true
IXP_FE_LAYER2_ADDRESSES_CUST_PARAMS_MIN_ADDRESSES=1
IXP_FE_LAYER2_ADDRESSES_CUST_PARAMS_MAX_ADDRESSES=2
IXP_FE_LAYER2_ADDRESSES_EMAIL_ON_CUSTOMER_CHANGE=true
IXP_FE_LAYER2_ADDRESSES_EMAIL_ON_SUPERUSER_CHANGE=false
IXP_FE_LAYER2_ADDRESSES_EMAIL_ON_CHANGE_DEST="noc@minap.it"
GRAPHER_ACCESS_IXP=0
GRAPHER_ACCESS_SWITCH=0
GRAPHER_ACCESS_CUSTOMER=1
GRAPHER_ACCESS_TRUNK=0
SUMMARY

Accessing the SNMP-related actions of some switches crashes IXP Manager.

STEPS TO REPRODUCE

Access the "View / Edit Ports" or "Live Ports State" menus or add a new switch with SNMP polling.

EXPECTED RESULTS

These actions should work as they do for other switches. They used to work in the past.

ACTUAL RESULTS

I get a 500 error for "Undefined array key 2".

After adding the new switch only the management port was detected with SNMP.

"Switches / View Switch" is reporting the current time for "Last Polled" and generally looks correct.

IMPORTANCE

Unsure: I had to manually configure the peering ports.

RELEVANT LOGS
[2024-09-27 04:11:15] production.INFO:  [sw-mynet2] Platform: Found Model: Chassis  
[2024-09-27 04:11:15] production.INFO:  [sw-mynet2] Platform: Found Os: IOS  
[2024-09-27 04:11:15] production.INFO:  [sw-mynet2] Platform: Updating OsDate from  to Unknown  
[2024-09-27 04:11:15] production.INFO:  [sw-mynet2] Platform: Found OsVersion: 17.12.3  
[2024-09-27 04:11:15] production.INFO:  [sw-mynet2] Platform: Found SerialNumber:   
[2024-09-27 04:11:15] production.INFO:  - sw-mynet2 - found pre-existing port for ifIndex 1  
[2024-09-27 04:11:15] production.ERROR: Undefined array key 2 {"userId":1,"exception":"[object] (ErrorException(code: 0): Undefined array key 2 at /srv/ixpmanager/app/Models/Switcher.php:385)
[stacktrace]
#0 /srv/ixpmanager/vendor/laravel/framework/src/Illuminate/Foundation/Bootstrap/HandleExceptions.php(270): Illuminate\\Foundation\\Bootstrap\\HandleExceptions->handleError()
#1 /srv/ixpmanager/app/Models/Switcher.php(385): Illuminate\\Foundation\\Bootstrap\\HandleExceptions->Illuminate\\Foundation\\Bootstrap\\{closure}()
#2 /srv/ixpmanager/app/Http/Controllers/Switches/SwitchPortController.php(594): IXP\\Models\\Switcher->snmpPollSwitchPorts()
#3 /srv/ixpmanager/vendor/laravel/framework/src/Illuminate/Routing/Controller.php(54): IXP\\Http\\Controllers\\Switches\\SwitchPortController->snmpPoll()
#4 /srv/ixpmanager/vendor/laravel/framework/src/Illuminate/Routing/ControllerDispatcher.php(43): Illuminate\\Routing\\Controller->callAction()
#5 /srv/ixpmanager/vendor/laravel/framework/src/Illuminate/Routing/Route.php(260): Illuminate\\Routing\\ControllerDispatcher->dispatch()
#6 /srv/ixpmanager/vendor/laravel/framework/src/Illuminate/Routing/Route.php(205): Illuminate\\Routing\\Route->runController()
#7 /srv/ixpmanager/vendor/laravel/framework/src/Illuminate/Routing/Router.php(798): Illuminate\\Routing\\Route->run()
#8 /srv/ixpmanager/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(141): Illuminate\\Routing\\Router->Illuminate\\Routing\\{closure}()
#9 /srv/ixpmanager/app/Utils/Http/Controllers/Frontend/EloquentController.php(188): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#10 /srv/ixpmanager/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(162): IXP\\Utils\\Http\\Controllers\\Frontend\\EloquentController->IXP\\Utils\\Http\\Controllers\\Frontend\\{closure}()
#11 /srv/ixpmanager/app/Http/Middleware/Google2FA.php(79): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#12 /srv/ixpmanager/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(180): IXP\\Http\\Middleware\\Google2FA->handle()
#13 /srv/ixpmanager/app/Http/Middleware/Eloquent2Frontend.php(72): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#14 /srv/ixpmanager/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(180): IXP\\Http\\Middleware\\Eloquent2Frontend->handle()
#15 /srv/ixpmanager/app/Http/Middleware/ControllerEnabled.php(94): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#16 /srv/ixpmanager/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(180): IXP\\Http\\Middleware\\ControllerEnabled->handle()
#17 /srv/ixpmanager/vendor/laravel/framework/src/Illuminate/Routing/Middleware/SubstituteBindings.php(50): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#18 /srv/ixpmanager/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(180): Illuminate\\Routing\\Middleware\\SubstituteBindings->handle()
#19 /srv/ixpmanager/vendor/laravel/framework/src/Illuminate/Foundation/Http/Middleware/VerifyCsrfToken.php(78): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#20 /srv/ixpmanager/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(180): Illuminate\\Foundation\\Http\\Middleware\\VerifyCsrfToken->handle()
#21 /srv/ixpmanager/vendor/laravel/framework/src/Illuminate/View/Middleware/ShareErrorsFromSession.php(49): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#22 /srv/ixpmanager/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(180): Illuminate\\View\\Middleware\\ShareErrorsFromSession->handle()
#23 /srv/ixpmanager/vendor/laravel/framework/src/Illuminate/Session/Middleware/StartSession.php(121): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#24 /srv/ixpmanager/vendor/laravel/framework/src/Illuminate/Session/Middleware/StartSession.php(64): Illuminate\\Session\\Middleware\\StartSession->handleStatefulRequest()
#25 /srv/ixpmanager/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(180): Illuminate\\Session\\Middleware\\StartSession->handle()
#26 /srv/ixpmanager/vendor/laravel/framework/src/Illuminate/Cookie/Middleware/AddQueuedCookiesToResponse.php(37): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#27 /srv/ixpmanager/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(180): Illuminate\\Cookie\\Middleware\\AddQueuedCookiesToResponse->handle()
#28 /srv/ixpmanager/vendor/laravel/framework/src/Illuminate/Cookie/Middleware/EncryptCookies.php(67): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#29 /srv/ixpmanager/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(180): Illuminate\\Cookie\\Middleware\\EncryptCookies->handle()
#30 /srv/ixpmanager/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(116): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#31 /srv/ixpmanager/vendor/laravel/framework/src/Illuminate/Routing/Router.php(799): Illuminate\\Pipeline\\Pipeline->then()
#32 /srv/ixpmanager/vendor/laravel/framework/src/Illuminate/Routing/Router.php(776): Illuminate\\Routing\\Router->runRouteWithinStack()
#33 /srv/ixpmanager/vendor/laravel/framework/src/Illuminate/Routing/Router.php(740): Illuminate\\Routing\\Router->runRoute()
#34 /srv/ixpmanager/vendor/laravel/framework/src/Illuminate/Routing/Router.php(729): Illuminate\\Routing\\Router->dispatchToRoute()
#35 /srv/ixpmanager/vendor/laravel/framework/src/Illuminate/Foundation/Http/Kernel.php(190): Illuminate\\Routing\\Router->dispatch()
#36 /srv/ixpmanager/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(141): Illuminate\\Foundation\\Http\\Kernel->Illuminate\\Foundation\\Http\\{closure}()
#37 /srv/ixpmanager/vendor/laravel/framework/src/Illuminate/Foundation/Http/Middleware/TransformsRequest.php(21): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#38 /srv/ixpmanager/vendor/laravel/framework/src/Illuminate/Foundation/Http/Middleware/ConvertEmptyStringsToNull.php(31): Illuminate\\Foundation\\Http\\Middleware\\TransformsRequest->handle()
#39 /srv/ixpmanager/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(180): Illuminate\\Foundation\\Http\\Middleware\\ConvertEmptyStringsToNull->handle()
#40 /srv/ixpmanager/vendor/laravel/framework/src/Illuminate/Foundation/Http/Middleware/TransformsRequest.php(21): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#41 /srv/ixpmanager/vendor/laravel/framework/src/Illuminate/Foundation/Http/Middleware/TrimStrings.php(40): Illuminate\\Foundation\\Http\\Middleware\\TransformsRequest->handle()
#42 /srv/ixpmanager/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(180): Illuminate\\Foundation\\Http\\Middleware\\TrimStrings->handle()
#43 /srv/ixpmanager/vendor/laravel/framework/src/Illuminate/Foundation/Http/Middleware/ValidatePostSize.php(27): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#44 /srv/ixpmanager/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(180): Illuminate\\Foundation\\Http\\Middleware\\ValidatePostSize->handle()
#45 /srv/ixpmanager/vendor/laravel/framework/src/Illuminate/Foundation/Http/Middleware/PreventRequestsDuringMaintenance.php(86): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#46 /srv/ixpmanager/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(180): Illuminate\\Foundation\\Http\\Middleware\\PreventRequestsDuringMaintenance->handle()
#47 /srv/ixpmanager/vendor/laravel/framework/src/Illuminate/Http/Middleware/HandleCors.php(49): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#48 /srv/ixpmanager/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(180): Illuminate\\Http\\Middleware\\HandleCors->handle()
#49 /srv/ixpmanager/vendor/fideloper/proxy/src/TrustProxies.php(57): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#50 /srv/ixpmanager/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(180): Fideloper\\Proxy\\TrustProxies->handle()
#51 /srv/ixpmanager/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(116): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#52 /srv/ixpmanager/vendor/laravel/framework/src/Illuminate/Foundation/Http/Kernel.php(165): Illuminate\\Pipeline\\Pipeline->then()
#53 /srv/ixpmanager/vendor/laravel/framework/src/Illuminate/Foundation/Http/Kernel.php(134): Illuminate\\Foundation\\Http\\Kernel->sendRequestThroughRouter()
#54 /srv/ixpmanager/public/index.php(89): Illuminate\\Foundation\\Http\\Kernel->handle()
#55 {main}
"}
rfc1036 commented 1 month ago

This is actually a major issue for us, because the manually added ports are not working in IXP Manager. e.g. I see the correct port name in /interfaces/physical/view/233, but no port (only the switch name) in the "Peering Port" column of /interfaces/virtual/edit/67.

Probably for the same reason the port SNMP target string output by ./artisan grapher:generate-configuration -B mrtg lacks the port name.

rfc1036 commented 1 month ago

I worked around this issue by adding this to line 385 of /srv/ixpmanager/app/Models/Switcher.php:

if ($index == 2) continue;

I have no idea of what I am doing, but at least now SNMP works for all switches.

(Then I had to delete the manually-created interfaces, because they were duplicates of the ones found with SNMP.)