OSS_SNMP exception polling switch

[Kergorn]

ISSUE TYPE

Bug Report

OS

Ubuntu 20.04.5 LTS

VERSION

define( 'APPLICATION_VERSION', '6.3.0' );
define( 'APPLICATION_VERDATE', '2022110200' );

ENVIRONMENT

PHP 8.0.25 (cli) (built: Oct 28 2022 18:02:51) ( NTS )
Copyright (c) The PHP Group
Zend Engine v4.0.25, Copyright (c) Zend Technologies
    with Zend OPcache v8.0.25, Copyright (c), by Zend Technologies

ii  libapache2-mod-php8.0                1:8.0.25-1+ubuntu20.04.1+deb.sury.org+1                             amd64        server-side, HTML-embedded scripting language (Apache 2 module)
ii  php                                  2:8.0+84+ubuntu20.04.1+deb.sury.org+1                               all          server-side, HTML-embedded scripting language (default)
ii  php-common                           2:92+ubuntu20.04.1+deb.sury.org+2                                   all          Common files for PHP packages
ii  php-composer-ca-bundle               1.2.6-1                                                             all          utility library to find a path to the system CA bundle
ii  php-composer-semver                  1.5.1-1                                                             all          utilities, version constraint parsing and validation
ii  php-composer-spdx-licenses           1.5.3-1                                                             all          SPDX licenses list and validation library
ii  php-composer-xdebug-handler          1.4.0-1                                                             all          Restarts a process without Xdebug
ii  php-ds                               1.3.0-6+ubuntu20.04.1+deb.sury.org+1                                amd64        PHP extension providing efficient data structures for PHP 7
ii  php-igbinary                         3.2.6+2.0.8-1+ubuntu20.04.1+deb.sury.org+1                          amd64        igbinary PHP serializer
ii  php-json-schema                      5.2.9-1                                                             all          implementation of JSON schema
ii  php-memcache                         8.0+4.0.5.2+3.0.9~20170802.e702b5f9+-1+ubuntu20.04.1+deb.sury.org+2 amd64        memcache extension module for PHP
ii  php-memcached                        3.1.5+2.2.0-9+ubuntu20.04.1+deb.sury.org+1                          amd64        memcached extension module for PHP, uses libmemcached
ii  php-msgpack                          2.1.2+0.5.7-6+ubuntu20.04.1+deb.sury.org+1                          amd64        PHP extension for interfacing with MessagePack
ii  php-psr-container                    1.0.0-2                                                             all          Common Container Interface (PHP FIG PSR-11)
ii  php-psr-log                          1.1.2-1                                                             all          common interface for logging libraries
ii  php-rrd                              2.0.3+1.1.3-1+ubuntu20.04.1+deb.sury.org+1                          amd64        PHP bindings to rrd tool system
ii  php-symfony-console                  4.3.8+dfsg-1ubuntu1                                                 all          run tasks from the command line
ii  php-symfony-filesystem               4.3.8+dfsg-1ubuntu1                                                 all          basic filesystem utilities
ii  php-symfony-finder                   4.3.8+dfsg-1ubuntu1                                                 all          find files and directories
ii  php-symfony-process                  4.3.8+dfsg-1ubuntu1                                                 all          execute commands in sub-processes
ii  php-symfony-service-contracts        1.1.8-1                                                             all          Generic abstractions related to writing services
ii  php-yaml                             2.2.1+2.1.0+2.0.4+1.3.2-6+ubuntu20.04.1+deb.sury.org+1              amd64        YAML-1.1 parser and emitter for PHP
ii  php8.0                               1:8.0.25-1+ubuntu20.04.1+deb.sury.org+1                             all          server-side, HTML-embedded scripting language (metapackage)
ii  php8.0-bcmath                        1:8.0.25-1+ubuntu20.04.1+deb.sury.org+1                             amd64        Bcmath module for PHP
ii  php8.0-cgi                           1:8.0.25-1+ubuntu20.04.1+deb.sury.org+1                             amd64        server-side, HTML-embedded scripting language (CGI binary)
ii  php8.0-cli                           1:8.0.25-1+ubuntu20.04.1+deb.sury.org+1                             amd64        command-line interpreter for the PHP scripting language
ii  php8.0-common                        1:8.0.25-1+ubuntu20.04.1+deb.sury.org+1                             amd64        documentation, examples and common module for PHP
ii  php8.0-curl                          1:8.0.25-1+ubuntu20.04.1+deb.sury.org+1                             amd64        CURL module for PHP
ii  php8.0-ds                            1.3.0-6+ubuntu20.04.1+deb.sury.org+1                                amd64        PHP extension providing efficient data structures for PHP 7
ii  php8.0-gd                            1:8.0.25-1+ubuntu20.04.1+deb.sury.org+1                             amd64        GD module for PHP
ii  php8.0-igbinary                      3.2.6+2.0.8-1+ubuntu20.04.1+deb.sury.org+1                          amd64        igbinary PHP serializer
ii  php8.0-intl                          1:8.0.25-1+ubuntu20.04.1+deb.sury.org+1                             amd64        Internationalisation module for PHP
ii  php8.0-mbstring                      1:8.0.25-1+ubuntu20.04.1+deb.sury.org+1                             amd64        MBSTRING module for PHP
ii  php8.0-memcache                      8.0+4.0.5.2+3.0.9~20170802.e702b5f9+-1+ubuntu20.04.1+deb.sury.org+2 amd64        memcache extension module for PHP
ii  php8.0-memcached                     3.1.5+2.2.0-9+ubuntu20.04.1+deb.sury.org+1                          amd64        memcached extension module for PHP, uses libmemcached
ii  php8.0-msgpack                       2.1.2+0.5.7-6+ubuntu20.04.1+deb.sury.org+1                          amd64        PHP extension for interfacing with MessagePack
ii  php8.0-mysql                         1:8.0.25-1+ubuntu20.04.1+deb.sury.org+1                             amd64        MySQL module for PHP
ii  php8.0-opcache                       1:8.0.25-1+ubuntu20.04.1+deb.sury.org+1                             amd64        Zend OpCache module for PHP
ii  php8.0-readline                      1:8.0.25-1+ubuntu20.04.1+deb.sury.org+1                             amd64        readline module for PHP
ii  php8.0-rrd                           2.0.3+1.1.3-1+ubuntu20.04.1+deb.sury.org+1                          amd64        PHP bindings to rrd tool system
ii  php8.0-snmp                          1:8.0.25-1+ubuntu20.04.1+deb.sury.org+1                             amd64        SNMP module for PHP
ii  php8.0-xml                           1:8.0.25-1+ubuntu20.04.1+deb.sury.org+1                             amd64        DOM, SimpleXML, XML, and XSL module for PHP
ii  php8.0-yaml                          2.2.1+2.1.0+2.0.4+1.3.2-6+ubuntu20.04.1+deb.sury.org+1              amd64        YAML-1.1 parser and emitter for PHP
ii  php8.0-zip                           1:8.0.25-1+ubuntu20.04.1+deb.sury.org+1                             amd64        Zip module for PHP

CONFIGURATION

APP_DEBUG=false
APP_URL="http://XXX
APP_TIMEZONE="UTC"
APP_LOG="single"
IXP_RESELLER_ENABLED=true
IXP_AS112_UI_ACTIVE=false
IXP_FE_FRONTEND_DISABLED_LOGO=false
 IXP_FE_FRONTEND_DISABLED_PEERING_MATRIX=true
 MAIL_MAILER="smtp"
 MAIL_HOST="XXX"
 MAIL_PORT=465
 MAIL_ENCRYPTION=ssl
 MAIL_USERNAME="XXX"
 MAIL_PASSWORD="XXX"
GRAPHER_BACKENDS="mrtg|sflow"
GRAPHER_CACHE_ENABLED=true
GRAPHER_BACKEND_MRTG_DBTYPE="rrd"
GRAPHER_BACKEND_MRTG_WORKDIR="/srv/mrtg"
GRAPHER_BACKEND_MRTG_LOGDIR="/srv/mrtg"
GRAPHER_BACKEND_SFLOW_ENABLED=true
GRAPHER_BACKEND_SFLOW_ROOT="XXX"
IXP_API_JSONEXPORTSCHEMA_PUBLIC=true
CACHE_DRIVER=memcached
DOCTRINE_PROXY_AUTOGENERATE=false
DOCTRINE_CACHE=memcached
DOCTRINE_CACHE_NAMESPACE=IXPMANAGERNAMESPACE
IXP_IRRDB_BGPQ3_PATH=/usr/bin/bgpq3

SUMMARY

Hi, Team.

I have discovered a problem with snmp polling some of my switches (only Arista and only 4 switches from many others). SNMP works itself, I have checked it via snmpwalk/snmpget. The situation looks like this:

0. SNMP-polling doesn't work through the shedule and doesn't work via web-interface of IXPm.
1. I do manual poll (/srv/ixpmanager/artisan switch:snmp-poll --log ar7508-XXX)
2. And I see ERROR: OSS_SNMP exception polling switch ar7508-XXXt by SNMP

After this I have seen that OSS_SNMP has tried to start process (information from log is below). Also, I have seen only one added new interface without information from switch in web-interface of IXP and in the MySQL.

Worth adding that these 4 switches were already added earlier (in version 5.7.0) and statistics were successfully collected from them. Unfortunately, I have only discovered the problem now. I can't say for sure whether it appeared: after the transition to 6.2.0 or later. I also tried to update to 6.3.0 but that didn't help.

What should I do next for debug? Thanks!

STEPS TO REPRODUCE

EXPECTED RESULTS

ACTUAL RESULTS

IMPORTANCE

RELEVANT LOGS

Polling ar7508-XXX with SNMP request to ar7508-XXX ERROR: OSS_SNMP exception polling switch ar7508-XXX by SNMP

[2022-11-02 12:48:51] production.INFO: [ar7508-XXX] Platform: Found Model: DCS-7508
[2022-11-02 12:48:51] production.INFO: [ar7508-XXX] Platform: Found Os:
[2022-11-02 12:48:51] production.INFO: [ar7508-XXX] Platform: Updating OsDate from to Unknown
[2022-11-02 12:48:51] production.INFO: [ar7508-XXX] Platform: Found OsVersion: 4.24.5M
[2022-11-02 12:48:51] production.INFO: [ar7508-XXX] Platform: Found SerialNumber: (not implemented)
[2022-11-02 12:48:53] production.INFO: - ar7508-XXX - found pre-existing port for ifIndex 3001
[2022-11-02 12:48:53] production.INFO: [ar7508-XXX]:Ethernet3/1 [Index: 3001] Updating Name from [] to [Ethernet3/1]
[2022-11-02 12:48:54] production.INFO: [ar7508-XXX]:Ethernet3/1 [Index: 3001] Updating IfName from [] to [Ethernet3/1]

[barryo]

Hi @Kergorn

Unfortunately, I have only discovered the problem now. I can't say for sure whether it appeared: after the transition to 6.2.0 or later. I also tried to update to 6.3.0 but that didn't help.

I'm guessing this is unlikely to be related to a specific version / upgrade right now. We've been running the code that makes up v6.3 at INEX for months on Arista's with no issue.

We'll need to dig into what the exception is.

The logs you've provided come from the command line run I think (or cron). Looks like the error comes from this line.

Could you edit that file so it looks like:

                } catch( Exception $e ) {
                    if( $sPolled ){
                        $this->error("ERROR: OSS_SNMP exception polling switch {$s->name} by SNMP");
                        throw $e;  // <<< new line, only change
                    } else {

And send the output?

[Kergorn]

Thanks for your advice! Okay, I've done it and...

Could not perform walk for OID .1.3.6.1.2.1.31.1.1.1.18

But if I do that: snmpwalk -v 2c -c XXX ar7508-XXX .1.3.6.1.2.1.31.1.1.1.18

Then everything is fine and I see a correct output from the switch.

[Kergorn]

Hi @Kergorn

Unfortunately, I have only discovered the problem now. I can't say for sure whether it appeared: after the transition to 6.2.0 or later. I also tried to update to 6.3.0 but that didn't help.

I'm guessing this is unlikely to be related to a specific version / upgrade right now. We've been running the code that makes up v6.3 at INEX for months on Arista's with no issue.

We'll need to dig into what the exception is.

The logs you've provided come from the command line run I think (or cron). Looks like the error comes from this line.

Could you edit that file so it looks like:

                } catch( Exception $e ) {
                    if( $sPolled ){
                        $this->error("ERROR: OSS_SNMP exception polling switch {$s->name} by SNMP");
                        throw $e;  // <<< new line, only change
                    } else {

And send the output?

I found the problem, it has to do with the lenght of interface's description (ifAlias). I'll see a error if the description is too long. What part of a code is responsible for this?

[barryo]

Hi,

can you send the output from storage/logs/laravel.log when you've made the change above and hit the error? The trace will identify the specific line.

[Kergorn]

[2022-11-16 10:41:15] production.INFO: [ar7280-XXX]:Ethernet1 [Index: 1] Updating Name from [] to [Ethernet1]  
[2022-11-16 10:41:15] production.INFO: [ar7280-XXX]:Ethernet1 [Index: 1] Updating IfName from [] to [Ethernet1]  
[2022-11-16 10:41:21] production.ERROR: Could not perform walk for OID .1.3.6.1.2.1.31.1.1.1.18 {"exception":"[object] (OSS_SNMP\\Exception(code: 0): Could not perform walk for OID .1.3.6.1.2.1.31.1.1.1.18 at /srv/ixpmanager/vendor/opensolutions/oss-snmp/src/OSS_SNMP/SNMP.php:327)
[stacktrace]
#0 /srv/ixpmanager/vendor/opensolutions/oss-snmp/src/OSS_SNMP/MIBS/Iface.php(461): OSS_SNMP\\SNMP->walk1d()
#1 /srv/ixpmanager/app/Models/SwitchPort.php(439): OSS_SNMP\\MIBS\\Iface->aliases()
#2 /srv/ixpmanager/app/Models/Switcher.php(430): IXP\\Models\\SwitchPort->snmpUpdate()
#3 /srv/ixpmanager/app/Console/Commands/Switches/SnmpPoll.php(107): IXP\\Models\\Switcher->snmpPollSwitchPorts()
#4 /srv/ixpmanager/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(36): IXP\\Console\\Commands\\Switches\\SnmpPoll->handle()
#5 /srv/ixpmanager/vendor/laravel/framework/src/Illuminate/Container/Util.php(40): Illuminate\\Container\\BoundMethod::Illuminate\\Container\\{closure}()
#6 /srv/ixpmanager/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(93): Illuminate\\Container\\Util::unwrapIfClosure()
#7 /srv/ixpmanager/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(37): Illuminate\\Container\\BoundMethod::callBoundMethod()
#8 /srv/ixpmanager/vendor/laravel/framework/src/Illuminate/Container/Container.php(653): Illuminate\\Container\\BoundMethod::call()
#9 /srv/ixpmanager/vendor/laravel/framework/src/Illuminate/Console/Command.php(136): Illuminate\\Container\\Container->call()
#10 /srv/ixpmanager/vendor/symfony/console/Command/Command.php(298): Illuminate\\Console\\Command->execute()
#11 /srv/ixpmanager/vendor/laravel/framework/src/Illuminate/Console/Command.php(121): Symfony\\Component\\Console\\Command\\Command->run()
#12 /srv/ixpmanager/vendor/symfony/console/Application.php(1028): Illuminate\\Console\\Command->run()
#13 /srv/ixpmanager/vendor/symfony/console/Application.php(299): Symfony\\Component\\Console\\Application->doRunCommand()
#14 /srv/ixpmanager/vendor/symfony/console/Application.php(171): Symfony\\Component\\Console\\Application->doRun()
#15 /srv/ixpmanager/vendor/laravel/framework/src/Illuminate/Console/Application.php(94): Symfony\\Component\\Console\\Application->run()
#16 /srv/ixpmanager/vendor/laravel/framework/src/Illuminate/Foundation/Console/Kernel.php(129): Illuminate\\Console\\Application->run()
#17 /srv/ixpmanager/artisan(39): Illuminate\\Foundation\\Console\\Kernel->handle()
#18 {main}
"} 

[Kergorn]

Hi,

can you send the output from storage/logs/laravel.log when you've made the change above and hit the error? The trace will identify the specific line.

Hi Barry, Could you help me with finding a point of crash?

[barryo]

Sorry @Kergorn - could you let me know the length of the alias string?

[Kergorn]

Sorry @Kergorn - could you let me know the length of the alias string?

Hmm. The longest example I have is 234 symbols (Oo) -)

[barryo]

That's pretty descriptive 😉

DB can hold 255 so I'll need to have a closer look at the code. That's 234 bytes, right? No unicode etc?

[Kergorn]

That's pretty descriptive 😉

DB can hold 255 so I'll need to have a closer look at the code. That's 234 bytes, right? No unicode etc?

That's right, 234 bytes without unicode.

[Kergorn]

That's pretty descriptive 😉

DB can hold 255 so I'll need to have a closer look at the code. That's 234 bytes, right? No unicode etc?

Hi Barry, Do you have any news?

[barryo]

Hi @Kergorn

I don't think this has anything to do with IXP Manager or OSS_SNMP.

The issue will probably be further down the chain with php-snmp or libsnmp or the snmp implementation on the switch. Essentially, it appears ifAlias is not meant to be more than 64 characters per rfc2863 (see pg 43):

ifAlias   OBJECT-TYPE
    SYNTAX      DisplayString (SIZE(0..64))
    MAX-ACCESS  read-write

I suggest you reduce the content and that should solve the problem.

[barryo]

[closed this issue as it is not an ixpm bug but feel free to follow up here]