search

inex / IXP-Manager

Full stack web application powering peering at over 200 Internet Exchange Points (IXPs) globally.
https://www.ixpmanager.org/
GNU General Public License v2.0
377 stars 161 forks source link

Interface falls over when editing location #291

Closed listerr closed 7 years ago

listerr commented 7 years ago

  1. Insists on there being a valid "tag" - what is meant to be entered in the tag field?

  2. Although it is meant to be fixed in #258 - and looking at our code, I think we have this fixed version, the interface is still crashing when we try to update a location. Error message seems to imply some backend database constraint preventing the update?

2017-01-31T16:51:21+00:00 ERR (3): ERROR - FAILED: User 2 edited a location object with id 4 ErrorException: PDOStatement::execute(): MySQL server has gone away in /srv/ixp4/vendor/doctrine/dbal/lib/Doctrine/DBAL/Driver/PDOStatement.php:91 Stack trace:

0 [internal function]: Illuminate\Foundation\Bootstrap\HandleExceptions->handleError(2, 'PDOStatement::e...', '/srv/ixp4/vendo...', 91, Array)

1 /srv/ixp4/vendor/doctrine/dbal/lib/Doctrine/DBAL/Driver/PDOStatement.php(91): PDOStatement->execute(NULL)

2 /srv/ixp4/vendor/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(987): Doctrine\DBAL\Driver\PDOStatement->execute()

3 /srv/ixp4/vendor/doctrine/orm/lib/Doctrine/ORM/Persisters/Entity/BasicEntityPersister.php(480): Doctrine\DBAL\Connection->executeUpdate('UPDATE location...', Array, Array)

4 /srv/ixp4/vendor/doctrine/orm/lib/Doctrine/ORM/Persisters/Entity/BasicEntityPersister.php(366): Doctrine\ORM\Persisters\Entity\BasicEntityPersister->updateTable(Object(Entities\Location), 'location', Array, NULL)

5 /srv/ixp4/vendor/doctrine/orm/lib/Doctrine/ORM/UnitOfWork.php(1069): Doctrine\ORM\Persisters\Entity\BasicEntityPersister->update(Object(Entities\Location))

6 /srv/ixp4/vendor/doctrine/orm/lib/Doctrine/ORM/UnitOfWork.php(384): Doctrine\ORM\UnitOfWork->executeUpdates(Object(Doctrine\ORM\Mapping\ClassMetadata))

7 /srv/ixp4/vendor/doctrine/orm/lib/Doctrine/ORM/EntityManager.php(356): Doctrine\ORM\UnitOfWork->commit(NULL)

8 /srv/ixp4/vendor/opensolutions/oss-framework/src/OSS/Controller/Action/Trait/Doctrine2Frontend.php(623): Doctrine\ORM\EntityManager->flush()

9 /srv/ixp4/vendor/opensolutions/oss-framework/src/OSS/Controller/Action/Trait/Doctrine2Frontend.php(707): IXP_Controller_Frontend->addProcessForm(Object(IXP_Form_Location), Object(Entities\Location), true)

10 /srv/ixp4/vendor/zendframework/zendframework1/library/Zend/Controller/Action.php(516): IXP_Controller_Frontend->addAction()

11 /srv/ixp4/vendor/zendframework/zendframework1/library/Zend/Controller/Dispatcher/Standard.php(308): Zend_Controller_Action->dispatch('addAction')

12 /srv/ixp4/vendor/zendframework/zendframework1/library/Zend/Controller/Front.php(954): Zend_Controller_Dispatcher_Standard->dispatch(Object(Zend_Controller_Request_Http), Object(Zend_Controller_Response_Http))

13 /srv/ixp4/vendor/zendframework/zendframework1/library/Zend/Application/Bootstrap/Bootstrap.php(105): Zend_Controller_Front->dispatch()

14 /srv/ixp4/vendor/zendframework/zendframework1/library/Zend/Application.php(384): Zend_Application_Bootstrap_Bootstrap->run()

15 /srv/ixp4/app/Exceptions/Handler.php(59): Zend_Application->run()

16 /srv/ixp4/vendor/laravel/framework/src/Illuminate/Routing/Pipeline.php(80): IXP\Exceptions\Handler->render(Object(Illuminate\Http\Request), Object(Symfony\Component\HttpKernel\Exception\NotFoundHttpException))

17 /srv/ixp4/vendor/laravel/framework/src/Illuminate/Routing/Pipeline.php(54): Illuminate\Routing\Pipeline->handleException(Object(Illuminate\Http\Request), Object(Symfony\Component\HttpKernel\Exception\NotFoundHttpException))

18 /srv/ixp4/bootstrap/cache/compiled.php(3271): Illuminate\Routing\Pipeline->Illuminate\Routing{closure}(Object(Illuminate\Http\Request))

19 [internal function]: Illuminate\Foundation\Http\Middleware\CheckForMaintenanceMode->handle(Object(Illuminate\Http\Request), Object(Closure))

20 /srv/ixp4/bootstrap/cache/compiled.php(9914): call_user_func_array(Array, Array)

21 [internal function]: Illuminate\Pipeline\Pipeline->Illuminate\Pipeline{closure}(Object(Illuminate\Http\Request))

22 /srv/ixp4/vendor/laravel/framework/src/Illuminate/Routing/Pipeline.php(32): call_user_func(Object(Closure), Object(Illuminate\Http\Request))

23 [internal function]: Illuminate\Routing\Pipeline->Illuminate\Routing{closure}(Object(Illuminate\Http\Request))

24 /srv/ixp4/bootstrap/cache/compiled.php(9899): call_user_func(Object(Closure), Object(Illuminate\Http\Request))

25 /srv/ixp4/bootstrap/cache/compiled.php(2363): Illuminate\Pipeline\Pipeline->then(Object(Closure))

26 /srv/ixp4/bootstrap/cache/compiled.php(2347): Illuminate\Foundation\Http\Kernel->sendRequestThroughRouter(Object(Illuminate\Http\Request))

27 /srv/ixp4/public/index.php(97): Illuminate\Foundation\Http\Kernel->handle(Object(Illuminate\Http\Request))

28 {main}

Next Doctrine\DBAL\DBALException: An exception occurred while executing 'UPDATE location SET pdb_facility_id = ?, tag = ? WHERE id = ?' with params ["45", "hex", 4]:

PDOStatement::execute(): MySQL server has gone away in /srv/ixp4/vendor/doctrine/dbal/lib/Doctrine/DBAL/DBALException.php:119 Stack trace:

0 /srv/ixp4/vendor/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(996): Doctrine\DBAL\DBALException::driverExceptionDuringQuery(Object(Doctrine\DBAL\Driver\PDOMySql\Driver), Object(ErrorException), 'UPDATE location...', Array)

1 /srv/ixp4/vendor/doctrine/orm/lib/Doctrine/ORM/Persisters/Entity/BasicEntityPersister.php(480): Doctrine\DBAL\Connection->executeUpdate('UPDATE location...', Array, Array)

2 /srv/ixp4/vendor/doctrine/orm/lib/Doctrine/ORM/Persisters/Entity/BasicEntityPersister.php(366): Doctrine\ORM\Persisters\Entity\BasicEntityPersister->updateTable(Object(Entities\Location), 'location', Array, NULL)

3 /srv/ixp4/vendor/doctrine/orm/lib/Doctrine/ORM/UnitOfWork.php(1069): Doctrine\ORM\Persisters\Entity\BasicEntityPersister->update(Object(Entities\Location))

4 /srv/ixp4/vendor/doctrine/orm/lib/Doctrine/ORM/UnitOfWork.php(384): Doctrine\ORM\UnitOfWork->executeUpdates(Object(Doctrine\ORM\Mapping\ClassMetadata))

5 /srv/ixp4/vendor/doctrine/orm/lib/Doctrine/ORM/EntityManager.php(356): Doctrine\ORM\UnitOfWork->commit(NULL)

6 /srv/ixp4/vendor/opensolutions/oss-framework/src/OSS/Controller/Action/Trait/Doctrine2Frontend.php(623): Doctrine\ORM\EntityManager->flush()

7 /srv/ixp4/vendor/opensolutions/oss-framework/src/OSS/Controller/Action/Trait/Doctrine2Frontend.php(707): IXP_Controller_Frontend->addProcessForm(Object(IXP_Form_Location), Object(Entities\Location), true)

8 /srv/ixp4/vendor/zendframework/zendframework1/library/Zend/Controller/Action.php(516): IXP_Controller_Frontend->addAction()

9 /srv/ixp4/vendor/zendframework/zendframework1/library/Zend/Controller/Dispatcher/Standard.php(308): Zend_Controller_Action->dispatch('addAction')

10 /srv/ixp4/vendor/zendframework/zendframework1/library/Zend/Controller/Front.php(954): Zend_Controller_Dispatcher_Standard->dispatch(Object(Zend_Controller_Request_Http), Object(Zend_Controller_Response_Http))

11 /srv/ixp4/vendor/zendframework/zendframework1/library/Zend/Application/Bootstrap/Bootstrap.php(105): Zend_Controller_Front->dispatch()

12 /srv/ixp4/vendor/zendframework/zendframework1/library/Zend/Application.php(384): Zend_Application_Bootstrap_Bootstrap->run()

13 /srv/ixp4/app/Exceptions/Handler.php(59): Zend_Application->run()

14 /srv/ixp4/vendor/laravel/framework/src/Illuminate/Routing/Pipeline.php(80): IXP\Exceptions\Handler->render(Object(Illuminate\Http\Request), Object(Symfony\Component\HttpKernel\Exception\NotFoundHttpException))

15 /srv/ixp4/vendor/laravel/framework/src/Illuminate/Routing/Pipeline.php(54): Illuminate\Routing\Pipeline->handleException(Object(Illuminate\Http\Request), Object(Symfony\Component\HttpKernel\Exception\NotFoundHttpException))

16 /srv/ixp4/bootstrap/cache/compiled.php(3271): Illuminate\Routing\Pipeline->Illuminate\Routing{closure}(Object(Illuminate\Http\Request))

17 [internal function]: Illuminate\Foundation\Http\Middleware\CheckForMaintenanceMode->handle(Object(Illuminate\Http\Request), Object(Closure))

18 /srv/ixp4/bootstrap/cache/compiled.php(9914): call_user_func_array(Array, Array)

19 [internal function]: Illuminate\Pipeline\Pipeline->Illuminate\Pipeline{closure}(Object(Illuminate\Http\Request))

20 /srv/ixp4/vendor/laravel/framework/src/Illuminate/Routing/Pipeline.php(32): call_user_func(Object(Closure), Object(Illuminate\Http\Request))

21 [internal function]: Illuminate\Routing\Pipeline->Illuminate\Routing{closure}(Object(Illuminate\Http\Request))

22 /srv/ixp4/bootstrap/cache/compiled.php(9899): call_user_func(Object(Closure), Object(Illuminate\Http\Request))

23 /srv/ixp4/bootstrap/cache/compiled.php(2363): Illuminate\Pipeline\Pipeline->then(Object(Closure))

24 /srv/ixp4/bootstrap/cache/compiled.php(2347): Illuminate\Foundation\Http\Kernel->sendRequestThroughRouter(Object(Illuminate\Http\Request))

25 /srv/ixp4/public/index.php(97): Illuminate\Foundation\Http\Kernel->handle(Object(Illuminate\Http\Request))

26 {main}

listerr commented 7 years ago

Okay,

Checked a bit more in to this. mySQL is throwing a wobbly when I try to update this specific field:

mysql> UPDATE location SET notes = 'foo' where id=1;
Query OK, 1 row affected (0.05 sec)
Rows matched: 1  Changed: 1  Warnings: 0

mysql> UPDATE location SET pdb_facility_id = '34' where id=1;
`ERROR 2013 (HY000): Lost connection to MySQL server during query

mysql> SHOW GLOBAL STATUS LIKE  'Aborted_connects';
+------------------+-------+
| Variable_name    | Value |
+------------------+-------+
| Aborted_connects | 0     |
+------------------+-------+
1 row in set (0.01 sec)

mysql> UPDATE location SET pdb_facility_id = '34' where id=1;
ERROR 2013 (HY000): Lost connection to MySQL server during query
listerr commented 7 years ago 
16:56:30 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

key_buffer_size=16777216
read_buffer_size=131072
max_used_connections=2
max_threads=151
thread_count=1
connection_count=1
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 346701 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x7f597d4451b0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7f597a0e4e98 thread_stack 0x30000
/usr/sbin/mysqld(my_print_stacktrace+0x20)[0x7f597a792eb0]
/usr/sbin/mysqld(handle_fatal_signal+0x3d5)[0x7f597a67c1f5]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x10330)[0x7f597940a330]
/usr/sbin/mysqld(+0x58a3bc)[0x7f597a8293bc]
/usr/sbin/mysqld(+0x58a72a)[0x7f597a82972a]
/usr/sbin/mysqld(+0x577f28)[0x7f597a816f28]
/usr/sbin/mysqld(+0x566a3e)[0x7f597a805a3e]
/usr/sbin/mysqld(_ZN7handler13ha_update_rowEPKhPh+0x55)[0x7f597a682bb5]
/usr/sbin/mysqld(_Z12mysql_updateP3THDP10TABLE_LISTR4ListI4ItemES6_PS4_jP8st_ordery15enum_duplicatesbPySB_+0x10a8)[0x7f597a6045d8]
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x3cdf)[0x7f597a5997df]
/usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x160)[0x7f597a59c8b0]
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x1e67)[0x7f597a59eeb7]
/usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x176)[0x7f597a62c146]
/usr/sbin/mysqld(handle_one_connection+0x41)[0x7f597a62c1a1]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x8184)[0x7f5979402184]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f5978b2537d]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (7f5954294d30): is an invalid pointer
Connection ID (thread ID): 48
Status: NOT_KILLED

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
170202 16:56:31 [Warning] Using unique option prefix myisam-recover instead of myisam-recover-options is deprecated and will be removed in a future release. Please use the full name instead.
170202 16:56:31 [Note] Plugin 'FEDERATED' is disabled.
170202 16:56:31 InnoDB: The InnoDB memory heap is disabled
170202 16:56:31 InnoDB: Mutexes and rw_locks use GCC atomic builtins
170202 16:56:31 InnoDB: Compressed tables use zlib 1.2.8
170202 16:56:31 InnoDB: Using Linux native AIO
170202 16:56:31 InnoDB: Initializing buffer pool, size = 128.0M
170202 16:56:31 InnoDB: Completed initialization of buffer pool
170202 16:56:31 InnoDB: highest supported file format is Barracuda.
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
170202 16:56:31  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
170202 16:56:31  InnoDB: Waiting for the background threads to start
170202 16:56:32 InnoDB: 5.5.54 started; log sequence number 52320135426
170202 16:56:32 [Note] Server hostname (bind-address): '0.0.0.0'; port: 3306
170202 16:56:32 [Note]   - '0.0.0.0' resolves to '0.0.0.0';
170202 16:56:32 [Note] Server socket created on IP: '0.0.0.0'.
170202 16:56:32 [Note] Event Scheduler: Loaded 0 events
170202 16:56:32 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.5.54-0ubuntu0.14.04.1'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  (Ubuntu)
170202 16:56:33 [ERROR] Build InnoDB index translation table for Table ./ixp/bgpsessiondata failed
170202 16:56:33 [ERROR] Table ixp/bgpsessiondata contains 1 indexes inside InnoDB, which is different from the number of indexes 2 defined in the MySQL
170202 16:56:33 [ERROR] Innodb could not find key n:o 1 with name idx_timestamp from dict cache for table ixp/bgpsessiondata
170202 16:56:33 [ERROR] Table ixp/bgpsessiondata contains fewer indexes inside InnoDB than are defined in the MySQL .frm file. Have you mixed up .frm files from different installations? See http://dev.mysql.com/doc/refman/5.5/en/innodb-troubleshooting.html

170202 16:56:33 [ERROR] Table ixp/bgpsessiondata contains 1 indexes inside InnoDB, which is different from the number of indexes 2 defined in the MySQL
170202 16:56:33 [ERROR] Innodb could not find key n:o 1 with name idx_timestamp from dict cache for table ixp/bgpsessiondata
170202 16:56:33 [ERROR] Table ixp/bgpsessiondata contains fewer indexes inside InnoDB than are defined in the MySQL .frm file. Have you mixed up .frm files from different installations? See http://dev.mysql.com/doc/refman/5.5/en/innodb-troubleshooting.html
nickhilliard commented 7 years ago

This is not a good sign and looks like either a mysql bug or db corruption, but it's not an IXP Manager problem.

As a matter of urgency, you would probably want to consider a mysqldump on your ixp manager db. Once this is done - and not before - it would probably be a good idea to try an innodb repair operation. There are guidelines here:

https://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html

If this doesn't fix the problem, it would probably be a good idea to blow away the entire db and restore from mysqldump.

The bgpsessiondata table seems to be hosed, but if this is the limit of the problem, I wouldn't worry about it - it will rebuild itself in a couple of days.

listerr commented 7 years ago

Looking in the error.log now shows various warnings when it restarts, such as:

170202 16:26:19 [ERROR] Cannot find or open table ixp3/irrdb_asn from
the internal data dictionary of InnoDB though the .frm file for the
table exists. Maybe you have deleted and recreated InnoDB data
files but have forgotten to delete the corresponding .frm files
of InnoDB tables, or you have moved .frm files to another database?
or, the table contains indexes that this version of the engine
doesn't support.
See http://dev.mysql.com/doc/refman/5.5/en/innodb-troubleshooting.html
how you can resolve the problem.

seems to relate to old tables for the v3 install though.

Agree some sort of innodb issue reading the various links.

This would also seem to be the reason that our SFLOW graphs stop updating and we have to restart things, usually every couple of days. The script loses the connection to mysql and dies.

nickhilliard commented 7 years ago

you should be able to delete the ixp3 db if you're finished with it.

The sflow scripts shouldn't die randomly. They certainly don't on the inex installation.

listerr commented 7 years ago

Thanks Nick.

What seems to happen in our case is that if mysql is not available, /srv/IXP-Manager/tools/runtime/sflow/sflow-to-rrd-handler falls over. Attempting to restart this script by itself doesn't work, as it complains that various things are already bound to ports.

v4 bind() failed, port = 5500 : Address already in use
unable to open UDP read socket
Oops, input pipe died. Aborting.

So stopping all the sflow related processes and starting again in order, gets it running again.

/usr/bin/nohup /usr/local/bin/sflowtool -4 -p 6343 -f 127.0.0.1/5500 -f 127.0.0.1/5501 -f 127.0.0.1/5502 &>/dev/null &
/usr/bin/nohup /srv/IXP-Manager/tools/runtime/sflow/sflow-to-rrd-handler &>/dev/null &
/srv/IXP-Manager/tools/runtime/sflow/control-sflow-detect-ixp-bgp-sessions start
/srv/IXP-Manager/tools/runtime/sflow/control-sflow-to-graphite-handler start

Recently we added a check to Nagios to check if the rrd files have stopped being updated. (since the process check doesn't seem able to check sflow-to-rrd-handler itself for some reason, and it does not always die straight away - seems to have a couple of goes and then dies.)

(mysql update of doom issued as mentioned above.)

root@sflow:/home/robl# DBD::mysql::st execute failed: MySQL server has gone away at /usr/local/bin/sflow-detect-ixp-bgp-sessions line 131, <SFLOWTOOL> line 177873.
DBD::mysql::st execute failed: MySQL server has gone away at /usr/local/bin/sflow-detect-ixp-bgp-sessions line 131, <SFLOWTOOL> line 177873.

root@sflow:/home/robl# !ps
ps xaguwww | grep sflow
root     28722  0.9  0.0   4276   672 pts/0    S    12:13   0:00 /usr/local/bin/sflowtool -4 -p 6343 -f 127.0.0.1/5500 -f 127.0.0.1/5501 -f 127.0.0.1/5502
root     28723 13.0  0.7 228476 32352 pts/0    S    12:13   0:12 /usr/bin/perl -w /srv/IXP-Manager/tools/runtime/sflow/sflow-to-rrd-handler
root     28725  1.5  0.0   4148   732 pts/0    S    12:13   0:01 /usr/local/bin/sflowtool -4 -p 5500 -l
root     28728  3.2  0.3 146224 15612 ?        S    12:13   0:03 /usr/bin/perl -w /usr/local/bin/sflow-detect-ixp-bgp-sessions
root     28730  2.1  0.0   4148   692 ?        S    12:13   0:02 /usr/local/bin/sflowtool -4 -p 5501 -l
root     28733  6.4  1.1 246420 47992 ?        S    12:13   0:05 /usr/bin/perl -w /usr/local/bin/sflow-to-graphite-handler
root     28735  2.2  0.0   4148   712 ?        S    12:13   0:02 /usr/local/bin/sflowtool -4 -p 5502 -l
root     28754  0.0  0.0  12728  2056 pts/0    S+   12:15   0:00 grep sflow

root@sflow:/home/robl# ps xaguwww | grep sflow
root     28722  0.9  0.0   4276   672 pts/0    S    12:13   0:01 /usr/local/bin/sflowtool -4 -p 6343 -f 127.0.0.1/5500 -f 127.0.0.1/5501 -f 127.0.0.1/5502
root     28723 11.6  0.7 228476 32352 pts/0    S    12:13   0:12 /usr/bin/perl -w /srv/IXP-Manager/tools/runtime/sflow/sflow-to-rrd-handler
root     28725  1.6  0.0   4148   732 pts/0    S    12:13   0:01 /usr/local/bin/sflowtool -4 -p 5500 -l
root     28728  2.8  0.3 146224 15612 ?        S    12:13   0:03 /usr/bin/perl -w /usr/local/bin/sflow-detect-ixp-bgp-sessions
root     28730  2.1  0.0   4148   692 ?        S    12:13   0:02 /usr/local/bin/sflowtool -4 -p 5501 -l
root     28733  6.3  1.1 246420 47992 ?        S    12:13   0:06 /usr/bin/perl -w /usr/local/bin/sflow-to-graphite-handler
root     28735  2.3  0.0   4148   712 ?        S    12:13   0:02 /usr/local/bin/sflowtool -4 -p 5502 -l
root     28756  0.0  0.0  12728  2132 pts/0    S+   12:15   0:00 grep sflow

root@sflow:/home/robl# DBD::mysql::st execute failed: MySQL server has gone away at /usr/local/bin/sflow-to-graphite-handler line 445, <SFLOWTOOL> line 257048.
DBD::mysql::st execute failed: MySQL server has gone away at /usr/local/bin/sflow-to-graphite-handler line 445, <SFLOWTOOL> line 257048.

[2]+  Alarm clock             /usr/bin/nohup /srv/IXP-Manager/tools/runtime/sflow/sflow-to-rrd-handler &> /dev/null

root@sflow:/home/robl# ps xaguwww | grep sflow
root     28722  0.9  0.0   4276   672 pts/0    S    12:13   0:01 /usr/local/bin/sflowtool -4 -p 6343 -f 127.0.0.1/5500 -f 127.0.0.1/5501 -f 127.0.0.1/5502
root     28725  1.9  0.0   4148   732 pts/0    S    12:13   0:03 /usr/local/bin/sflowtool -4 -p 5500 -l
root     28728  1.8  0.3 146224 15612 ?        S    12:13   0:03 /usr/bin/perl -w /usr/local/bin/sflow-detect-ixp-bgp-sessions
root     28730  2.2  0.0   4148   692 ?        S    12:13   0:03 /usr/local/bin/sflowtool -4 -p 5501 -l
root     28733  4.9  1.1 246420 47992 ?        S    12:13   0:07 /usr/bin/perl -w /usr/local/bin/sflow-to-graphite-handler
root     28735  2.3  0.0   4148   712 ?        S    12:13   0:03 /usr/local/bin/sflowtool -4 -p 5502 -l

Anyway. straying onto a different issue in the process of diagnosing another one. Probably we should close this issue and look into this separately.

nickhilliard commented 7 years ago

The script already attempts to deal with mysql failures, but if you start it with nohup, it's likely that this would cause problems if sflow-to-rrd-handler tries to kill the sflowtool subprocess.

You should start sflow-to-rrd-handler with the control-sflow-to-rrd-handler script rather than depending on nohup.