search

SNAS / openbmp-mysql-consumer

OpenBMP Kafka MySQL consumer
http://www.openbmp.org
Eclipse Public License 1.0
11 stars 10 forks source link

DB does not get populated with stats #12

Open gunarser opened 6 years ago

gunarser commented 6 years ago

Hi,

We just started testing OpenBMP with the newest stable packages (Kafka, MySQL consumer and UI) but some data does not appear in SNAS GUI. For example, Tops page show only Top20 graphs for Withdraws but Update graphs are empty. Also, peer information is incorrectly displayed. Half of peers are reported down when in reality they are up.

It seems that data is not parsed correctly and/or populated in database.

How to troubleshoot this issue?

BR, Gunars

TimEvens commented 6 years ago

Hi @gunarser ,

Those are two different sources/handling of data.

  1. the updates over time report in the tops view is driven by the path_attr_log table. This table is updated by a mysql trigger. The trigger checks if there is a path attribute hash change (e.g. change in attribute value from previous) and if so logs it. If no change, it doesn’t log it. You can of course change the trigger to log dups if you want. If, for example, you withdraw then advertise repeatedly, the trigger will only log the withdrawals as changes since the attributes are the same when it is advertised again. If you really want to see every update, even when duplicated, you can update the trigger. Maybe in your case you want to log the flapping. In this case the trigger can be updated to log a new update when the previous is changed from isWithdrawn True to False. When you expect update in tops, are you sure your updates are changes in attributes?

  2. Half the peers being reported down is normally an issue with offsets starting after the peer ups. There is a new config setting in the mysql consumer to set the starting offset as latest or earliest. There was a default config mistake a few weeks back that set this value to latest when it should have been earliest. This has since been fixed. Are you running the latest container/build?

gunarser commented 6 years ago

Hi Tim,

Thank you for the answers!

Currently we are not running OpenBMP containter. We did a manual installation on Ubuntu 16.04 server. We have the following packages installed:

Are there newer builds available for the above programs?

When we tested container version we got stats about Updates over time but were missing other stats that we see now. I restarted BMP sessions on our routers but still some peers appear down when in fact they are up (although more now appear up).

I attached to the email several screenshots showing the issues. Could you please guide us in the right direction to what might be wrong with ourt configuration?

Thank you in advance, Gunars

On 01.03.2018 08:33, Tim Evens wrote:

Hi @gunarser https://github.com/gunarser ,

Those are two different sources/handling of data.

1.

the updates over time report in the tops view is driven by the
path_attr_log table. This table is updated by a mysql trigger. The
trigger checks if there is a path attribute hash change (e.g.
change in attribute value from previous) and if so logs it. If no
change, it doesn’t log it. You can of course change the trigger to
log dups if you want. If, for example, you withdraw then advertise
repeatedly, the trigger will only log the withdrawals as changes
since the attributes are the same when it is advertised again. If
you really want to see every update, even when duplicated, you can
update the trigger. Maybe in your case you want to log the
flapping. In this case the trigger can be updated to log a new
update when the previous is changed from isWithdrawn True to
False. When you expect update in tops, are you sure your updates
are changes in attributes?

2.

Half the peers being reported down is normally an issue with
offsets starting after the peer ups. There is a new config setting
in the mysql consumer to set the starting offset as latest or
earliest. There was a default config mistake a few weeks back that
set this value to latest when it should have been earliest. This
has since been fixed. Are you running the latest container/build?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/OpenBMP/openbmp-mysql-consumer/issues/12#issuecomment-369488772, or mute the thread https://github.com/notifications/unsubscribe-auth/Ai_-Dvk2Kzt8upxZX87_sA8gngtyJn_Eks5tZ5YigaJpZM4SPPc9.

TimEvens commented 6 years ago

Hi @gunarser ,

Those are current, but there is a newer one that adds large community support.

There can be a few reasons why the peer up isn’t making its way to the dB.

Can you also confirm the peer up messages are in Kafka? You can do this using kafkacat -u -b localhost:9092 -t openbmp.parsed.peer -o -100

gunarser commented 6 years ago

Hi Tim,

In openbmpd.log file I see 'PEER UP Received' messages when the OpenBMP session is re-established. If I understand correctly that is the default behavior when BMP session is brought up on Juniper routers and they send these initial 'PEER UP' messages? In the same log file I see the following messages:

<...>Ignoring deprecated extended community 128/0<...> <...>Ignoring deprecated extended community 128/1<...>

Should we worry about them?

The 'peer messages' count increments. Here is the message from yesterday and today for comparison:

2018-03-12 12:09:13 [main] INFO org.openbmp.MySQLConsumerApp - peer messages: 0 2018-03-12 12:14:13 [main] INFO org.openbmp.MySQLConsumerApp - peer messages: 35 <...> 2018-03-13 11:14:14 [main] INFO org.openbmp.MySQLConsumerApp - peer messages: 47 2018-03-13 11:19:14 [main] INFO org.openbmp.MySQLConsumerApp - peer messages: 47

I deactivated and re-activated one peer and this is what I see in Kafka consumer:

$ kafkacat -u -b localhost:9092 -t openbmp.parsed.peer -o -100 % Auto-selecting Consumer mode (use -P or -C to override) V: 1.7 C_HASH_ID: 1a3e29f5a8c84ea7bc519f159166d4b3 T: peer L: 224 R: 1

down 19 de26e4d608509af1abca9df8579f8593 8727bdff39320e3166201264086b2af7 example.com 172.16.130.32 2018-03-13 09:30:05.000000 24651 0:0 16 3 Peer de-configured 0 1 1 0 0

- V: 1.7 C_HASH_ID: 1a3e29f5a8c84ea7bc519f159166d4b3 T: peer L: 457 R: 1

up 20 de26e4d608509af1abca9df8579f8593 8727bdff39320e3166201264086b2af7 example.com 172.16.130.32 2018-03-13 09:30:40.000000 24651 0:0 54985 12847 195.246.227.50 179 MPBGP (1) : afi=1 safi=1 : Unicast IPv4, Route Refresh Old (128), Route Refresh (2), Graceful Restart (64), 4 Octet ASN (65), 71 MPBGP (1) : afi=1 safi=1 : Unicast IPv4, Route Refresh Old (128), Route Refresh (2) 180 90

Do the messages look correct?

In gen-active-asns.log file I noticed the following messages: ---- INIT: Creating tables and loading data: 2018-03-13 07:55:03.398611 ERROR: query failed - 1054 (42S22): Unknown column 'isWithdrawn' in 'where clause' ---- DONE: 2018-03-13 07:55:03.417072

Perhaps that's one of the reasons why tables are not populated with stats?

How can we check that DB triggers are working correctly? Perhaps they are another reason why we don't see stats in GUI and DB?

bofh16 commented 3 years ago

Hello, Is this repository still supported? I try to run OpenBMP in all-in-one container and get the same message in gen-active-asns.log. There is clearly a gap, between gen-asn-stats.py script and the database structure. Does the container needs a schema upgrade or the script points in the wrong direction?

abdul95 commented 1 year ago

@gunarser I am running to an issue installing while doing a manual install. I am not seeing message counter inclement in the db it is stable at 0.

can you share the steps you took to install ? so I can compare, or if you have any thoughts on how to get passed that issue here are the logs I see in mysql-consumer 

2023-02-20 01:49:51 [main] INFO  org.openbmp.MySQLConsumerApp - -- STATS --   thread: 0  read: 0           queue: 0         
2023-02-20 01:49:51 [main] INFO  org.openbmp.MySQLConsumerApp -            collector messages: 0
2023-02-20 01:49:51 [main] INFO  org.openbmp.MySQLConsumerApp -               router messages: 0
2023-02-20 01:49:51 [main] INFO  org.openbmp.MySQLConsumerApp -                 peer messages: 0
2023-02-20 01:49:51 [main] INFO  org.openbmp.MySQLConsumerApp -              reports messages: 0
2023-02-20 01:49:51 [main] INFO  org.openbmp.MySQLConsumerApp -       base attribute messages: 0
2023-02-20 01:49:51 [main] INFO  org.openbmp.MySQLConsumerApp -       unicast prefix messages: 0
2023-02-20 01:49:51 [main] INFO  org.openbmp.MySQLConsumerApp -       l3vpn prefix messages: 0
2023-02-20 01:49:51 [main] INFO  org.openbmp.MySQLConsumerApp -              LS node messages: 0
2023-02-20 01:49:51 [main] INFO  org.openbmp.MySQLConsumerApp -              LS link messages: 0
2023-02-20 01:49:51 [main] INFO  org.openbmp.MySQLConsumerApp -            LS prefix messages: 0