irrdb-cli.update-prefix-db takes very long on AS-HURRICANE

[kieber]

Thu 15 Dec 03:00:01 CET 2016

Running irrdb-cli.update-asn-db: Processing ...

Thu 15 Dec 03:03:05 CET 2016

Running irrdb-cli.update-prefix-db: Processing ...

Thu 15 Dec 05:29:20 CET 2016

Nick pointed out it's probably AS-HURRICANE that's chewing up all the time.

[barryo]

The first comment I'll make on this is that it doesn't really matter from a production point of view how long this takes. It's intended as a cron script that's run in the back ground. It's also transaction safe at the database level so route server generation does not need to be tied to running this. They can safely overlap with zero consequences.

For reference, documentation is at: https://github.com/inex/IXP-Manager/wiki/IRRDB-Prefixes

I'm just doing a timed run now on INEX's version to compare. Will come back shortly.

[barryo]

@kieber - could you run it again with surrounding date's and the -v switch (and strip any members that are inconsequential in terms of number of prefixes) to get an out put like:

date ; /srv/ixpmanager/bin/ixptool.php -a irrdb-cli.update-prefix-db -v ; date
Mon Dec 19 08:34:10 GMT 2016
Processing 3 Ireland: [IPv4: IRRDB 27; 0 stale; 0 new; DB updated] [IPv6: IRRDB 0; DB not updated]
IRRDB 1; 0 stale; 0 new; DB updated]
...
Processing euNetworks: [IPv4: IRRDB 103935; 0 stale; 2 new; DB updated] [IPv6: IRRDB 5930; 0 stale; 0 new; DB updated]
...
Processing Yahoo!: [IPv4: IRRDB 1035; 0 stale; 0 new; DB updated] [IPv6: IRRDB 91; 0 stale; 0 new; DB updated]
Database time  : 3.5925595760345
Processing time: 315.24506354332
Network time   : 96.480045080185
Mon Dec 19 08:41:06 GMT 2016

[kieber]

It may be safe transaction-wise, but: If you add a new peer to IXPM and then run route server config generation before running irrdb-cli.update-{asn,prefix}-db, the new peer generates errors. Thus running RS config generation before irrdb-cli.update-{asn,prefix}-db is also kinda operationally useless, running it considerably later doesn't give an advantage either, and running it more than once after irrdb-cli.update-{asn,prefix}-db doesn't produce any different results than the first run. That said, I'd prefer doing a full run of both, irrdb-cli.update-{asn,prefix}-db followed by RS config generation before starting another cycle, just to spare me from the cron mails.

The other output comes tomorrow morning. :)

[kieber]

So, here is an output of

date ; echo "Running irrdb-cli.update-asn-db:" ixptool.php -v -a irrdb-cli.update-asn-db | egrep -v "Processing .*: \[IPv4: IRRDB .*; 0 stale; 0 new; DB updated\] \[IPv6: IRRDB .*; 0 stale; 0 new; DB updated\] $" date ; echo "Running irrdb-cli.update-prefix-db:" ixptool.php -v -a irrdb-cli.update-prefix-db | egrep -v "Processing .*: \[AS-MACRO: .*; IPv4: IRRDB .*; 0 stale; 0 new; DB updated\] \[AS-MACRO: .*; IPv6: IRRDB (.*; 0 stale; 0 new; DB|0; DB not) updated\] $" date

Tue 20 Dec 03:00:01 CET 2016

Running irrdb-cli.update-asn-db: Processing British Telecommunications plc: [IPv4: IRRDB 2349; 0 stale; 3 new; DB updated] [IPv6: IRRDB 2349; 0 stale; 3 new; DB updated] Processing Core-Backbone GmbH: [IPv4: IRRDB 2550; 1 stale; 3 new; DB updated] [IPv6: IRRDB 2550; 1 stale; 3 new; DB updated] Processing Hibernia Networks: [IPv4: IRRDB 8567; 1 stale; 4 new; DB updated] [IPv6: IRRDB 8567; 1 stale; 4 new; DB updated] Processing Hurricane Electric: [IPv4: IRRDB 18022; 12 stale; 12 new; DB updated] [IPv6: IRRDB 18022; 12 stale; 12 new; DB updated] Processing Init Seven AG: [IPv4: IRRDB 3037; 1 stale; 3 new; DB updated] [IPv6: IRRDB 3037; 1 stale; 3 new; DB updated] Processing Link11 GmbH: [IPv4: IRRDB 1331; 0 stale; 1 new; DB updated] [IPv6: IRRDB 1331; 0 stale; 1 new; DB updated] Processing M247 Ltd.: [IPv4: IRRDB 1224; 0 stale; 1 new; DB updated] [IPv6: IRRDB 1224; 0 stale; 1 new; DB updated] Processing next layer GmbH: [IPv4: IRRDB 435; 0 stale; 1 new; DB updated] [IPv6: IRRDB 435; 0 stale; 1 new; DB updated] Processing Open Peering BV: [IPv4: IRRDB 578; 0 stale; 1 new; DB updated] [IPv6: IRRDB 578; 0 stale; 1 new; DB updated] Processing PT Telekomunikasi Indonesia In: [IPv4: IRRDB 822; 0 stale; 2 new; DB updated] [IPv6: IRRDB 822; 0 stale; 2 new; DB updated] Processing RETN Ltd.: [IPv4: IRRDB 15064; 0 stale; 3 new; DB updated] [IPv6: IRRDB 15064; 0 stale; 3 new; DB updated] Processing Swisscom (Schweiz) AG: [IPv4: IRRDB 1456; 0 stale; 1 new; DB updated] [IPv6: IRRDB 1456; 0 stale; 1 new; DB updated] Processing TNG AG: [IPv4: IRRDB 1620; 1 stale; 2 new; DB updated] [IPv6: IRRDB 1620; 1 stale; 2 new; DB updated] Processing Zayo France: [IPv4: IRRDB 1400; 0 stale; 2 new; DB updated] [IPv6: IRRDB 670; 0 stale; 3 new; DB updated] Database time : 1.2448251247406 Processing time: 0.19893598556519 Network time : 182.96543216705

Tue 20 Dec 03:03:06 CET 2016

Running irrdb-cli.update-prefix-db: Processing Akamai B.V.: [AS-MACRO: AS-AKAMAI; IPv4: IRRDB 3945; 0 stale; 2 new; DB updated] [AS-MACRO: AS-AKAMAI; IPv6: IRRDB 764; 0 stale; 0 new; DB updated] Processing British Telecommunications plc: [AS-MACRO: AS-BT-EU; IPv4: IRRDB 41944; 6 stale; 14 new; DB updated] [AS-MACRO: AS-BT-EU; IPv6: IRRDB 1766; 0 stale; 0 new; DB updated] Processing Cloudflare, Inc.: [AS-MACRO: AS-CLOUDFLARE; IPv4: IRRDB 951; 0 stale; 1 new; DB updated] [AS-MACRO: AS-CLOUDFLARE; IPv6: IRRDB 176; 0 stale; 0 new; DB updated] Processing Colt Telecom: [AS-MACRO: AS-COLT; IPv4: IRRDB 7690; 1 stale; 4 new; DB updated] [AS-MACRO: AS-COLT; IPv6: IRRDB 840; 1 stale; 1 new; DB updated] Processing Console, Inc.: [AS-MACRO: AS-IXREACH; IPv4: IRRDB 12221; 5 stale; 2 new; DB updated] [AS-MACRO: AS-IXREACH; IPv6: IRRDB 1208; 0 stale; 0 new; DB updated] Processing Core-Backbone GmbH: [AS-MACRO: AS-CORE-BACKBONE; IPv4: IRRDB 28175; 4 stale; 28 new; DB updated] [AS-MACRO: AS-CORE-BACKBONE; IPv6: IRRDB 3434; 1 stale; 2 new; DB updated] Processing Digital Telecommunication Serv: [AS-MACRO: AS-DTSRETEIVO; IPv4: IRRDB 111; 0 stale; 1 new; DB updated] [AS-MACRO: AS-DTSRETEIVO; IPv6: IRRDB 20; 0 stale; 0 new; DB updated] Processing Emirates Telecom Corp.: [AS-MACRO: AS-EMIX; IPv4: IRRDB 32323; 0 stale; 2 new; DB updated] [AS-MACRO: AS-EMIX; IPv6: IRRDB 942; 0 stale; 0 new; DB updated] Processing Equinix (Switzerland) GmbH: [AS-MACRO: AS-EQUINIX-EU; IPv4: IRRDB 2351; 0 stale; 1 new; DB updated] [AS-MACRO: AS-EQUINIX-EU; IPv6: IRRDB 327; 0 stale; 0 new; DB updated] Processing Hibernia Networks: [AS-MACRO: AS-HIBERNIA; IPv4: IRRDB 85453; 68 stale; 55 new; DB updated] [AS-MACRO: AS-HIBERNIA; IPv6: IRRDB 6274; 1 stale; 1 new; DB updated] Processing Hurricane Electric: [AS-MACRO: AS-HURRICANE; IPv4: IRRDB 588556; 73 stale; 56 new; DB updated] [AS-MACRO: AS-HURRICANE; IPv6: IRRDB 26965; 0 stale; 14 new; DB updated] Processing Init Seven AG: [AS-MACRO: AS-INIT7; IPv4: IRRDB 31014; 4 stale; 28 new; DB updated] [AS-MACRO: AS-INIT7; IPv6: IRRDB 3895; 1 stale; 2 new; DB updated] Processing Jaguar Network: [AS-MACRO: AS-JAGUAR; IPv4: IRRDB 1156; 0 stale; 0 new; DB updated] [AS-MACRO: AS-JAGUAR; IPv6: IRRDB 170; 0 stale; 1 new; DB updated] Processing Link11 GmbH: [AS-MACRO: AS-LINK11; IPv4: IRRDB 12536; 1 stale; 5 new; DB updated] [AS-MACRO: AS-LINK11; IPv6: IRRDB 1090; 0 stale; 0 new; DB updated] Processing M247 Ltd.: [AS-MACRO: AS-GBXS; IPv4: IRRDB 7773; 1 stale; 4 new; DB updated] [AS-MACRO: AS-GBXS; IPv6: IRRDB 894; 0 stale; 0 new; DB updated] Processing next layer GmbH: [AS-MACRO: AS-NEXTLAYER; IPv4: IRRDB 2176; 0 stale; 1 new; DB updated] [AS-MACRO: AS-NEXTLAYER; IPv6: IRRDB 484; 0 stale; 0 new; DB updated] Processing Open Peering BV: [AS-MACRO: AS-OPENPEERING-EU; IPv4: IRRDB 8158; 4 stale; 11 new; DB updated] [AS-MACRO: AS-OPENPEERING-EU; IPv6: IRRDB 1411; 0 stale; 1 new; DB updated] Processing RETN Ltd.: [AS-MACRO: AS-RETN; IPv4: IRRDB 144465; 75 stale; 187 new; DB updated] [AS-MACRO: AS-RETN; IPv6: IRRDB 8008; 0 stale; 1 new; DB updated] Processing SG.GS: [AS-MACRO: AS-SGGS; IPv4: IRRDB 2986; 3 stale; 0 new; DB updated] [AS-MACRO: AS-SGGS; IPv6: IRRDB 107; 0 stale; 0 new; DB updated] Processing Sunrise Communications AG: [AS-MACRO: AS-GLOBAL; IPv4: IRRDB 4675; 0 stale; 2 new; DB updated] [AS-MACRO: AS-GLOBAL; IPv6: IRRDB 915; 0 stale; 0 new; DB updated] Processing Swisscom (Schweiz) AG: [AS-MACRO: AS-SWCMGLOBAL; IPv4: IRRDB 18100; 2 stale; 12 new; DB updated] [AS-MACRO: AS-SWCMGLOBAL; IPv6: IRRDB 1777; 0 stale; 0 new; DB updated] Processing SwissIX: [AS-MACRO: AS-SWISSIX; IPv4: IRRDB 43471; 0 stale; 1 new; DB updated] [AS-MACRO: AS-SWISSIX; IPv6: IRRDB 66; 0 stale; 0 new; DB updated] Processing SysEleven GmbH: [AS-MACRO: AS-SYSELEVEN; IPv4: IRRDB 759; 0 stale; 1 new; DB updated] [AS-MACRO: AS-SYSELEVEN; IPv6: IRRDB 167; 0 stale; 0 new; DB updated] Processing TNG AG: [AS-MACRO: AS-TNG; IPv4: IRRDB 12779; 3 stale; 12 new; DB updated] [AS-MACRO: AS-TNG; IPv6: IRRDB 1657; 1 stale; 0 new; DB updated] Processing VeriSign Registry: [AS-MACRO: AS-GTLD; IPv4: IRRDB 43464; 0 stale; 1 new; DB updated] [AS-MACRO: AS-GTLD; IPv6: IRRDB 63; 0 stale; 0 new; DB updated] Processing Zayo France: [AS-MACRO: AS-NEOT; IPv4: IRRDB 8486; 0 stale; 3 new; DB updated] [AS-MACRO: AS-NEOT6; IPv6: IRRDB 880; 0 stale; 1 new; DB updated] Database time : 29.259595394135 Processing time: 8570.2932422161 Network time : 358.56370425224

Tue 20 Dec 05:32:42 CET 2016

[barryo]

@kieber

It may be safe transaction-wise, but: If you add a new peer to IXPM and then run route server config generation before running irrdb-cli.update-{asn,prefix}-db, the new peer generates errors.

Not sure what the situation was on v3 but certainly on v4, config gen won't produce an error in this situation. The member will just be reject all until the data is populated.

[barryo]

@kieber

Database time : 29.259595394135 Processing time: 8570.2932422161 Network time : 358.56370425224

So the issue here was PHP's general well-known efficiency with arrays (internal hash table) and how we were using them (n x linear traversal's where n is the number of prefixes).

It didn't come up for us previously as we did not have a route server member with this scale of prefixes.

In the above commit (6b5a0aa5bf15d6c3987b1e123fd4248c26472e46) I have replaced arrays with a new PHP 7 data structure Ds\Set.

The result is that for the same data set (AS-HURRICANE), processing time has gone from ~8570 secs to ~3.5 secs.

[nickhilliard]

that was a github comment delivered with no small level of satisfaction.

[barryo]

Note that to make use of this, you'll need the php-ds extension.

Ubuntu mainline are slow to the party here: https://packages.ubuntu.com/search?keywords=php-ds&searchon=names&suite=all&section=all

Options:

• use the Ubuntu PHP PPA: https://launchpad.net/~ondrej/+archive/ubuntu/php (IXP Manager will work perfectly fine with PHP 7.1 btw)
• manual install (pecl install ds - more details here)

[barryo]

For posterity - results from my Macbook Pro 2016 (2.7GHz i7):

Data set as of today's date:

$ bgpq3 -4 AS-HURRICANE  | wc -l
1055279
 $ bgpq3 -6 AS-HURRICANE  | wc -l
166103

From a freshly initialised database with this data (i.e. already populated - which takes longer due to ~1m inserts), an update runs as:

Hurricane Electric: [IPv4: 943875 total; 0 stale; 0 new; DB updated] [IPv6: 161275 total; 0 stale; 0 new; DB updated]
    Time for net/database/processing: 10.428121/30.826060/3.132205 (secs)

For something much smaller:

HEAnet: [IPv4: 49 total; 0 stale; 0 new; DB updated] [IPv6: 23 total; 0 stale; 0 new; DB updated]
    Time for net/database/processing: 2.158729/0.026693/0.003169 (secs)