Possible Performance improvements

[thomas-mangin]

Hello, Thank you very much for writing this very useful tool.

I noticed that exabgp_conf.py can be generate very large configuration files which will most often will then take ages to load. A faster way to do would be to send the routes using the API. The process has been documented by another ExaBGP user here: https://github.com/FastVPSEestiOu/fastnetmon/blob/master/docs/EXABGP_INTEGRATION.md or more elegantly https://github.com/job/irrexplorer/blob/master/exabgp.conf

Also, I added an "undocumented" (i.e.: needing some improvement but working) command to the API allowing to reduce parsing by grouping updates with the same attributes. From memory it is something like "announce attribute ... route ... ". I would need to double check if this is of interest.

Good continuation.

[YoshiyukiYamauchi]

Thank you for your advice! We added the option using the API. Option of grouping updates with the same attributes also added. Please check "mrt2exabgp.py".

[ExaBGP] How to send the routes using the API [ExaBGP] How to reduce parsing by grouping updates with the same attributes

[thomas-mangin]

Thank you very much for adding this feature :+1:

[YoshiyukiYamauchi]

Now I close this case.

[YoshiyukiYamauchi]

Sorry! I have forgot to describe performance... Here is a comparison of the results.

Config format(Previous method) -> 5:23

$ /usr/bin/time /usr/bin/pypy ~/mrtparse/examples/mrt2exabgp.py -r 127.0.0.2 -l 64512 -p 65000 -L 127.0.0.2 -n 127.0.0.1 -4 192.168.1.254 ~/Downloads/bview.20150601.0000.gz > ~/mrtparse/examples/static.conf
66.03user 0.30system 1:06.39elapsed 99%CPU (0avgtext+0avgdata 110352maxresident)k
0inputs+166472outputs (0major+17068minor)pagefaults 0swaps

$ /usr/bin/exabgp ~/mrtparse/examples/static.conf 1>~/mrtparse/examples/static.log 2>&1
==> static.log <==
Sun, 21 Jun 2015 02:38:23 | INFO     | 21988  | configuration | environment file missing
Sun, 21 Jun 2015 02:41:53 | INFO     | 21988  | network       | Connected to peer neighbor 127.0.0.1 local-ip 127.0.0.2 local-as 64512 peer-as 65000 router-id 127.0.0.2 family-allowed in-open (out)

$ tshark -q -i lo -w ~/mrtparse/examples/static.pcap tcp port 179 1>/dev/null 2>&1 &
==> static.pcap <==
16  2015-06-21 02:41:53.528978000   127.0.0.2   127.0.0.1   BGP 132 UPDATE Message
1080422 2015-06-21 02:42:46.563973000   127.0.0.2   127.0.0.1   BGP 77  UPDATE Message

Using API without grouping format -> 4:33

$ /usr/bin/exabgp ~/mrtparse/examples/api.conf 1>~mrtparse/examples/api.log 2>&1
==> api.log <==
Sun, 21 Jun 2015 03:00:05 | INFO     | 22364  | configuration | environment file missing
Sun, 21 Jun 2015 03:04:37 | INFO     | 22364  | reactor       | Updated peers dynamic routes successfully

$ tshark -q -i lo -w ~/mrtparse/examples/api.pcap tcp port 179 1>/dev/null 2>&1 &
==> api.pcap <==
1   2015-06-21 03:00:11.397807000   127.0.0.2   127.0.0.1   BGP 136 UPDATE Message
1095912 2015-06-21 03:04:38.086106000   127.0.0.2   127.0.0.1   BGP 77  UPDATE Message

Using API with grouping format -> 03:36

$ /usr/bin/exabgp ~/mrtparse/exabgp/group.conf 1>~mrtparse/exabgp/group.log 2>&1
==> group.log <==
Sun, 21 Jun 2015 03:05:52 | INFO     | 22479  | configuration | environment file missing
Sun, 21 Jun 2015 03:09:28 | INFO     | 22479  | reactor       | Updated peers dynamic routes successfully

$ tshark -q -i lo -w ~/mrtparse/exabgp/group.pcap tcp port 179 1>/dev/null 2>&1 &
==> group.pcap <==
5   2015-06-21 03:07:13.192208000   127.0.0.2   127.0.0.1   BGP 128 UPDATE Message
1103865 2015-06-21 03:09:28.597166000   127.0.0.2   127.0.0.1   BGP 132 UPDATE Message

Using API with grouping format(10,000 prefix units) -> 2:57

$ /usr/bin/exabgp ~/mrtparse/examples/group100000.conf 1>~mrtparse/examples/group100000.log 2>&1
==> group100000.log <==
Sun, 21 Jun 2015 03:21:30 | INFO     | 22750  | configuration | environment file missing
Sun, 21 Jun 2015 03:24:26 | INFO     | 22750  | reactor       | Updated peers dynamic routes successfully

$ tshark -q -i lo -w ~/mrtparse/examples/group100000.pcap tcp port 179 1>/dev/null 2>&1 &
==> group100000.pcap <==
1   2015-06-21 03:21:44.925858000   127.0.0.2   127.0.0.1   BGP 128 UPDATE Message
1093828 2015-06-21 03:24:27.302069000   127.0.0.2   127.0.0.1   BGP 113 UPDATE Message

Please let us know what you think if you like.

[thomas-mangin]

Thank you very much for taking the time to put these numbers together. The change does seems to improve things :v:

The API way should also use much less memory (on ExaBGP's side).

It may be possible to improve the performance even more using the following neighbor options: group-updates enable; auto-flush disable;

And them only flushing the routes when you have some grouped enough NLRI to fill your NLRI. The command to do so "route flush" ( from memory ).

something like : announce attribute... [... a few hundred ...] announce attribute ... flush route;

[thomas-mangin]

I forgot to say that in this way the initial update will be regrouped so it would be a like for like replay.

[YoshiyukiYamauchi]

OK, Thank you for the info.