search

t2mune / mrtparse

MRT format data parser
Apache License 2.0
137 stars 39 forks source link

Python3 Error in mrtparse #27

Open mahmutaydin1 opened 3 years ago

mahmutaydin1 commented 3 years ago

Hello Dear All,

When I run below command, I am receiving error. Could you please help me?

[root@centos examples]# time python3 mrt2exabgp.py -G -P latest-bview.gz > fullbgptable.py ^CTraceback (most recent call last): File "mrt2exabgp.py", line 514, in main() File "mrt2exabgp.py", line 511, in main conv_format(args, flags, d) File "mrt2exabgp.py", line 179, in conv_format for m in d: File "/usr/local/lib/python3.6/site-packages/mrtparse/init.py", line 94, in next self.unpack_msg(mrt) File "/usr/local/lib/python3.6/site-packages/mrtparse/init.py", line 138, in unpack_msg self.unpack_td_v2(buf, mrt) File "/usr/local/lib/python3.6/site-packages/mrtparse/init.py", line 185, in unpack_td_v2 rib.unpack() File "/usr/local/lib/python3.6/site-packages/mrtparse/init.py", line 381, in unpack self.p += entry.unpack() File "/usr/local/lib/python3.6/site-packages/mrtparse/init.py", line 410, in unpack self.p += attr.unpack() File "/usr/local/lib/python3.6/site-packages/mrtparse/init.py", line 703, in unpack self.unpack_next_hop() File "/usr/local/lib/python3.6/site-packages/mrtparse/init.py", line 766, in unpack_next_hop self.data['value'] = self.val_addr(AFI_T['IPv4']) File "/usr/local/lib/python3.6/site-packages/mrtparse/base.py", line 301, in val_addr addr = socket.inet_ntop(_af, buf + b'\x00'*(plen_max // 8 - n)) KeyboardInterrupt

real 0m5.786s user 0m5.513s sys 0m0.058s [root@centos examples]#

t2mune commented 3 years ago

Hi,

Thank you for using mrtparse. It looks like an error when you send ctrl+c.

Using " -G" option, all entries of MRT data are temporarily stored in python dictionary to group prefixes with the same path attributes. After all MRT entries are parsed, output finally begins. Therefore, it takes a long time to output(not hang). If you want to accelerate, I recommend using pypy instead of python3.

In my environment, the result follows:

(mrtparse-python3) [root@host-12 ~]# python3 --version
Python 3.6.8
(mrtparse-python3) [root@host-12 ~]# time python3 mrt2exabgp.py -G -P latest-bview.gz > fullbgptable-python3.py                                                                                                     

real    231m14.710s
user    230m5.261s
sys     0m24.153s

(mrtparse-pypy) [root@host-12 ~]# pypy --version
Python 2.7.18 (a29ef73f9b32, Nov 09 2020, 18:42:06)
[PyPy 7.3.3 with GCC 7.3.1 20180303 (Red Hat 7.3.1-5)]

(mrtparse-pypy) [root@host-12 ~]# time pypy mrt2exabgp.py -G -P latest-bview.gz > fullbgptable-pypy.py

real    100m0.023s
user    94m19.296s
sys     4m59.085s
mahmutaydin1 commented 3 years ago

Hi

Thank you for your help. However when I run the python I am receiving below error. Same error is occurring both windows 10 and centos 8.3.

mrt2exabgp.py -G 10 latest-update.gz > test3.py

Traceback (most recent call last):

KeyError: 'asn'

Tetsumune KISO notifications@github.com, 30 Ara 2020 Çar, 00:13 tarihinde şunu yazdı:

Hi,

Thank you for using mrtparse. It looks like an error when you send ctrl+c.

Using " -G" option, All entries of MRT data are temporarily stored in python dictionary to group prefixes with the same path attributes. After all MRT entries are parsed, output finally begins. Therefore, it takes a long time to output(not hang). If you want to accelerate, I recommend using pypy instead of python3.

In my environment, the result follows:

(mrtparse-python3) [root@host-12 ~]# python3 --version Python 3.6.8 (mrtparse-python3) [root@host-12 ~]# time python3 mrt2exabgp.py -G -P latest-bview.gz > fullbgptable-python3.py

real 231m14.710s user 230m5.261s sys 0m24.153s

(mrtparse-pypy) [root@host-12 ~]# pypy --version Python 2.7.18 (a29ef73f9b32, Nov 09 2020, 18:42:06) [PyPy 7.3.3 with GCC 7.3.1 20180303 (Red Hat 7.3.1-5)]

(mrtparse-pypy) [root@host-12 ~]# time pypy mrt2exabgp.py -G -P latest-bview.gz > fullbgptable-pypy.py

real 100m0.023s user 94m19.296s sys 4m59.085s

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/t2mune/mrtparse/issues/27#issuecomment-752242622, or unsubscribe https://github.com/notifications/unsubscribe-auth/AIZCGRYYNDO7VQPKNTMP6UDSXJA6PANCNFSM4VMPBVIA .

t2mune commented 3 years ago

It's a typo, sorry. I fixed it, so could you try again? 3dfd9a4a3be6338f51ce8f013ece41b4c5d40923

mahmutaydin1 commented 3 years ago

Hello,

Thank you for your help. Now I am not receiving key error. But, this script is not done for 8 hours. When I interıupt manual, I am receiving below error. This time, I added also other parameters like local and neighbor as and IP. Could you please chek it? thank you

(venv) C:\Users\asus\PycharmProjects\Ebebek>ryu.py -l 100 -p 200 -n:10.90.90.2 -L 10.90.90.1 -G latest-update.gz > test5.py Traceback (most recent call last): File "C:\Users\asus\PycharmProjects\Ebebek\ryu.py", line 504, in main() File "C:\Users\asus\PycharmProjects\Ebebek\ryu.py", line 501, in main conv_format(args, flags, d) File "C:\Users\asus\PycharmProjects\Ebebek\ryu.py", line 191, in conv_format time.sleep(1) KeyboardInterrupt

Tetsumune KISO notifications@github.com, 30 Ara 2020 Çar, 11:48 tarihinde şunu yazdı:

It's a typo, sorry. I fixed it, so could you try again? 3dfd9a4 https://github.com/t2mune/mrtparse/commit/3dfd9a4a3be6338f51ce8f013ece41b4c5d40923

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/t2mune/mrtparse/issues/27#issuecomment-752378142, or unsubscribe https://github.com/notifications/unsubscribe-auth/AIZCGR3GIZDODU7FZYFRDNDSXLSNBANCNFSM4VMPBVIA .

t2mune commented 3 years ago

You need "-P" option. Without "-P" option, It goes into an infinite loop and never exits.

(...snip)
if params['flags'] & FLAG_T['API_PROG']:
    print_api_prog_footer()
else:
    while True:
        time.sleep(1)
(...snip)
mahmutaydin1 commented 3 years ago

Dear,

I have added -P paramater and script is completed. BUt please this knowledge should be added to help or documentation. Also, when I run again same command, there is no LOCAL AS, PEERAS and NEIGHBOUR in output. When I add -P parameter, it outputs in API format of announce subnets. As below:

Thank you,

!/usr/bin/env python

import sys import time msgs = [ 'announce attributes origin IGP as-path [139589 49752 137490 39533 267397 ] community [0:39533 39533:49666 64700:26162] large-community [57463:0:8966 57463:0:44901 57463:0:52863 57463:0:201133 57463:0:263009 57463:0:263080 57463:0:267242 57463:0:267613 139589:0:0] next-hop 45.14.68.30 nlri 45.233.206.0/23 45.233.205.0/24 45.233.207.0/24 45.233.204.0/24', 'announce attributes origin IGP as-path [34549 5511 1299 9498 714 ] community [5511:521 5511:666 5511:710 5511:5511 34549:100 34549:5511] next-hop 80.77.16.114 nlri 17.76.120.0/24', 'announce attributes origin IGP as-path [34800 58057 6939 3356 3549 27951 ] community [34800:65010 58057:65010] next-hop 2001:7f8:e7:c232::63fe nlri 2801:172:2::/48', 'announce attributes origin IGP as-path [139589 6939 201808 ] med 0 community [0:714 0:2906 0:6939 0:12876 0:12989 0:13335 0:15133 0:15169 0:16265 0:16276 0:16509 0:20940 0:22822 0:32590 0:48641 0:49029] large-community [139589:0:139589] next-hop 2a07:59c6:ee00:9589::1 nlri 2a01:a5e0::/32 2a0c:8c80::/32', 'announce attributes origin IGP as-path [139589 49752 34549 3356 13904 13904 13904 13904 13904 13904 13904 13904 13904 ] community [3356:3 3356:22 3356:100 3356:123 3356:575 3356:2026 34549:100 34549:3356] large-community [139589:0:0] next-hop 45.14.68.30 nlri 216.238.254.0/23', 'announce attributes origin IGP as-path [34549 3257 2914 61317 ] community [3257:8115 3257:30342 3257:50001 3257:53300 3257:53301 34549:100 34549:3257] next-hop 80.77.16.114 nlri ...

t2mune commented 3 years ago

Thank you for your comment. I will update a documentation for using ExaBGP with mrtparse.

When using ExaBGP API, you have to wirte ExaBGP config by yourself, and need to specify a generated script using mrt2exabgp in a process section. ExaBGP documentation might be helpful.

group TEST {
    neighbor 10.90.90.2 {
        router-id 10.90.90.1;
        local-address 10.90.90.1;
        local-as 100;
        peer-as 200;
        graceful-restart;

        process TEST {
            run GENERATED_SCRIPT.py;
        }
    }
}
mahmutaydin1 commented 3 years ago

Dear, Thank you. I have taken progress with your help. But, now another problem exists.

this is my exabgp config file:

 process test  {
            run    fullroutebgp2.py;
            encoder json;
        }

    neighbor 10.90.90.2 {
        router-id 10.90.90.1;
        local-address 10.90.90.1;
        local-as 100;
        peer-as 200;

       api {
        processes  [test];
    }
    }

When I run command:[root@centos exabgp-git]# ./sbin/exabgp etc/exabgp/fullroutebgp.conf

my error is below:

[root@centos exabgp-git]# ./sbin/exabgp etc/exabgp/fullroutebgp.conf
welcome       Thank you for using ExaBGP
advice        environment file missing
advice        generate it using "exabgp env --fi > /root/exabgp-git/etc/exabgp/exabgp.env"
cli           could not find the named pipes (exabgp.in and exabgp.out) required for the cli
cli           we scanned the following folders (the number is your PID):
cli control    - /run/exabgp/
cli control    - /run/0/
cli control    - /run/
cli control    - /var/run/exabgp/
cli control    - /var/run/0/
cli control    - /var/run/
cli control    - /root/exabgp-git/run/exabgp/
cli control    - /root/exabgp-git/run/0/
cli control    - /root/exabgp-git/run/
cli control    - /root/exabgp-git/var/run/exabgp/
cli control    - /root/exabgp-git/var/run/0/
cli control    - /root/exabgp-git/var/run/
cli control   please make them in one of the folder with the following commands:
cli control   > mkfifo /root/exabgp-git/run/exabgp.{in,out}
cli control   > chmod 600 /root/exabgp-git/run/exabgp.{in,out}
configuration performing reload of exabgp master-d2c7c074134ca6015bf112076604110439bfcf4c
reactor       loaded new configuration successfully
/usr/bin/env: ‘python\r’: No such file or directory
/usr/bin/env: ‘python\r’: No such file or directory
/usr/bin/env: ‘python\r’: No such file or directory
/usr/bin/env: ‘python\r’: No such file or directory
/usr/bin/env: ‘python\r’: No such file or directory
outgoing-1    --------------------------------------------------------------------
outgoing-1    the connection can not carry the following family/families
outgoing-1     - peer is not configured for ipv4/multicast
outgoing-1     - peer is not configured for bgp-ls/bgp-ls
outgoing-1     - peer is not configured for ipv6/flow-vpn
outgoing-1     - peer is not configured for bgp-ls/bgp-ls-vpn
outgoing-1     - peer is not configured for ipv6/multicast
outgoing-1     - peer is not configured for l2vpn/evpn
outgoing-1     - peer is not configured for ipv4/rtc
outgoing-1     - peer is not configured for ipv4/nlri-mpls
outgoing-1     - peer is not configured for ipv6/unicast
outgoing-1     - peer is not configured for ipv4/flow
outgoing-1     - peer is not configured for ipv6/mpls-vpn
outgoing-1     - peer is not configured for ipv4/flow-vpn
outgoing-1     - peer is not configured for l2vpn/vpls
outgoing-1     - peer is not configured for ipv6/nlri-mpls
outgoing-1     - peer is not configured for ipv4/mpls-vpn
outgoing-1     - peer is not configured for ipv6/flow
outgoing-1    therefore no routes of this kind can be announced on the connection
outgoing-1    --------------------------------------------------------------------
process       Too many death for test (5) terminating program
reactor       Problem when sending message(s) to helper program, stopping
reactor       performing shutdown
: No such file or directory
outgoing-1    outgoing-1 10.90.90.1-10.90.90.2, closing connection

fullroutebgp2.py is afile that was converted via script of latest-update.gz (not latest-bview.gz)

t2mune commented 3 years ago

Hi,

Sorry for my late reply.

The problem is shown as below:

/usr/bin/env: ‘python\r’: No such file or directory
/usr/bin/env: ‘python\r’: No such file or directory
/usr/bin/env: ‘python\r’: No such file or directory
/usr/bin/env: ‘python\r’: No such file or directory
/usr/bin/env: ‘python\r’: No such file or directory

So, you need to edit the first line of "fullroutebgp2.py" as follows:

#!/usr/bin/python3

In my envirtonment, the result is as follows:

welcome         Thank you for using ExaBGP                                                                                                                                                                          advice          environment file missing                                                                                                                                                                            advice          generate it using "exabgp env --fi > /home/t2mune/exabgp/etc/exabgp/exabgp.env"
cli             could not find the named pipes (exabgp.in and exabgp.out) required for the cli                                                                                                                      cli             we scanned the following folders (the number is your PID):                                                                                                                                          
cli control      - /run/exabgp/                                                                                                                                                                                     cli control      - /run/10001/                                                                                                                                                                                      
cli control      - /run/                                                                                                                                                                                            cli control      - /var/run/exabgp/                                                                                                                                                                                 
cli control      - /var/run/10001/                                                                                                                                                                                  cli control      - /var/run/ 
cli control      - /home/t2mune/exabgp/run/exabgp/                                                                                                                                                                  cli control      - /home/t2mune/exabgp/run/10001/
cli control      - /home/t2mune/exabgp/run/                                                               
cli control      - /home/t2mune/exabgp/var/run/exabgp/                                         
cli control      - /home/t2mune/exabgp/var/run/10001/                                                                                                                                                               cli control      - /home/t2mune/exabgp/var/run/                                                           
cli control     please make them in one of the folder with the following commands:                                                                                                                                  cli control     > mkfifo /home/t2mune/exabgp/run/exabgp.{in,out} 
cli control     > chmod 600 /home/t2mune/exabgp/run/exabgp.{in,out}                                                                                                                                                 cli control     > chown 10001:10001 /home/t2mune/exabgp/run/exabgp.{in,out}
configuration   performing reload of exabgp master-a07a3a833c57549812465ade446ee2decf31b109                                                                                                                         reactor         loaded new configuration successfully    
api             route added to neighbor 10.90.90.2 local-ip 10.90.90.1 local-as 100 peer-as 200 router-id 10.90.90.1 family-allowed in-open : 103.140.227.0/24 next-hop 5.178.95.254 origin igp as-path [ 59919 6939 9299 23930 ] med 0 aggregator ( 23930:121.127.1.235 ) community 59919:65002
api             route added to neighbor 10.90.90.2 local-ip 10.90.90.1 local-as 100 peer-as 200 router-id 10.90.90.1 family-allowed in-open : 95.175.64.0/22 next-hop 5.178.95.254 origin igp as-path [ 59919 41327 2914 1299 3225 ] aggregator ( 3225:10.20.100.28 ) community [ 2914:420 2914:1004 2914:2000 2914:3000 17152:1 41327:2914 41327:4500 41327:10001 41327:13000 41327:13001 ]
api             route added to neighbor 10.90.90.2 local-ip 10.90.90.1 local-as 100 peer-as 200 router-id 10.90.90.1 family-allowed in-open : 95.175.84.0/22 next-hop 5.178.95.254 origin igp as-path [ 59919 41327 2914 1299 3225 ] aggregator ( 3225:10.20.100.28 ) community [ 2914:420 2914:1004 2914:2000 2914:3000 17152:1 41327:2914 41327:4500 41327:10001 41327:13000 41327:13001 ]
api             route added to neighbor 10.90.90.2 local-ip 10.90.90.1 local-as 100 peer-as 200 router-id 10.90.90.1 family-allowed in-open : 95.175.68.0/22 next-hop 5.178.95.254 origin igp as-path [ 59919 41327 2914 1299 3225 ] aggregator ( 3225:10.20.100.28 ) community [ 2914:420 2914:1004 2914:2000 2914:3000 17152:1 41327:2914 41327:4500 41327:10001 41327:13000 41327:13001 ]
api             route added to neighbor 10.90.90.2 local-ip 10.90.90.1 local-as 100 peer-as 200 router-id 10.90.90.1 family-allowed in-open : 95.175.80.0/22 next-hop 5.178.95.254 origin igp as-path [ 59919 41327 2914 1299 3225 ] aggregator ( 3225:10.20.100.28 ) community [ 2914:420 2914:1004 2914:2000 2914:3000 17152:1 41327:2914 41327:4500 41327:10001 41327:13000 41327:13001 ]
api             route added to neighbor 10.90.90.2 local-ip 10.90.90.1 local-as 100 peer-as 200 router-id 10.90.90.1 family-allowed in-open : 95.175.92.0/22 next-hop 5.178.95.254 origin igp as-path [ 59919 41327 2914 1299 3225 ] aggregator ( 3225:10.20.100.28 ) community [ 2914:420 2914:1004 2914:2000 2914:3000 17152:1 41327:2914 41327:4500 41327:10001 41327:13000 41327:13001 ]
api             route added to neighbor 10.90.90.2 local-ip 10.90.90.1 local-as 100 peer-as 200 router-id 10.90.90.1 family-allowed in-open : 95.175.88.0/22 next-hop 5.178.95.254 origin igp as-path [ 59919 41327 2914 1299 3225 ] aggregator ( 3225:10.20.100.28 ) community [ 2914:420 2914:1004 2914:2000 2914:3000 17152:1 41327:2914 41327:4500 41327:10001 41327:13000 41327:13001 ]
api             route added to neighbor 10.90.90.2 local-ip 10.90.90.1 local-as 100 peer-as 200 router-id 10.90.90.1 family-allowed in-open : 162.217.44.0/22 next-hop 23.139.160.31 origin igp as-path [ 26073 6939 3356 13614 ] community [ 6939:7025 6939:8840 6939:9001 ]
(...snip)