search

napalm-automation / napalm

Network Automation and Programmability Abstraction Layer with Multivendor support
Apache License 2.0
2.25k stars 555 forks source link

key error when BGP peer has the same import/export policy names #1359

Closed lucasalvatore closed 3 years ago

lucasalvatore commented 3 years ago

Description of Issue/Question

When running get_bgp_config() on a junos device, if a peer as the same peer level import/export policies, a key error is thrown.

Did you follow the steps from https://github.com/napalm-automation/napalm#faq

(Place an x between the square brackets where applicable)

Setup

napalm version

(Paste verbatim output from pip freeze | grep napalm between quotes below)

~ » pip freeze | grep napalm  
napalm==3.2.0

Network operating system version

(Paste verbatim output from show version - or equivalent - between quotes below)

Model: qfx10002-72q
Junos: 19.4R2-S1.2

Steps to Reproduce the Issue

have a bgp group configured like this:

luca@bsr1.ams1> show configuration protocols bgp group V6-INTERNAL-EDGE
type external;
import [ V6-INTERNAL-EDGE-IN DENY ];
family inet6 {
    unicast;
}
export [ V6-INTERNAL-EDGE-OUT DENY ];
peer-as 65530;
local-as 65520 
multipath;
neighbor x.x.x.x {
    description "switch1";
    }
}
neighbor y.y.y.y {
    description "switch2";
    import DENY;
    export DENY;
}

Then try to get the bgp config for that group:

def napalm_connect(device):
    username = getpass.getuser()
    driver = get_network_driver("junos")
    optional_args = {"use_keys": True, "allow_agent": True}
    router = driver(str(device), username, "", optional_args=optional_args)
    router.open()
    return router

router = napalm_connect('bsr1.ams1')
print(router.get_bgp_config(group='V6-INTERNAL-EDGE'))

Error Traceback

In [214]: router = napalm_connect('bsr1.ams1')
     ...: print(router.get_bgp_config(group='V6-INTERNAL-EDGE'))
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-214-591df75972d8> in <module>
      1 router = napalm_connect('bsr1.ams1')
----> 2 print(router.get_bgp_config(group='V6-INTERNAL-EDGE'))

~/virtualenv/python3.8/lib/python3.8/site-packages/napalm/junos/junos.py in get_bgp_config(self, group, neighbor)
   1219                     if key in ["export_policy"]:
   1220                         # next-hop self is applied on export IBGP sessions
-> 1221                         bgp_peer_details["nhs"] = _check_nhs(value, nhs_policies)
   1222                     if key in ["export_policy", "import_policy"]:
   1223                         if isinstance(value, list):

~/virtualenv/python3.8/lib/python3.8/site-packages/napalm/junos/junos.py in _check_nhs(policies, nhs_policies)
   1025             # Return True if "next-hop self" was found in any of the policies p
   1026             for p in policies:
-> 1027                 if nhs_policies[p] is True:
   1028                     return True
   1029             return False

KeyError: 'DENY'

Additional info

Trying to get the other neighbor without the peer level policies works fine:

In [213]: print(router.get_bgp_config(group='V6-INTERNAL-EDGE',neighbor='x.x.x.x'))

{'V6-INTERNAL-EDGE': {'type': 'external', 'apply_groups': [], 'remove_private_as': False, 'multipath': True, 'multihop_ttl': 0, 'description': 'INTERNAL: POP-Local ToRs (IPv6)', 'local_address': '', 'local_as': 65520, 'remote_as': 65530, 'import_policy': 'V6-INTERNAL-EDGE-IN DENY', 'export_policy': 'V6-INTERNAL-EDGE-OUT DENY', 'prefix_limit': {}, 'neighbors': {'x.x.x.x': {'authentication_key': '', 'route_reflector_client': False, 'nhs': False, 'description': 'switch1', 'local_address': '', 'local_as': 0, 'remote_as': 0, 'import_policy': '', 'export_policy': '', 'prefix_limit': {}}}}}

however trying to get the one with the peer level policy fails:

In [215]: print(router.get_bgp_config(group='V6-INTERNAL-EDGE',neighbor='y.y.y.y'))
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-215-f6135de9bf29> in <module>
----> 1 print(router.get_bgp_config(group='V6-INTERNAL-EDGE',neighbor='y.y.y.y'))

~/virtualenv/python3.8/lib/python3.8/site-packages/napalm/junos/junos.py in get_bgp_config(self, group, neighbor)
   1219                     if key in ["export_policy"]:
   1220                         # next-hop self is applied on export IBGP sessions
-> 1221                         bgp_peer_details["nhs"] = _check_nhs(value, nhs_policies)
   1222                     if key in ["export_policy", "import_policy"]:
   1223                         if isinstance(value, list):

~/virtualenv/python3.8/lib/python3.8/site-packages/napalm/junos/junos.py in _check_nhs(policies, nhs_policies)
   1025             # Return True if "next-hop self" was found in any of the policies p
   1026             for p in policies:
-> 1027                 if nhs_policies[p] is True:
   1028                     return True
   1029             return False

KeyError: 'DENY'
mirceaulinic commented 3 years ago

Hi @lucasalvatore,

I have configured the following:

test@router1> show configuration protocols bgp group V6-INTERNAL-EDGE 
type external;
import [ V6-INTERNAL-EDGE-IN DENY ];
family inet6 {
    unicast;
}
export [ V6-INTERNAL-EDGE-OUT DENY ];
peer-as 65530;
local-as 65520;
multipath;
neighbor 10.0.0.1 {
    description switch1;
}
neighbor 10.0.0.2 {
    description switch2;
    import DENY;
    export DENY;
}

With NAPALM 3.2.0 I am not able to reproduce the error:

>>> import napalm
>>> d = napalm.get_network_driver('junos')
>>> j = d('router1', 'test', 'test1234')
>>> j.open()
>>> 
>>> j.get_bgp_config()
{'V6-INTERNAL-EDGE': {'type': 'external', 'apply_groups': [], 'remove_private_as': False, 'multipath': True, 'multihop_ttl': 0, 'description': '', 'local_address': '', 'local_as': 65520, 'remote_as': 65530, 'import_policy': 'V6-INTERNAL-EDGE-IN DENY', 'export_policy': 'V6-INTERNAL-EDGE-OUT DENY', 'prefix_limit': {}, 'neighbors': {'10.0.0.1': {'authentication_key': '', 'route_reflector_client': False, 'nhs': False, 'description': 'switch1', 'local_address': '', 'local_as': 0, 'remote_as': 0, 'import_policy': '', 'export_policy': '', 'prefix_limit': {}}, '10.0.0.2': {'authentication_key': '', 'route_reflector_client': False, 'nhs': False, 'description': 'switch2', 'local_address': '', 'local_as': 0, 'remote_as': 0, 'import_policy': 'DENY', 'export_policy': 'DENY', 'prefix_limit': {}}}}}
>>> j.get_bgp_config(group='V6-INTERNAL-EDGE')
{'V6-INTERNAL-EDGE': {'type': 'external', 'apply_groups': [], 'remove_private_as': False, 'multipath': True, 'multihop_ttl': 0, 'description': '', 'local_address': '', 'local_as': 65520, 'remote_as': 65530, 'import_policy': 'V6-INTERNAL-EDGE-IN DENY', 'export_policy': 'V6-INTERNAL-EDGE-OUT DENY', 'prefix_limit': {}, 'neighbors': {'10.0.0.1': {'authentication_key': '', 'route_reflector_client': False, 'nhs': False, 'description': 'switch1', 'local_address': '', 'local_as': 0, 'remote_as': 0, 'import_policy': '', 'export_policy': '', 'prefix_limit': {}}, '10.0.0.2': {'authentication_key': '', 'route_reflector_client': False, 'nhs': False, 'description': 'switch2', 'local_address': '', 'local_as': 0, 'remote_as': 0, 'import_policy': 'DENY', 'export_policy': 'DENY', 'prefix_limit': {}}}}}
>>> j.get_bgp_config(group='V6-INTERNAL-EDGE', neighbor='10.0.0.1')
{'V6-INTERNAL-EDGE': {'type': 'external', 'apply_groups': [], 'remove_private_as': False, 'multipath': True, 'multihop_ttl': 0, 'description': '', 'local_address': '', 'local_as': 65520, 'remote_as': 65530, 'import_policy': 'V6-INTERNAL-EDGE-IN DENY', 'export_policy': 'V6-INTERNAL-EDGE-OUT DENY', 'prefix_limit': {}, 'neighbors': {'10.0.0.1': {'authentication_key': '', 'route_reflector_client': False, 'nhs': False, 'description': 'switch1', 'local_address': '', 'local_as': 0, 'remote_as': 0, 'import_policy': '', 'export_policy': '', 'prefix_limit': {}}}}}
>>> j.get_bgp_config(group='V6-INTERNAL-EDGE', neighbor='10.0.0.2')
{'V6-INTERNAL-EDGE': {'type': 'external', 'apply_groups': [], 'remove_private_as': False, 'multipath': True, 'multihop_ttl': 0, 'description': '', 'local_address': '', 'local_as': 65520, 'remote_as': 65530, 'import_policy': 'V6-INTERNAL-EDGE-IN DENY', 'export_policy': 'V6-INTERNAL-EDGE-OUT DENY', 'prefix_limit': {}, 'neighbors': {'10.0.0.2': {'authentication_key': '', 'route_reflector_client': False, 'nhs': False, 'description': 'switch2', 'local_address': '', 'local_as': 0, 'remote_as': 0, 'import_policy': 'DENY', 'export_policy': 'DENY', 'prefix_limit': {}}}}}
>>> 
>>> napalm.__version__
'3.2.0'
>>>

Let me know if I'm missing anything. Would you also be able to share your XML view? I'm seeing the following on my vMX (similar output on a prod QFX10k2 running Junos 17, but don't exclude the possibility something may differ on a another model / version combination):

test@router1> show configuration protocols bgp group V6-INTERNAL-EDGE | display xml 
<rpc-reply xmlns:junos="http://xml.juniper.net/junos/17.2R1/junos">
    <configuration junos:commit-seconds="1610708495" junos:commit-localtime="2021-01-15 11:01:35 UTC" junos:commit-user="test">
            <protocols>
                <bgp>
                    <group>
                        <name>V6-INTERNAL-EDGE</name>
                        <type>external</type>
                        <import>V6-INTERNAL-EDGE-IN</import>
                        <import>DENY</import>
                        <family>
                            <inet6>
                                <unicast>
                                </unicast>
                            </inet6>
                        </family>
                        <export>V6-INTERNAL-EDGE-OUT</export>
                        <export>DENY</export>
                        <peer-as>65530</peer-as>
                        <local-as>
                            <as-number>65520</as-number>
                        </local-as>
                        <multipath>
                        </multipath>
                        <neighbor>
                            <name>10.0.0.1</name>
                            <description>switch1</description>
                        </neighbor>
                        <neighbor>
                            <name>10.0.0.2</name>
                            <description>switch2</description>
                            <import>DENY</import>
                            <export>DENY</export>
                        </neighbor>
                    </group>
                </bgp>
            </protocols>
    </configuration>
    <cli>
        <banner></banner>
    </cli>
</rpc-reply>
lucasalvatore commented 3 years ago

hi @mirceaulinic interesting indeed. Here is the XML

luca@bsr1.ams1> show configuration protocols bgp group V6-INTERNAL-EDGE | display xml
<rpc-reply xmlns:junos="http://xml.juniper.net/junos/19.4R0/junos">
    <configuration junos:commit-seconds="1610706240" junos:commit-localtime="2021-01-15 10:24:00 UTC" junos:commit-user="acl-updater">
            <protocols>
                <bgp>
                    <group>
                        <name>V6-INTERNAL-EDGE</name>
                        <type>external</type>
                        <description>INTERNAL: POP-Local ToRs (IPv6)</description>
                        <import>V6-INTERNAL-EDGE-IN</import>
                        <import>DENY</import>
                        <family>
                            <inet6>
                                <unicast>
                                </unicast>
                            </inet6>
                        </family>
                        <export>V6-INTERNAL-EDGE-OUT</export>
                        <export>DENY</export>
                        <peer-as>65530</peer-as>
                        <local-as>
                            <as-number>65520</as-number>
                            <loops>10</loops>
                        </local-as>
                        <multipath>
                        </multipath>
                        <neighbor>
                            <name>x.x.x.x</name>
                            <description>switch1</description>
                        </neighbor>
                        <neighbor>
                            <name>y.y.y.y</name>
                            <description>switch2</description>
                            <import>DENY</import>
                            <export>DENY</export>
                        </neighbor>
                    </group>
                </bgp>
            </protocols>
    </configuration>
    <cli>
        <banner>{master:0}</banner>
    </cli>
</rpc-reply>
lucasalvatore commented 3 years ago

i tested this on a different device with similar config but running junos 18.4R3.3 and it worked without issues. Perhaps there is something weird on junos 19.4

mirceaulinic commented 3 years ago

Sounds like there's something different on 19 indeed. The issue might be caused by the junos_policy_nhs_config_table. Do you have any next-hop self policy? Do you notice any difference between the XML for the policies in Junos 19 vs 18? Are you able to share?

lucasalvatore commented 3 years ago

I checked again and i was able to replicate this issue on another QFX10K running junos 18 (not sure what i did previously). There are next-hop self policies configured on these devices, but not for this specific group. Here is the full config including policies which causes the issue (IP addresses have been sanitized) 

luca@bsr1.atl2> show configuration protocols bgp group V6-INTERNAL-EDGE
type external;
description "INTERNAL: POP-Local ToRs (IPv6)";
import V6-INTERNAL-EDGE-IN;
family inet6 {
    unicast;
}
export V6-INTERNAL-EDGE-OUT;
peer-as 65530;
local-as 65520 loops 10;
multipath;
neighbor fc00::111 {
    description "esr1.rk03.atl2 - ae83";
}
neighbor fc00::107 {
    description "esr1.rk02.atl2 - ae81";
}
neighbor fc00::f7 {
    description esr2a.rk02.atl2;
}
neighbor fc00::12d {
    description esr2.rk03.atl2;
    import DENY;
    export DENY;
}

luca@bsr1.atl2> show configuration policy-options policy-statement V6-INTERNAL-EDGE-IN
term PREPEND {
    then {
        as-path-prepend 65520;
        next term;
    }
}
term CUSTOMER-TE {
    from community [ CUSTOMER-TE-TRANSIT-ONLY CUSTOMER-TE-PEERING-ONLY ];
    then {
        local-preference 300;
        community delete PACKET-CUSTOMER;
        community add PACKET-ATL2-CUSTOMER;
        accept;
    }
}
term PACKET-CUSTOMER {
    from community PACKET-CUSTOMER;
    then {
        local-preference 300;
        community add PACKET-ATL2-CUSTOMER;
        accept;
    }
}
term REST {
    then accept;
}

luca@bsr1.atl2> show configuration policy-options policy-statement V6-INTERNAL-EDGE-OUT
term DEFAULT-ROUTE {
    from {
        route-filter ::0/0 exact;
    }
    then accept;
}
then reject;

{master:0}
luca@bsr1.atl2>

luca@bsr1.atl2> show configuration policy-options policy-statement DENY | display inheritance no-comments
term DENY {
    then reject;
}

{master:0}
luca@bsr1.atl2>

XML

luca@bsr1.atl2> show configuration protocols bgp group V6-INTERNAL-EDGE | display xml
<rpc-reply xmlns:junos="http://xml.juniper.net/junos/18.4R1/junos">
    <configuration junos:commit-seconds="1610986158" junos:commit-localtime="2021-01-18 16:09:18 UTC" junos:commit-user="luca">
            <protocols>
                <bgp>
                    <group>
                        <name>V6-INTERNAL-EDGE</name>
                        <type>external</type>
                        <description>INTERNAL: POP-Local ToRs (IPv6)</description>
                        <import>V6-INTERNAL-EDGE-IN</import>
                        <family>
                            <inet6>
                                <unicast>
                                </unicast>
                            </inet6>
                        </family>
                        <export>V6-INTERNAL-EDGE-OUT</export>
                        <peer-as>65530</peer-as>
                        <local-as>
                            <as-number>65520</as-number>
                            <loops>10</loops>
                        </local-as>
                        <multipath>
                        </multipath>
                        <neighbor>
                            <name>fc00::111</name>
                            <description>esr1.rk03.atl2 - ae83</description>
                        </neighbor>
                        <neighbor>
                            <name>fc00::107</name>
                            <description>esr1.rk02.atl2 - ae81</description>
                        </neighbor>
                        <neighbor>
                            <name>fc00::f7</name>
                            <description>esr2a.rk02.atl2</description>
                        </neighbor>
                        <neighbor>
                            <name>fc00::12d</name>
                            <description>esr2.rk03.atl2</description>
                            <import>DENY</import>
                            <export>DENY</export>
                        </neighbor>
                    </group>
                </bgp>
            </protocols>
    </configuration>
    <cli>
        <banner>{master:0}</banner>
    </cli>
</rpc-reply>

Napalm

In [69]: username = getpass.getuser()
    ...: driver = get_network_driver("junos")
    ...: optional_args = {"use_keys": True, "allow_agent": True}
    ...: router = driver("bsr1.atl2", username, "", optional_args=optional_args)
    ...: router.open()

In [70]: router.get_bgp_config(group='V6-INTERNAL-EDGE')
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-70-329755a443fa> in <module>
----> 1 router.get_bgp_config(group='V6-INTERNAL-EDGE')

~/virtualenv/python3.8/lib/python3.8/site-packages/napalm/junos/junos.py in get_bgp_config(self, group, neighbor)
   1219                     if key in ["export_policy"]:
   1220                         # next-hop self is applied on export IBGP sessions
-> 1221                         bgp_peer_details["nhs"] = _check_nhs(value, nhs_policies)
   1222                     if key in ["export_policy", "import_policy"]:
   1223                         if isinstance(value, list):

~/virtualenv/python3.8/lib/python3.8/site-packages/napalm/junos/junos.py in _check_nhs(policies, nhs_policies)
   1025             # Return True if "next-hop self" was found in any of the policies p
   1026             for p in policies:
-> 1027                 if nhs_policies[p] is True:
   1028                     return True
   1029             return False

KeyError: 'DENY'
mirceaulinic commented 3 years ago

Could you share the text and the XML view for the policies having next-hop self @lucasalvatore? The code that is failing is evaluating all the policies (although strange that it doesn't fail when retrieving all). Would you also be able to add some print() lines in the code, after this line, e.g., print(policy_name, '==>', boolean). Not sure it's that, but may be starting point.

lucasalvatore commented 3 years ago

here you go @mirceaulinic 

>>> from napalm import get_network_driver
>>> username = getpass.getuser()
>>> driver = get_network_driver("junos")
>>> optional_args = {"use_keys": True, "allow_agent": True}
>>> device = 'bsr1.atl2'
>>> router = driver(str(device), username, "", optional_args=optional_args)
>>> router.open()
router.get_bgp_config(group='V6-INTERNAL-EDGE')
>>> router.get_bgp_config(group='V6-INTERNAL-EDGE')
INTERNAL-ATL2-IN ==> None
INTERNAL-ATL2-OUT ==> [True, True, True]
INTERNAL-ATL2-RR-IN ==> None
INTERNAL-ATL2-RR-OUT ==> [True, True, True, True]
INTERNAL-BSR-OUT ==> [True, True, True]
INTERNAL-EDGE-IN ==> None
INTERNAL-EDGE-OUT ==> None
INTERNAL-MSR-IN ==> None
INTERNAL-MSR-OUT ==> True
INTERNAL-OUT ==> None
KENTIK-OUT ==> None
V6-INTERNAL-ATL2-IN ==> None
V6-INTERNAL-ATL2-OUT ==> [True, True, True]
V6-INTERNAL-ATL2-RR-IN ==> None
V6-INTERNAL-ATL2-RR-OUT ==> [True, True]
V6-INTERNAL-EDGE-IN ==> None
V6-INTERNAL-EDGE-OUT ==> None
VXLAN-IN ==> None
VXLAN-OUT ==> None
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/luca/virtualenv/python3.8/lib/python3.8/site-packages/napalm/junos/junos.py", line 1223, in get_bgp_config
    bgp_peer_details["nhs"] = _check_nhs(value, nhs_policies)
  File "/Users/luca/virtualenv/python3.8/lib/python3.8/site-packages/napalm/junos/junos.py", line 1027, in _check_nhs
    if nhs_policies[p] is True:
KeyError: 'DENY'

All policies with NHS are nhs_policies.txt in the attached text file.

mirceaulinic commented 3 years ago

Should be solved via #1362 and #1363.