search

CiscoTestAutomation / genieparser

sub-component of Genie that parse the device output into structured datastructure
Apache License 2.0
253 stars 388 forks source link

NX-OS show interface status parser overflows in some cases #736

Open blue212121 opened 1 year ago

blue212121 commented 1 year ago

Issue is with p1_1 for me, example input, and result: input:

Eth1/2        --                 xcvrAbsen 1         auto    auto    --         
Eth1/3        testdesc           notconnec 1         auto    auto    1000base-T 
Eth1/4        testdesc           connected 1         full    1000    1000base-T

result:

{'interface': 'Eth1/2', 'name': '--                 xcvrAbsen', 'status': '1', 'vlan': 'auto', 'duplex_code': 'auto', 'port_speed': '--', 'type': ''}
{'interface': 'Eth1/3', 'name': 'testdesc           notconnec', 'status': '1', 'vlan': 'auto', 'duplex_code': 'auto', 'port_speed': '1000base-T', 'type': ''}
{'interface': 'Eth1/4', 'name': 'testdesc', 'status': 'connected', 'vlan': '1', 'duplex_code': 'full', 'port_speed': '1000', 'type': '1000base-T '}

'connected' state is working i suppose because the port speed is num in that case, hence not matchin [a-z] in duplex_code

Working regex for me by defining the possible vlan values, so that the <name> wont overflow and match the <status> variable as well (since <type> can match whitespace only as well, it will match even a single space at the end after the actual type is what im guessing):

(?P<interface>(\S+))\s+
(?P<name>([\S\s]+))(?<! )\s+
(?P<status>([a-zA-Z]+))\s+
(?P<vlan>(\d+|routed|trunk|f-path|--))\s+
(?P<duplex_code>([a-z]+))\s+
(?P<port_speed>(\S+))
(\s+(?P<type>([\S]+)))?\s*$

but may be better regex solution (not sure if there are other values for vlan field) thats why i opened an issue and not a pull request

hardware: N3K-C3064PQ-10GE software: NXOS: version 7.0(3)I7(9) genie==23.2 pyats==23.2

sumitsehgal123 commented 1 year ago

Hi . Thanks for reaching out piestack support team, I will be looking at your ticket and reviewing it . Please make sure add the logs / pyats version / package list . Recommend is to always use the latest pyats version (i.e. Current released version 23.3)

blue212121 commented 1 year ago

Hello, current released version was 23.2 at time of reporting, as indicated in post. Is this issue confirmed fixed in 23.3? thanks

sumitsehgal123 commented 1 year ago

Hi blue212121, Thanks for your patience. I reviewed your ticket and found that the result you framed is not in a proper format as the key value paired wrongly. Kindly frame that in a proper format and try use this regex.

p1 = re.compile(r'(?P(\S+))\s+(?P(\S+))?\s' r'+(?P(\S+))?\s+(?P(\S+))' r' +(?P(\S+))\s' r'+(?P(\S+))(\s(?P(\S)))?$')

p1_1 = re.compile(r'(?P(\S+))\s+' r'(?P([\S\s]+))(?<! )\s+' r'(?P(\S+))\s+' r'(?P(\S+))\s+' r'(?P([a-z]+))\s+' r'(?P(\S+))\s' r'(?P([\S\s]))$')

sumitsehgal123 commented 1 year ago

Hi blue212121, I hope your issue has been resolved with above solution . Please let me know are you still facing any issue or not ?

blue212121 commented 1 year ago

Hi sumit, havent got time to test the above solution yet, but looks similar to the original p1_1 which produced the issue, but will check eventually, thank you for your patience

blue212121 commented 1 year ago

Hi @sumitsehgal123 ,

sorry, accidentally closed issue, but it is still not fixed, proposed solution is syntactically incorrect as it is missing the names from the named groups:

p1_1 = re.compile(r'(?P(\S+))\s+'
r'(?P([\S\s]+))(?<! )\s+'
r'(?P(\S+))\s+'
r'(?P(\S+))\s+'
r'(?P<duplex_code>([a-z]+))\s+'
r'(?P<port_speed>(\S+))\s*'
r'(?P([\S\s]*))$')

but even when filling in from source code, the result still produces the same wrong groupdict from the original post from this issue, where description and status merge together.

problem regex:


        p1_1 = re.compile(r'(?P<interface>(\S+))\s+'
                          r'(?P<name>([\S\s]+))(?<! )\s+'
                          r'(?P<status>(\S+))\s+'
                          r'(?P<vlan>(\S+))\s+'
                          r'(?P<duplex_code>([a-z]+))\s+'
                          r'(?P<port_speed>(\S+))\s*'
                          r'(?P<type>([\S\s]*))$')

so unfortunely this issue still exists

sumitsehgal123 commented 1 year ago

Hi blue212121, I am working on your ticket ,I will get back to you .

sumitsehgal123 commented 1 year ago

Hi Blue212121, I tried your input and I am getting the below output . Please try to do from your end . the output should be as the below format.

dev.parse("show interface status", output=output) {'interfaces': {'Ethernet1/2': {'status': 'xcvrAbsen', 'vlan': '1', 'duplex_code': 'auto', 'port_speed': 'auto'}, 'Ethernet1/3': {'name': 'testdesc', 'status': 'notconnec', 'vlan': '1', 'duplex_code': 'auto', 'port_speed': 'auto', 'type': '1000base-T'}, 'Ethernet1/4': {'name': 'testdesc', 'status': 'connected', 'vlan': '1', 'duplex_code': 'full', 'port_speed': '1000', 'type': '1000base-T'}}}

sumitsehgal123 commented 1 year ago

After make changes in the source file , you have to give make develop and make json in the genieparser directory. After that please try to execute the parser.it will give the excepted output .

blue212121 commented 1 year ago

Hi Blue212121, I tried your input and I am getting the below output . Please try to do from your end . the output should be as the below format.

dev.parse("show interface status", output=output) {'interfaces': {'Ethernet1/2': {'status': 'xcvrAbsen', 'vlan': '1', 'duplex_code': 'auto', 'port_speed': 'auto'}, 'Ethernet1/3': {'name': 'testdesc', 'status': 'notconnec', 'vlan': '1', 'duplex_code': 'auto', 'port_speed': 'auto', 'type': '1000base-T'}, 'Ethernet1/4': {'name': 'testdesc', 'status': 'connected', 'vlan': '1', 'duplex_code': 'full', 'port_speed': '1000', 'type': '1000base-T'}}}

Hi @sumitsehgal123

apparently this issue is reproducible when using the nxos parser, when there is a space in the description, see below on Ethernet1/3:

>>> print(json.dumps(dev.parse("show interface status", output=teststring), indent=4))
{
    "interfaces": {
        "Ethernet1/2": {
            "status": "xcvrAbsen",
            "vlan": "1",
            "duplex_code": "auto",
            "port_speed": "auto"
        },
        "Ethernet1/3": {
            "name": "testdesc is 1      notconnec",
            "status": "1",
            "vlan": "auto",
            "duplex_code": "auto",
            "port_speed": "1000base-T"
        },
        "Ethernet1/4": {
            "name": "testdesc is 2",
            "status": "connected",
            "vlan": "1",
            "duplex_code": "full",
            "port_speed": "1000",
            "type": "1000base-T"
        }
    }
}

teststring: 

Eth1/2        --                 xcvrAbsen 1         auto    auto    --         
Eth1/3        testdesc is 1      notconnec 1         auto    auto    1000base-T 
Eth1/4        testdesc is 2      connected 1         full    1000    1000base-T
sumitsehgal123 commented 1 year ago

Hi Blue212121, I am not able to reproduce the issue from my side ,

blue212121 commented 1 year ago

Hi Blue212121, I am not able to reproduce the issue from my side ,

Hello @sumitsehgal123,

that is strange indeed... with provided teststring, the resulting json is correct for you without the description regex group overflowing into the status?

here is my pip freeze (only installed pyats[full] inside venv):

aiofiles==23.1.0
aiohttp==3.8.4
aiohttp-swagger==1.0.16
aiosignal==1.3.1
arrow==1.2.3
async-lru==2.0.2
async-timeout==4.0.2
attrs==23.1.0
backports.ssl==0.0.9
bcrypt==4.0.1
binaryornot==0.4.4
certifi==2022.12.7
cffi==1.15.1
chardet==4.0.0
charset-normalizer==3.1.0
click==8.1.3
cookiecutter==2.1.1
cryptography==40.0.2
dill==0.3.6
distro==1.8.0
frozenlist==1.3.3
genie==23.3
genie.libs.clean==23.3
genie.libs.conf==23.3
genie.libs.filetransferutils==23.3
genie.libs.health==23.3
genie.libs.ops==23.3
genie.libs.parser==23.3
genie.libs.robot==23.3
genie.libs.sdk==23.3
genie.telemetry==23.3
genie.trafficgen==23.3
gitdb==4.0.10
GitPython==3.1.31
grpcio==1.54.0
idna==3.4
IxNetwork==9.0.1915.16
ixnetwork-restpy==1.1.9
Jinja2==3.1.2
jinja2-time==0.2.0
jsonpickle==3.0.1
junit-xml==1.9
lxml==4.9.2
MarkupSafe==2.1.2
multidict==6.0.4
ncclient==0.6.13
netaddr==0.8.0
packaging==23.1
paramiko==3.1.0
pathspec==0.11.1
prettytable==3.7.0
protobuf==3.20.3
psutil==5.9.5
pyats==23.3
pyats.aereport==23.3
pyats.aetest==23.3
pyats.async==23.3
pyats.connections==23.3
pyats.contrib==23.3
pyats.datastructures==23.3
pyats.easypy==23.3
pyats.kleenex==23.3
pyats.log==23.3
pyats.reporter==23.3
pyats.results==23.3
pyats.robot==23.3
pyats.tcl==23.3
pyats.topology==23.3
pyats.utils==23.3
pycparser==2.21
pyftpdlib==1.5.7
PyNaCl==1.5.0
pyOpenSSL==23.1.1
python-dateutil==2.8.2
python-engineio==3.14.2
python-slugify==8.0.1
python-socketio==4.6.1
PyYAML==6.0
requests==2.28.2
requests-toolbelt==0.10.1
robotframework==6.0.2
ruamel.yaml==0.17.21
ruamel.yaml.clib==0.2.7
six==1.16.0
smmap==5.0.0
text-unidecode==1.3
tftpy==0.8.0
tqdm==4.65.0
typing_extensions==4.5.0
unicon==23.3
unicon.plugins==23.3
urllib3==1.26.15
wcwidth==0.2.6
websocket-client==1.5.1
xlrd==1.2.0
XlsxWriter==3.1.0
xlwt==1.3.0
xmltodict==0.13.0
yamllint==1.31.0
yang.connector==23.3
yarl==1.9.1
sumitsehgal123 commented 1 year ago

Hi blue212121, Based on the output provided, I tried to reproduce the issue by following the below steps,

pyats shell --testbed-file testbed.yaml dev = testbed.devices['***'] output = ''' Eth1/2 -- xcvrAbsen 1 auto auto -- Eth1/3 testdesc notconnec 1 auto auto 1000base-T Eth1/4 testdesc connected 1 full 1000 1000base-T ''' dev.parse("show interface status ", output=output)

output: {'interfaces': {'Ethernet1/2': {'status': 'xcvrAbsen', 'vlan': '1', 'duplex_code': 'auto', 'port_speed': 'auto'}, 'Ethernet1/3': {'name': 'testdesc', 'status': 'notconnec', 'vlan': '1', 'duplex_code': 'auto', 'port_speed': 'auto', 'type': '1000base-T'}, 'Ethernet1/4': {'name': 'testdesc', 'status': 'connected', 'vlan': '1', 'duplex_code': 'full', 'port_speed': '1000', 'type': '1000base-T'}}}

Since I'm unable to reproduce the issue, can you please try again. The pyats version used is 23.3 and python version 3.8. Let me know if I can close this issue since it is not reproducible in my end.

blue212121 commented 1 year ago

Hi blue212121, Based on the output provided, I tried to reproduce the issue by following the below steps,

pyats shell --testbed-file testbed.yaml dev = testbed.devices['***'] output = ''' Eth1/2 -- xcvrAbsen 1 auto auto -- Eth1/3 testdesc notconnec 1 auto auto 1000base-T Eth1/4 testdesc connected 1 full 1000 1000base-T ''' dev.parse("show interface status ", output=output)

output: {'interfaces': {'Ethernet1/2': {'status': 'xcvrAbsen', 'vlan': '1', 'duplex_code': 'auto', 'port_speed': 'auto'}, 'Ethernet1/3': {'name': 'testdesc', 'status': 'notconnec', 'vlan': '1', 'duplex_code': 'auto', 'port_speed': 'auto', 'type': '1000base-T'}, 'Ethernet1/4': {'name': 'testdesc', 'status': 'connected', 'vlan': '1', 'duplex_code': 'full', 'port_speed': '1000', 'type': '1000base-T'}}}

Since I'm unable to reproduce the issue, can you please try again. The pyats version used is 23.3 and python version 3.8. Let me know if I can close this issue since it is not reproducible in my end.

Hello @sumitsehgal123, you are using wrong test string, as mentioned before, issue occures to me, when there is whitespace in the description. Could you please try testing with below string?

Eth1/2        --                 xcvrAbsen 1         auto    auto    --         
Eth1/3        testdesc is 1      notconnec 1         auto    auto    1000base-T 
Eth1/4        testdesc is 2      connected 1         full    1000    1000base-T
sumitsehgal123 commented 1 year ago

Hi Blue , Please find the below patter which is matching with below output .

output= Eth1/4 testdesc is 2 connected 1 full 1000 1000base-T Eth1/3 testdesc is 1 notconnec 1 auto auto 1000base-T Eth1/2 -- xcvrAbsen 1 auto auto --

Pattern : ^(?P[\w\/]+)\s+(?P(\w+\s\w+\s\d|[\w-]+|[\w\s()[]_,]+))\s+(?P(\w+\s\d|\w+))\s+(?P(\w+))\s+(?P(\w+))\s+(?P([\w-]+))(\s*(?P([\w-]+)))?$

blue212121 commented 1 year ago

Hello @sumitsehgal123

it is matching, but the parsing itself is incorrect as shown previously... hence why issue is created...

sumitsehgal123 commented 1 year ago

Hi , Since genieparser is community driven, would you like to contribute to the library by following this documentation.

Please let me know if you need any help during the process of PR creation. If not let me know we will add this to our backlog and let you know once it is fixed

blue212121 commented 1 year ago

Hello @sumitsehgal123 i proposed a solution in the original post, which fixes my issue, but not sure if it breaks anything else, as i have no time to thoroughly test it at this time. Hence issue and not pr.

sumitsehgal123 commented 1 year ago

Hi I have raised the PR for issue , The change will reflect from next released of pyats .

blue212121 commented 1 year ago

Hello @sumitsehgal123 ,

thank you, looking forward to switching back to main :)

sumitsehgal123 commented 1 year ago

Hi , The PR is merged related to issue and its not reproducible .I checked from my side , Please check and confirm .


pyats shell --testbed-file testbed.yaml
dev = testbed.devices['***']
output='''Eth1/2 -- xcvrAbsen 1 auto auto --
Eth1/3 testdesc notconnec 1 auto auto 1000base-T
Eth1/4 testdesc connected 1 full 1000 1000base-T '''
dev.parse("show interface status",output=output)

Output=
{'interfaces': {'Ethernet1/2': {'name': '--', 'status': 'xcvrAbsen 1', 'vlan': 'auto', 'duplex_code': 'auto', 'port_speed': '--'},
 'Ethernet1/3': {'name': 'testdesc', 'status': 'notconnec 1', 'vlan': 'auto', 'duplex_code': 'auto', 'port_speed': '1000base-T'}, 
'Ethernet1/4': {'name': 'testdesc', 'status': 'connected 1', 'vlan': 'full', 'duplex_code': '1000', 'port_speed': '1000base-T'}}}

if you need any further assistance .please let me know .