search

Juniper / py-junos-eznc

Python library for Junos automation
https://www.juniper.net/documentation/en_US/junos-pyez/information-products/pathway-pages/junos-pyez-developer-guide.html
Apache License 2.0
670 stars 344 forks source link

ignore_warnings causing "'str' object has no attribute '_root'" #1245

Closed kpetremann closed 1 month ago

kpetremann commented 1 year ago

Hello team,

We are encountering this issue when loading a configuration with ignore_warning feature enabled:

Traceback (most recent call last):
  File "/opt/salt/venv/lib64/python3.9/site-packages/salt/utils/napalm.py", line 169, in call
    out = getattr(napalm_device.get("DRIVER"), method)(*args, **kwargs)
  File "/opt/salt/venv/lib64/python3.9/site-packages/napalm/junos/junos.py", line 284, in load_merge_candidate
    self._load_candidate(filename, config, False)
  File "/opt/salt/venv/lib64/python3.9/site-packages/napalm/junos/junos.py", line 263, in _load_candidate
    self.device.cu.load(
  File "/opt/salt/venv/lib64/python3.9/site-packages/jnpr/junos/utils/config.py", line 589, in load
    return try_load(rpc_contents, rpc_xattrs, ignore_warning=ignore_warning)
  File "/opt/salt/venv/lib64/python3.9/site-packages/jnpr/junos/utils/config.py", line 500, in try_load
    got = self.rpc.load_config(
  File "/opt/salt/venv/lib64/python3.9/site-packages/jnpr/junos/rpcmeta.py", line 287, in load_config
    return self._junos.execute(rpc, ignore_warning=ignore_warning)
  File "/opt/salt/venv/lib64/python3.9/site-packages/jnpr/junos/decorators.py", line 76, in wrapper
    return function(*args, **kwargs)
  File "/opt/salt/venv/lib64/python3.9/site-packages/jnpr/junos/decorators.py", line 31, in wrapper
    return function(*args, **kwargs)
  File "/opt/salt/venv/lib64/python3.9/site-packages/jnpr/junos/device.py", line 834, in execute
    rpc_rsp_e = self._rpc_reply(
  File "/opt/salt/venv/lib64/python3.9/site-packages/jnpr/junos/decorators.py", line 155, in wrapper
    rsp = NCElement(
  File "/opt/salt/venv/lib64/python3.9/site-packages/ncclient/xml_.py", line 177, in __init__
    self.__doc = self.__transform_reply(result._root)
AttributeError: 'str' object has no attribute '_root'

This is causing an exception resulting, and the commit is discard.

The code in pyeznc is calling NCElement with an argument with the wrong type:

                rsp = NCElement(
                    etree.tostring(rsp, encoding=encode), self.transform()  <<< first arg is a string
                )._NCElement__doc
class NCElement(object):
    def __init__(self, result, transform_reply, huge_tree=False):
        self.__result = result                                   <<< a string
        self.__transform_reply = transform_reply
        self.__huge_tree = huge_tree
        if isinstance(transform_reply, types.FunctionType):
            self.__doc = self.__transform_reply(result._root)    <<< string does not have a _root argument
        else:
            self.__doc = self.remove_namespaces(self.__result)

Bypassing the error with a try except block confirms this is the root cause. I don't know what can be passed to NCElement instead of a string. I am not familiar with ncclient.

Can you help please?

Tested versions: junos-eznc=2.6.5 and 2.6.7

chidanandpujar commented 1 year ago

Hi @kpetremann Could you please share the RPC details or PyEZ script , So that I can reproduce the issue .

Thanks & Regards Chidanand

kpetremann commented 1 year ago

Hi @chidanandpujar,

Thanks for your response.

So far, we had the issue only when using Saltstack scheduler:

Our code base is open-source:

The Salt proxy-minion is setting ignore_warning to True.

I can try to create a docker compose based lab to reproduce the issue on demand. But I will be on vacation starting tomorrow, so I won't be able to provide you this before June :/

Note: I don't know why it only happens in a schedule context, which is weird. Maybe it happens after multiple execution. We did not run the state manually a lot.

FYI, we deployed this workaround which works. But it is not a proper way to fix the issue.

kpetremann commented 1 year ago

The loaded configuration looks like this: https://github.com/criteo/openconfig-saltstack/blob/main/tests/states/openconfig_routing_policy/data/functional_tests/expected_result_junos.txt

It happens even if the configuration is already up to date.

kpetremann commented 1 year ago

Additional information, we saw the issue on QFX 5100 & 5110, at least on version 18.4 and 20.2.

chidanandpujar commented 1 year ago

Hi @kpetremann , Thanks for the details . Is it possible to collect the netconf trace logs from the QFX device .

set system services netconf ssh set system services netconf traceoptions file netcof_trace.log set system services netconf traceoptions file size 1g set system services netconf traceoptions flag all commit.

Thanks & Regards Chidanand

kpetremann commented 1 year ago

@chidanandpujar, I reverted my fix to retrigger the issue, but so far no occurrence.

I'll check the result in June after my vacations. I'll keep you updated then.

kpetremann commented 1 year ago

hi @chidanandpujar, I am able to reproduce the issue. But when the workflow is triggered, the netconf_trace.log is created but it remains empty.

> show configuration system services netconf      
ssh;
traceoptions {
    file netconf_trace.log size 1g;
    flag all;
}
> show log netconf_trace.log                    

{master:0}
>
kpetremann commented 1 year ago

hi @chidanandpujar any update?

chidanandpujar commented 1 year ago

Hi @kpetremann Thanks for the information , I am yet to replicate this issue on our local setup. I will update you .

Thanks & Regards Chidanand

chidanandpujar commented 12 months ago

Hi @kpetremann 


     Thanks very much for sharing the details ,
     Netconf traces are empty , so it is not clear what is the RPC and configuration details are loaded .
    from the exception details , it is visible that  config load operation is performed and there might be error/warnings emitted as part of rpc execution .
something similar , cu.load(cnf, ignore_warning='statement not found')
``
Also,  I am not able to access the config file -https://github.com/criteo/openconfig-saltstack/blob/main/tests/states/openconfig_routing_policy/data/functional_tests/expected_result_junos.txt

Thanks
kpetremann commented 12 months ago

Hi @chidanandpujar,

For the config file example, the a directory has been renamed, here the new link: https://github.com/criteo/afk-saltstack/blob/main/tests/states/openconfig_routing_policy/data/integration_tests/expected_result_junos.txt

there might be error/warnings emitted as part of rpc execution

As we set the ignore_warning parameter to true, the warning should be ignore, am I right?

chidanandpujar commented 11 months ago

Hi @kpetremann Thanks for sharing the config file , yeah correct , if we set the ignore_warning to (true or string or list of strings ) which are emitted as part of rpc response, will be ignored .

https://www.juniper.net/documentation/us/en/software/junos-pyez/junos-pyez-developer/topics/task/junos-pyez-warnings-ignoring.html

I will try to replicate the issue on local test setup .

Thanks

chidanandpujar commented 11 months ago

Hi @kpetremann when I try to load the provided config statements , ignore_warning looks to be not working . I am able to replicated the issue with ignore_warning .

Traceback (most recent call last):
  File "/root/pyez_release_268_test1/venv/lib/python3.9/site-packages/junos_eznc-2.6.8+0.g0745dd0e.dirty-py3.9.egg/jnpr/junos/device.py", line 834, in execute
    rpc_rsp_e = self._rpc_reply(
  File "/root/pyez_release_268_test1/venv/lib/python3.9/site-packages/junos_eznc-2.6.8+0.g0745dd0e.dirty-py3.9.egg/jnpr/junos/decorators.py", line 145, in wrapper
    raise ex
  File "/root/pyez_release_268_test1/venv/lib/python3.9/site-packages/junos_eznc-2.6.8+0.g0745dd0e.dirty-py3.9.egg/jnpr/junos/decorators.py", line 117, in wrapper
    rsp = function(self, *args, **kwargs)
  File "/root/pyez_release_268_test1/venv/lib/python3.9/site-packages/junos_eznc-2.6.8+0.g0745dd0e.dirty-py3.9.egg/jnpr/junos/device.py", line 1471, in _rpc_reply
    return self._conn.rpc(rpc_cmd_e, filter_xml)._NCElement__doc
  File "/root/pyez_release_268_test1/venv/lib/python3.9/site-packages/ncclient-0.6.13-py3.9.egg/ncclient/manager.py", line 246, in execute
  File "/root/pyez_release_268_test1/venv/lib/python3.9/site-packages/ncclient-0.6.13-py3.9.egg/ncclient/operations/third_party/juniper/rpc.py", line 52, in request
  File "/root/pyez_release_268_test1/venv/lib/python3.9/site-packages/ncclient-0.6.13-py3.9.egg/ncclient/operations/rpc.py", line 373, in _request
ncclient.operations.rpc.RPCError: warning: statement not found
warning: statement not found
warning: statement not found
warning: statement not found
warning: statement not found
warning: statement not found
warning: statement not found
warning: statement not found
warning: statement not found
warning: statement not found
warning: statement not found
warning: statement not found
warning: statement not found
error: syntax error
warning: statement not found
error: syntax error
warning: statement not found
warning: statement not found
warning: statement not found
warning: statement not found

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/pyez_release_268_test1/venv/lib/python3.9/site-packages/junos_eznc-2.6.8+0.g0745dd0e.dirty-py3.9.egg/jnpr/junos/utils/config.py", line 500, in try_load
    got = self.rpc.load_config(
  File "/root/pyez_release_268_test1/venv/lib/python3.9/site-packages/junos_eznc-2.6.8+0.g0745dd0e.dirty-py3.9.egg/jnpr/junos/rpcmeta.py", line 287, in load_config
    return self._junos.execute(rpc, ignore_warning=ignore_warning)
  File "/root/pyez_release_268_test1/venv/lib/python3.9/site-packages/junos_eznc-2.6.8+0.g0745dd0e.dirty-py3.9.egg/jnpr/junos/decorators.py", line 76, in wrapper
    return function(*args, **kwargs)
  File "/root/pyez_release_268_test1/venv/lib/python3.9/site-packages/junos_eznc-2.6.8+0.g0745dd0e.dirty-py3.9.egg/jnpr/junos/decorators.py", line 31, in wrapper
    return function(*args, **kwargs)
  File "/root/pyez_release_268_test1/venv/lib/python3.9/site-packages/junos_eznc-2.6.8+0.g0745dd0e.dirty-py3.9.egg/jnpr/junos/device.py", line 854, in execute
    raise EzErrors.RpcError(cmd=rpc_cmd_e, rsp=rsp, errs=ex)
jnpr.junos.exception.RpcError: RpcError(severity: error, bad_element: 65000, message: warning: statement not found
warning: statement not found
warning: statement not found
warning: statement not found
warning: statement not found
warning: statement not found
warning: statement not found
warning: statement not found
warning: statement not found
warning: statement not found
warning: statement not found
warning: statement not found
warning: statement not found
error: syntax error
warning: statement not found
error: syntax error
warning: statement not found
warning: statement not found
warning: statement not found
warning: statement not found)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/pyez_release_268_test1/py-junos-eznc/tests/functional/issue_1245.py", line 9, in <module>
    cu.load(path='mx-config.conf', ignore_warning=True, format='set')
  File "/root/pyez_release_268_test1/venv/lib/python3.9/site-packages/junos_eznc-2.6.8+0.g0745dd0e.dirty-py3.9.egg/jnpr/junos/utils/config.py", line 589, in load
    return try_load(rpc_contents, rpc_xattrs, ignore_warning=ignore_warning)
  File "/root/pyez_release_268_test1/venv/lib/python3.9/site-packages/junos_eznc-2.6.8+0.g0745dd0e.dirty-py3.9.egg/jnpr/junos/utils/config.py", line 506, in try_load
    raise ConfigLoadError(cmd=err.cmd, rsp=err.rsp, errs=err.errs)
jnpr.junos.exception.ConfigLoadError: ConfigLoadError(severity: error, bad_element: 65000, message: warning: statement not found
warning: statement not found
warning: statement not found
warning: statement not found
warning: statement not found
warning: statement not found
warning: statement not found
warning: statement not found
warning: statement not found
warning: statement not found
warning: statement not found
warning: statement not found
warning: statement not found
error: syntax error
warning: statement not found
error: syntax error
warning: statement not found
warning: statement not found
warning: statement not found
warning: statement not found)

~/pyez_release_268_test1/py-junos-eznc/tests/functional# cat issue_1245.py 
from jnpr.junos import Device
from jnpr.junos.utils.config import Config
load_warnings = ['statement not found']

dev = Device(host='xx.xx.xx.xx', user='xyz', password='xyz')
dev.open()

with Config(dev, mode='exclusive') as cu:  
    cu.load(path='mx-config.conf', ignore_warning=True, format='set')

dev.close()

Thanks

chidanandpujar commented 11 months ago

Hi @kpetremann There were some rpc-erros with severity as error reported during loading of the config statements , I have removed them since ,we don't have option to ignore severity errors .

with following config statements , I am able to ignore the warnings .

~/pyez_release_268_test1/py-junos-eznc/tests/functional# cat mx-config.set
delete policy-options community CL-LOCAL
delete policy-options community CL-MAIN
set policy-options community CL-MAIN members 649..:20000
delete policy-options community CL-SERVICE
delete policy-options community CL-DEFAULT
delete policy-options community CL-LOCATION
delete policy-options community CL-CLOS_INFRA
delete policy-options community CL-SERVER
delete policy-options route-filter-list PF-LOOPBACK_IPV4
set policy-options route-filter-list PF-LOOPBACK_IPV4 10.0.0.0/22 exact
set policy-options route-filter-list PF-LOOPBACK_IPV4 10.0.1.0/22 exact
delete policy-options route-filter-list PF-LOOPBACK_IPV6
set policy-options route-filter-list PF-LOOPBACK_IPV6 2001:db8:1::/64 exact
set policy-options route-filter-list PF-LOOPBACK_IPV6 2001:db8:1::/128 exact
delete policy-options policy-statement RM-TEST
delete policy-options policy-statement AUTOGENERATED::RM-TEST::IPV4_UNICAST
delete policy-options policy-statement AUTOGENERATED::RM-TEST::IPV6_UNICAST
delete policy-options policy-statement AUTOGENERATED::RM-TEST::L2VPN_EVPN
set policy-options policy-statement RM-TEST term 10 from route-filter-list PF-LOOPBACK_IPV4
set policy-options policy-statement RM-TEST term 10 from protocol direct
set policy-options policy-statement RM-TEST term 10 from local-preference 1234
set policy-options policy-statement RM-TEST term 10 from community CL-LOCAL
set policy-options policy-statement RM-TEST term 10 then origin egp
set policy-options policy-statement RM-TEST term 10 then local-preference 5678
set policy-options policy-statement RM-TEST term 10 then metric 250
delete policy-options community AUTOGENERATED::RM-TEST:10
set policy-options policy-statement RM-TEST term 10 then community set AUTOGENERATED::RM-TEST:10
set policy-options policy-statement RM-TEST term 10 then reject
set policy-options policy-statement RM-TEST then reject
set policy-options policy-statement AUTOGENERATED::RM-TEST::IPV4_UNICAST term 10 from family inet
set policy-options policy-statement AUTOGENERATED::RM-TEST::IPV4_UNICAST term 10 from route-filter-list PF-LOOPBACK_IPV4
set policy-options policy-statement AUTOGENERATED::RM-TEST::IPV4_UNICAST term 10 from protocol direct
set policy-options policy-statement AUTOGENERATED::RM-TEST::IPV4_UNICAST term 10 from local-preference 1234
set policy-options policy-statement AUTOGENERATED::RM-TEST::IPV4_UNICAST term 10 from community CL-LOCAL
set policy-options policy-statement AUTOGENERATED::RM-TEST::IPV4_UNICAST term 10 then origin egp
set policy-options policy-statement AUTOGENERATED::RM-TEST::IPV4_UNICAST term 10 then local-preference 5678
set policy-options policy-statement AUTOGENERATED::RM-TEST::IPV4_UNICAST term 10 then metric 250
delete policy-options community AUTOGENERATED::RM-TEST:10
set policy-options policy-statement AUTOGENERATED::RM-TEST::IPV4_UNICAST term 10 then community set AUTOGENERATED::RM-TEST:10
set policy-options policy-statement AUTOGENERATED::RM-TEST::IPV4_UNICAST term 10 then reject
set policy-options policy-statement AUTOGENERATED::RM-TEST::IPV4_UNICAST then reject
delete policy-options policy-statement RM-TEST-OUT
delete policy-options policy-statement AUTOGENERATED::RM-TEST-OUT::IPV4_UNICAST
delete policy-options policy-statement AUTOGENERATED::RM-TEST-OUT::IPV6_UNICAST
delete policy-options policy-statement AUTOGENERATED::RM-TEST-OUT::L2VPN_EVPN
set policy-options policy-statement RM-TEST-OUT term 10 then reject
set policy-options policy-statement RM-TEST-OUT then reject
set policy-options policy-statement AUTOGENERATED::RM-TEST-OUT::IPV4_UNICAST term 10 from family inet
set policy-options policy-statement AUTOGENERATED::RM-TEST-OUT::IPV4_UNICAST term 10 then reject
set policy-options policy-statement AUTOGENERATED::RM-TEST-OUT::IPV4_UNICAST then reject

~/pyez_release_268_test1/py-junos-eznc/tests/functional# cat issue_1245.py 
from jnpr.junos import Device
from jnpr.junos.utils.config import Config
load_warnings = ['statement not found']

dev = Device(host='xx.xx.xx.xx', user='xyz', password='xyz')
dev.open()

with Config(dev, mode='exclusive') as cu:  
    cu.load(path="mx-config.set", ignore_warning=True, format='set')

dev.close()

:~/pyez_release_268_test1/py-junos-eznc/tests/functional# python issue_1245.py 
:~/pyez_release_268_test1/py-junos-eznc/tests/functional#

Thanks

kpetremann commented 11 months ago

Hi @chidanandpujar,

I had a look to your stacktrace when you trigger the exception, I do not think this is the same one. We do not have errors but only warnings.

The issues we have is that a warning triggers an unhandled exception because the warning is supposed to be an XML message with _root, but according the stacktrace we get, the message it is a string, which of course does not have _root attribute.

See https://github.com/Juniper/py-junos-eznc/compare/master...kpetremann:py-junos-eznc:root_error_workaround#diff-4bb7532ee354a6233592a5357ecb0d59daa727fccf2afc4544c5d07baf74e052R145

at line 125: it checks if it is a warning or an error. If it is an error it raises an exception at line 145 and so exits the function, which is fine. The issue is at line 156, which is not run when this is an error.

With the hacky patch, we do have an exception when there is an error and it is ok. Only a warning is causing an issue, and I do not know which one exactly. Most of the warning are well handled.

I'll try to set up a lab today to reproduce the issue, as I cannot reproduce the issue because of business freeze.

chidanandpujar commented 11 months ago

Hi @kpetremann , Sure, Thanks , Please try to replicate the issue with PyEZ script , while debugging the issue , I have found a issue where ignoring_warning was set to False by default and fixed it via https://github.com/Juniper/py-junos-eznc/issues/1245

once you are able to replicate the issue, please share config and PyEZ script .

Thanks

apurvaraghu commented 8 months ago

Hi @kpetremann, Were you able to replicate the issue via a PyEz script? Do you still see the issue? Kindly update.

kpetremann commented 8 months ago

Hi @apurvaraghu,

I've given it a try but it was not enough.

I'll also check the differences in eznc usage between salt (via napalm) and your script), and I'll try with both the script and salt at the same time.

apurvaraghu commented 8 months ago

Thanks, Please update the results.

chidanandpujar commented 6 months ago

Hi @kpetremann Please replicate this issue using PyEZ script and update the config and results . Till then I will move the issue to need-info state .

Please refer the following PyEZ script .

from jnpr.junos import Device
from jnpr.junos.utils.config import Config
load_warnings = ['statement not found']

dev = Device(host='xx.xx.xx.xx', user='xyz', password='xyz')
dev.open()

with Config(dev, mode='exclusive') as cu:  
    cu.load(path='mx-config.conf', ignore_warning=True, format='set')

dev.close()

Thanks Chidanand

kpetremann commented 1 month ago

hi @chidanandpujar,

I am still able to reproduce the issue with Salt, but sadly not when calling PyEZ directly. I did not catch the parameters Salt uses to call PyEZ and trigger the issue.

I am closing this ticket as our fork of PyEZ with the workaround is working well with Salt.

Cheers