search

ncclient / ncclient

Python library for NETCONF clients
Apache License 2.0
679 stars 398 forks source link

patch computing correct length of UTF-8 messages #256

Open ricky-rav opened 6 years ago

ricky-rav commented 6 years ago

When pushing a configuration with UTF-8 characters, like è or é, ncclient does not correctly compute the length of messages, resulting in a framing error and subsequent closed connection on the server side.

This is because transport/ssh.py supposes that each character in the message to send takes one byte, but this might not always be the case.

The following patch fixes this problem for netconf connections over ssh:

--- transport/ssh.py    2018-07-19 18:38:30.701517055 +0200
+++ transport/ssh.py    2018-07-19 18:38:54.953422052 +0200
@@ -568,7 +568,7 @@
                             if self._server_capabilities:
                                 if 'urn:ietf:params:netconf:base:1.1' in self._server_capabilities:
                                     # send using v1.1 chunked framing
-                                    data = "%s%s%s"%(start_delim(len(data)), data, END_DELIM)
+                                    data = "%s%s%s"%(start_delim(len(data.encode())), data, END_DELIM)
                                 elif 'urn:ietf:params:netconf:base:1.0' in self._server_capabilities or 'urn:ietf:params:xml:ns:netconf:base:1.0' in self._server_capabilities:
                                     # send using v1.0 EOM markers
                                     data = "%s%s"%(data, MSG_DELIM)
einarnn commented 6 years ago

There is definitely an issue here, but the possibility of collateral damage needs investigated before a fix is merged as we need to know what may happen in the stack of calls through to Paramiko when we UTF-8 encode the data.

Just now few clients use unicode, so we have some time to investigate.

paulo-g commented 5 years ago

Just a note, this seems to be a problem with Python3, but not with Python2. I do confirm that the patch above seems to fix the problem.

jaweeki commented 4 years ago

Hi, I am a "simple user" of a netconf-client which use ncclient (https://bitbucket.org/martin_volf/ncc) and I obtained an issue when launching a very simple command : a "get-config".

I obtain following error message : Operation failed: UnicodeDecodeError - 'ascii' codec can't decode byte 0xf0 in position 8956: ordinal not in range(128)

After some investigations, I understood that my issue was linked to the fact the the configuration was containing nice UTF-8 characters (like bullet ⬤ and arrow : ▶)

It seems that the very simple patch proposed by ricky-r solve the issue. Is it plan to integrate this patch ?

Many thanks, Jacques

einarnn commented 4 years ago

Jacques,

The patch below is actually not going to resolve any issue with a get-config operation. IIRC, that patch is purely for cases where a client is sending in config that contains UTF-8 characters that encode to >1 byte. The patch has not yet been applied as I wasn't 100% sure if it would cause any collateral damage for existing clients.

Your issue is that the device you are working with is actually returning bad data that does not comply with the valid character set for netconf. This has been the cases with, for example, some versions of IOS XE in the past.

There is another tool, similar to netconf-client, called ncc (pip install ncc). It specifically catches this exception and displays the context of the error. It also has an extra command line flag to configure the amount of context it displays:

--unicode-error-bytes UNICODE_ERROR_BYTES Specify number of +/- bytes to display for UnicodeDecodeError exceptions (default 30)

I suggest that you give this a quick try with the same get-config filter. ncc supports both XPath and subtree filters, but doesn't have an interactive mode like netconf-client. See https://github.com/CiscoDevNet/ncc/blob/master/README.md#ncc https://github.com/CiscoDevNet/ncc/blob/master/README.md#ncc.

Finally, can you share the device type and OS version?

Cheers,

Einar

On 22 Nov 2019, at 15:06, jaweeki notifications@github.com wrote:

Hi, I am a "simple user" of a netconf-client which use ncclient (https://bitbucket.org/martin_volf/ncc https://bitbucket.org/martin_volf/ncc) and I obtained an issue when launching a very simple command : a "get-config".

I obtain following error message : Operation failed: UnicodeDecodeError - 'ascii' codec can't decode byte 0xf0 in position 8956: ordinal not in range(128)

After some investigations, I understood that my issue was linked to the fact the the configuration was containing nice UTF-8 characters (like bullet ⬤ and arrow : ▶)

It seems that the very simple patch proposed by ricky-r solve the issue. Is it plan to integrate this patch ?

Many thanks, Jacques

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/ncclient/ncclient/issues/256?email_source=notifications&email_token=ABJIM4HKGW6ABFBP2VIKMCTQU7YQNA5CNFSM4FK6MR32YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEE54U4A#issuecomment-557566576, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABJIM4FUCX62HWQ2IOYGJA3QU7YQNANCNFSM4FK6MR3Q.

jaweeki commented 4 years ago

Hi, Thank you Einar for your answer. You are right, there is no link between the patch proposed here and the issue that I reported. Last week, I obtained several issues in different conditions and they had been mixed up together. Sorry.

After some new investigations, here is a result summary:

  1. I obtain the "UnicodeDecodeError ... " when launching a "get config" ONLY when python2 is used.
  2. I confirm paulo-g message: the issue reported by ricky-r is in NOT reproducible when using python2.
  3. I can reproduce the issue reported by ricky-r when committing a configuration with UTF-8 characters. The patch proposed solve this issue.

More details: As Debian addict, I use to NOT use pip to install python module. I use apt install python3-ModuleName instead. Note that I know some drawbacks:

Here is the configuration of a client appliance: GNU/Linux Debian Buster 10.2:

With this configuration, when launching a "--get-config" with following command : /opt/netconf-console-2.2.0/netconf-console --port 830 -u <user> -p <password> --host <host> --get-config (python 2 is used because the shebang is : "#!/usr/bin/env python") => I obtain: Operation failed: UnicodeDecodeError - 'ascii' codec can't decode byte 0xf0 in position 5774: ordinal not in range(128)

But if I launch: python3 /opt/netconf-console-2.2.0/netconf-console --port 830 -u <user> -p <password> --host <host> --get-config => I obtain a correct result (including the UTF-8 characters).

With this configuration, when launching a "commit" with a configuration containing UTF-8 characters, with following command: /opt/netconf-console-2.2.0/netconf-console --port 830 -u <user> -p <password> --host <host> --db candidate --edit-config Config_with_UTF8.xml --commit (python 2 is used) => I obtain a correct result (including the UTF-8 characters)

But if I launch: python3 /opt/netconf-console-2.2.0/netconf-console --port 830 -u <user> -p <password> --host <host> --db candidate --edit-config Config_with_UTF8.xml --commit => I obtain: Operation failed: TimeoutExpiredError - ncclient timed out while waiting for an rpc reply.

If I apply the patch proposed by ricky-r, the same request is correctly sent and I obtain a correct result.

For information, if netconf-console is installed with python3 pip, python3 version is always used. So with following configuration: GNU/Linux Debian Buster 10.2

I can reproduce the issue concerning "--commit" and the patch proposed by ricky-r solve the issue.

Finally, for information, the device that I am using is an appliance with an OS based on Debian and confd server is provided by tail-f.

Best regards, Jacques