Chinese symbols in DHCP lease

[lst123]

Hi. Thank you for your library, it's really helpful. Recently I stuck with a problem while I was picking up DHCP static bindings from Mikrotiks: <--- '/ip/dhcp-server/lease/print' <--- EOS ---> '!re' ---> '=.id=4' ---> '=address=10.6.68.250' ---> '=mac-address=3C:D9:2B:A0:E3:B0' ---> '=address-lists=' ---> '=server=dhcp1' ---> '=dhcp-option=' ---> '=status=bound' ---> '=expires-after=23h28m14s' ---> '=last-seen=31m46s' ---> '=active-address=10.6.68.250' ---> '=active-mac-address=3C:D9:2B:A0:E3:B0' ---> '=active-server=dhcp1' ---> '=host-name=NPIA0E3B0' ---> '=radius=false' ---> '=dynamic=false' ---> '=blocked=false' ---> '=disabled=false' ---> EOS ---> '!re' ---> '=.id=7DC' ---> '=address=10.6.68.234' ---> '=mac-address=70:54:D2:E3:B3:A7' ---> '=client-id=1:70:54:d2:e3:b3:a7' ---> '=address-lists=' ---> '=server=dhcp1' ---> '=dhcp-option=' ---> '=status=bound' ---> '=expires-after=23h9m9s' ---> '=last-seen=5m36s' ---> '=active-address=10.6.68.234' ---> '=active-mac-address=70:54:D2:E3:B3:A7' ---> '=active-client-id=1:70:54:d2:e3:b3:a7' ---> '=active-server=dhcp1' ---> '=host-name=wbf-342' ---> '=radius=false' ---> '=dynamic=true' ---> '=blocked=false' ---> '=disabled=false' ---> EOS Traceback (most recent call last): File "./apicli.py", line 75, in main() File "./apicli.py", line 65, in main selectloop(api) File "./apicli.py", line 42, in selectloop proto.readSentence() File "/usr/local/lib/python3.5/dist-packages/librouteros/connections.py", line 147, in readSentence sentence = tuple(word for word in iter(self.readWord, b'\x00')) File "/usr/local/lib/python3.5/dist-packages/librouteros/connections.py", line 147, in sentence = tuple(word for word in iter(self.readWord, b'\x00')) File "/usr/local/lib/python3.5/dist-packages/librouteros/connections.py", line 164, in readWord return self.transport.read(length).decode(encoding=self.encoding, errors='strict') UnicodeDecodeError: 'ascii' codec can't decode byte 0x8f in position 14: ordinal not in range(128)

The bad hostname in DHCP leases was (but i have many of them): sk-\8F\8A

b'\x8f\x8a'.decode('utf-16') '誏' (that means, play upon words)

I tried to change encoding from ASCII to utf-16 but nothing changed (script timed out). After that I made an attempt to get rid of this address: <--- '/ip/dhcp-server/lease/print' <--- '?=dynamic=false' <--- EOS ---> '!re' ---> '=.id=*4' ---> '=address=10.6.68.250' ---> '=mac-address=3C:D9:2B:A0:E3:B0' ---> '=address-lists=' ---> '=server=dhcp1' ---> '=dhcp-option=' ---> '=status=bound' ---> '=expires-after=22h33m19s' ---> '=last-seen=1h26m41s' ---> '=active-address=10.6.68.250' ---> '=active-mac-address=3C:D9:2B:A0:E3:B0' ---> '=active-server=dhcp1' ---> '=host-name=NPIA0E3B0' ---> '=radius=false' ---> '=dynamic=false' ---> '=blocked=false' ---> '=disabled=false' ---> EOS ---> '!done' ---> EOS

from your apicli.py everything was working fine, but when I tried the same from a script: dhcp_lease = '/ip/dhcp-server/lease/print' dhcp_s = s(cmd=dhcp_lease, dynamic=False)

or dhcp_lease = '/ip/dhcp-server/lease/print' params = {'dynamic': False} dhcp_s = s(cmd=dhcp_lease, **params)

I had that: Traceback (most recent call last): File "./net_crawler.py", line 183, in get_config dhcp_s = s(cmd=dhcp_lease, dynamic=False) File "/usr/local/lib/python3.5/dist-packages/librouteros/api.py", line 80, in call return self._readResponse() File "/usr/local/lib/python3.5/dist-packages/librouteros/api.py", line 106, in _readResponse self._trapCheck(response) File "/usr/local/lib/python3.5/dist-packages/librouteros/api.py", line 124, in _trapCheck raise TrapError(message=trap['message'], category=trap.get('category')) librouteros.exceptions.TrapError: unknown parameter

I have two questions hopefully you will be able to help me out with.... Is there any way to use unicode-16, or how to avoid this problem with unicode-16 symbols? How can I pass the params like that {'dynamic': False} in any print?

[luqasz]

How can I pass the params like that {'dynamic': False} in any print?

You have to use api query. I plan to add support for it in #11 but it is more tricky then I thought.

from your apicli.py everything was working fine

What you did exactly so that decoding works in apicli.py ?

[lst123]

Sorry for misleading you. I've just put these symbols into REPL and do .decode(decode('utf-16'))

[luqasz]

Have you tried changing encoding ?

api = connect(username='admin', password='abc', host='some.address.com', encoding='UTF-16')

[lst123]

Yes I've tried. With these code: routeros = connect(username=default_user, \ password=default_secret, host=lan_ip, timeout=90, encoding='UTF-16')

I had: exc.ConnectionError: Socket timed out. timed out When I change to UTF-8, my connection is OK.

[luqasz]

In [12]: '/login'.encode('UTF-8')
Out[12]: b'/login'

In [13]: '/login'.encode('UTF-16')
Out[13]: b'\xff\xfe/\x00l\x00o\x00g\x00i\x00n\x00'

This is why you can not use utf-16 and you get a timeout. There is nothing I can do since API does not recognize utf-16.

Scratch that one. I think I can change the way encoding / decoding works. Later I will write some more info.

[luqasz]

I will try with encoding just a value of a word e.g. =host-name=ENCODED_VALUE

[luqasz]

@lst123 I've pushed value-encoding branch that works as described above. Pass utf-16 encoding and check if you see Chinese symbols. Let me know.

[lst123]

@luqasz I've tried your changes and got this error: (virtualenv) pc@npc-PC:/git/value_encoding/virtualenv$ ./utf16.py 'ascii' codec can't encode characters in position 0-15: ordinal not in range(128) (virtualenv) pc@npc-PC:/git/value_encoding/virtualenv$

My connection string was: s = connect(host=lan_ip, username=default_user, password=default_pw, timeout=120, encoding='UTF-16') I executed these code: dhcp_l = '/ip/dhcp-server/lease/print' d_l = s(cmd=dhcp_l) print(d_l)

[luqasz]

Try apicli.py. It will print out output and crash when can not decode. Please paste output here.

[lst123]

@luqasz hello. Please see the output: (virtualenv) pc@npc-PC:/git/value_encoding$ ./apicli.py 10.6.68.1 -u testuser Password: <--- '/login' <--- '=name=dummy_user' <--- '=password=dummy_password' <--- EOS ---> '!done' ---> '=ret=㐲户㕦㈴㐵攳㌹挷つ㙤㐷ㄲ昱㔵㠵㔶' ---> EOS Traceback (most recent call last): File "./apicli.py", line 75, in main() File "./apicli.py", line 58, in main api = connect(args.host, args.user, pw, logger=mainlog, encoding='UTF-16') File "/home/pc/git/value_encoding/virtualenv/lib/python3.5/site-packages/librouteros/init.py", line 58, in connect api('/login', **{'name': username, 'response': encode_password(token, password)}) File "/home/pc/git/value_encoding/virtualenv/lib/python3.5/site-packages/librouteros/init.py", line 75, in encode_password token = token.encode('ascii', 'strict') UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-15: ordinal not in range(128) (virtualenv) pc@npc-PC:~/git/value_encoding$

[luqasz]

You can not connect because you have Chinese symbols in password. Am I right ? Please try ASCII only characters in password and then login and print e.g. DHCP leases which have as far as I remember Chinese symbols.

[lst123]

Hello. If I open a connection with apicli.py:

api = connect(args.host, args.user, pw, logger=mainlog, encoding='UTF-16')

with a user: testuser and password: testpassword there will be an error: (virtualenv) pc@npc-PC:/git/value_encoding/virtualenv$ ./apicli.py 10.6.68.1 -u testuser Password: <--- '/login' <--- '=password=dummy_password' <--- '=name=dummy_user' <--- EOS ---> '!done' ---> '=ret=〷㈰㝣〱搱㈶㉦㘸㈲㘳戵ㄱ㜴ㄷ戱㍤' ---> EOS Traceback (most recent call last): File "./apicli.py", line 75, in main() File "./apicli.py", line 58, in main api = connect(args.host, args.user, pw, logger=mainlog, encoding='UTF-16') File "/home/pc/git/value_encoding/virtualenv/lib/python3.5/site-packages/librouteros/init.py", line 58, in connect api('/login', **{'name': username, 'response': encode_password(token, password)}) File "/home/pc/git/value_encoding/virtualenv/lib/python3.5/site-packages/librouteros/init.py", line 75, in encode_password token = token.encode('ascii', 'strict') UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-15: ordinal not in range(128) (virtualenv) pc@npc-PC:~/git/value_encoding/virtualenv$

If I remove an encoding statement, the script will suffer from chinese symbols: Traceback (most recent call last): File "./apicli.py", line 75, in main() File "./apicli.py", line 65, in main selectloop(api) File "./apicli.py", line 42, in selectloop proto.readSentence() File "/home/pc/git/value_encoding/virtualenv/lib/python3.5/site-packages/librouteros/connections.py", line 147, in readSentence sentence = tuple(word for word in iter(self.readWord, b'\x00')) File "/home/pc/git/value_encoding/virtualenv/lib/python3.5/site-packages/librouteros/connections.py", line 147, in sentence = tuple(word for word in iter(self.readWord, b'\x00')) File "/home/pc/git/value_encoding/virtualenv/lib/python3.5/site-packages/librouteros/connections.py", line 168, in readWord value = value.decode(encoding=self.encoding, errors='strict') UnicodeDecodeError: 'ascii' codec can't decode byte 0x8f in position 3: ordinal not in range(128)

[luqasz]

<--- '/login'
<--- '=password=dummy_password'
<--- '=name=dummy_user'
<--- EOS
---> '!done'
---> '=ret=〷㈰㝣〱搱㈶㉦㘸㈲㘳戵ㄱ㜴ㄷ戱㍤'
---> EOS

This tells me that you still have Chinese symbols in password. Even though you've stated that you have testpassword set.

If I remove an encoding statement, the script will suffer from chinese symbols:

What encoding statement ?

Be specific and exact. Use markdown for code samples.

[lst123]

Hello.

I don't understand why you think that I've used chinese symbols in the password. In order to prove you my statements I've changed your apicli.py a little bit:

(virtualenv) pc@npc-PC:~/git/value_encoding/virtualenv$ diff apicli.py new_apicli.py 
22a23,24
> argParser.add_argument(
>         '-pw', '--password', type=str, help="password")
56c58
<     pw = getpass.getpass()
---
>     #pw = getpass.getpass()
58c60
<         api = connect(args.host, args.user, pw, logger=mainlog)
---
>         api = connect(args.host, args.user, args.password, logger=mainlog, encoding='UTF-16')
(virtualenv) pc@npc-PC:~/git/value_encoding/virtualenv$ 

When I try to open a connection with this credentials:

(virtualenv) pc@npc-PC:~/git/value_encoding/virtualenv$ ./new_apicli.py 10.6.68.1 -u testuser --password testuser
<--- '/login'
<--- '=password=dummy_password'
<--- '=name=dummy_user'
<--- EOS
---> '!done'
---> '=ret=摣㙥㔳㌲㥦攲㜶㉥㘹昴捦昲㙣㝣晤换'
---> EOS
Traceback (most recent call last):
  File "./new_apicli.py", line 77, in <module>
    main()
  File "./new_apicli.py", line 60, in main
    api = connect(args.host, args.user, args.password, logger=mainlog, encoding='UTF-16')
  File "/home/pc/git/value_encoding/virtualenv/lib/python3.5/site-packages/librouteros/__init__.py", line 58, in connect
    api('/login', **{'name': username, 'response': encode_password(token, password)})
  File "/home/pc/git/value_encoding/virtualenv/lib/python3.5/site-packages/librouteros/__init__.py", line 75, in encode_password
    token = token.encode('ascii', 'strict')
UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-15: ordinal not in range(128)

I get this error. Even if I put some credentials that don't exist on my Mikrotik router, I'll get the same error (but "=ret" line, differs a little bit):

(virtualenv) pc@npc-PC:~/git/value_encoding/virtualenv$ ./new_apicli.py 10.6.68.1 -u mrpickles --password mrpickles
<--- '/login'
<--- '=password=dummy_password'
<--- '=name=dummy_user'
<--- EOS
---> '!done'
---> '=ret=挳搸っ㉥愴捤㔵挷换ㅥ㐱㐲昶㡢〸㍢'
---> EOS
Traceback (most recent call last):
  File "./new_apicli.py", line 77, in <module>
    main()
  File "./new_apicli.py", line 60, in main
    api = connect(args.host, args.user, args.password, logger=mainlog, encoding='UTF-16')
  File "/home/pc/git/value_encoding/virtualenv/lib/python3.5/site-packages/librouteros/__init__.py", line 58, in connect
    api('/login', **{'name': username, 'response': encode_password(token, password)})
  File "/home/pc/git/value_encoding/virtualenv/lib/python3.5/site-packages/librouteros/__init__.py", line 75, in encode_password
    token = token.encode('ascii', 'strict')
UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-15: ordinal not in range(128)

After that I've changed encoding to UTF-8:

(virtualenv) pc@npc-PC:~/git/value_encoding/virtualenv$ diff apicli.py new_apicli.py 
22a23,24
> argParser.add_argument(
>         '-pw', '--password', type=str, help="password")
56c58
<     pw = getpass.getpass()
---
>     #pw = getpass.getpass()
58c60
<         api = connect(args.host, args.user, pw, logger=mainlog)
---
>         api = connect(args.host, args.user, args.password, logger=mainlog, encoding='UTF-8')
(virtualenv) pc@npc-PC:~/git/value_encoding/virtualenv$ 

Then I test it:

(virtualenv) pc@npc-PC:~/git/value_encoding/virtualenv$ ./new_apicli.py 10.6.68.1 -u testuser --password testuser
<--- '/login'
<--- '=name=dummy_user'
<--- '=password=dummy_password'
<--- EOS
---> '!done'
---> '=ret=d7ba7817dcb7ed0870f8fdaf4b3e8040'
---> EOS
<--- '/login'
<--- '=name=testuser'
<--- '=response=006ef45bda167b800723f613966f93dfa9'
<--- EOS
---> '!done'
---> EOS
/ip/dhcp-server/lease/print

<--- '/ip/dhcp-server/lease/print'
<--- EOS
---> '!re'
---> '=.id=*4'
---> '=address=10.6.68.250'
---> '=mac-address=4C:F9:2C:A0:E3:B0'
---> '=address-lists='
---> '=server=dhcp1'
---> '=dhcp-option='
---> '=status=bound'
---> '=expires-after=17h18m32s'
---> '=last-seen=6h41m28s'
---> '=active-address=10.6.68.250'
---> '=active-mac-address=4C:F9:2C:A0:E3:B0'
---> '=active-server=dhcp1'
---> '=host-name=NPIA0E3B0'
---> '=radius=false'
---> '=dynamic=false'
---> '=blocked=false'
---> '=disabled=false'
---> EOS
---> '!re'
---> '=.id=*7DC'
---> '=address=10.6.68.234'
---> '=mac-address=7D:34:D3:E3:B3:A7'
---> '=client-id=1:7D:34:d3:e3:b3:a7'
---> '=address-lists='
---> '=server=dhcp1'
---> '=dhcp-option='
---> '=status=bound'
---> '=expires-after=14h27m43s'
---> '=last-seen=2m2s'
---> '=active-address=10.6.68.234'
---> '=active-mac-address=7D:34:D3:E3:B3:A7'
---> '=active-client-id=1:7D:34:d3:e3:b3:a7'
---> '=active-server=dhcp1'
---> '=host-name=wbf-342'
---> '=radius=false'
---> '=dynamic=true'
---> '=blocked=false'
---> '=disabled=false'
---> EOS
Traceback (most recent call last):
  File "./new_apicli.py", line 77, in <module>
    main()
  File "./new_apicli.py", line 67, in main
    selectloop(api)
  File "./new_apicli.py", line 44, in selectloop
    proto.readSentence()
  File "/home/pc/git/value_encoding/virtualenv/lib/python3.5/site-packages/librouteros/connections.py", line 147, in readSentence
    sentence = tuple(word for word in iter(self.readWord, b'\x00'))
  File "/home/pc/git/value_encoding/virtualenv/lib/python3.5/site-packages/librouteros/connections.py", line 147, in <genexpr>
    sentence = tuple(word for word in iter(self.readWord, b'\x00'))
  File "/home/pc/git/value_encoding/virtualenv/lib/python3.5/site-packages/librouteros/connections.py", line 168, in readWord
    value = value.decode(encoding=self.encoding, errors='strict')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8f in position 3: invalid start byte
(virtualenv) pc@npc-PC:~/git/value_encoding/virtualenv$ 

I can send a TCPdump file to you, if you don't believe me after all.

[luqasz]

Ok. Value of =ret= is encoded by api with UTF-16 which results in some symbols:

In [4]: b'd7ba7817dcb7ed0870f8fdaf4b3e8040'.decode('UTF-16')
Out[4]: '㝤慢㠷㜱捤㝢摥㠰〷㡦摦晡戴攳〸〴'

As for other values and utf16, it is impossible to implement thath. Observe:

In [6]: '7D:34:D3:E3:B3:A7'.encode('UTF16')
Out[6]: b'\xff\xfe7\x00D\x00:\x003\x004\x00:\x00D\x003\x00:\x00E\x003\x00:\x00B\x003\x00:\x00A\x007\x00'

In [7]: '7D:34:D3:E3:B3:A7'.encode('UTF-16')
Out[7]: b'\xff\xfe7\x00D\x00:\x003\x004\x00:\x00D\x003\x00:\x00E\x003\x00:\x00B\x003\x00:\x00A\x007\x00'

In [8]: '7D:34:D3:E3:B3:A7'.encode('UTF-8')
Out[8]: b'7D:34:D3:E3:B3:A7'

In [9]: b'7D:34:D3:E3:B3:A7'.decode('utf16')
---------------------------------------------------------------------------
UnicodeDecodeError                        Traceback (most recent call last)
<ipython-input-9-560726224de8> in <module>()
----> 1 b'7D:34:D3:E3:B3:A7'.decode('utf16')

UnicodeDecodeError: 'utf-16-le' codec can't decode byte 0x37 in position 16: truncated data

Can you read correct Chinese symbols in winbox, ssh terminal ? MikroTik does not support any kind of encoding. In winbox it uses encoding which is set in your windows machine. MikroTik just stores bytes.

[lst123]

Thank you for your reply.

Can you read correct Chinese symbols in winbox, ssh terminal?

No, I can't.

[luqasz]

I have no idea how to fix this issue.

[feute]

I've come across with this issue when I was trying to make a script to take RouterOS users from the Hotspot and save them to the database.

In my case, the error message was:

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf1 in position 17: invalid continuation byte

I tried to do this with Go, and it read it properly in some cases (it won't print the character on the terminal, but it did showed correctly in a file). But when trying to insert to the database, it showed the same error about the \xf1. I figured this is shown in Winbox as in ñ; I searched in Google the hex code for that character and it showed a different one, not 0xf1, but 0xc3b1. I replaced any occurrences of the invalid byte f1 to the literal letter, and it worked.

I think Winbox has a problem with encoding characters, showing these invalid bytes; it truncates (or corrupts) the characters. So, when trying to decode it, it shows this error.

I couldn't manage to fix this with librouteros since I can't touch the retrieved data, it throws the exception before doing anything with it. But it might give you an idea of why this happens and how to (possibly) fix it.

[luqasz]

Hi. Thx for information. How MikroTik stores and then returns non ASCII characters is really a mystery. I do not understand how it is done. I also haven't seen code that reliably handles those use cases.