Use unicode in pyTSon?

[Bluscream]

Unicode really gives me an headache in python, how would one get a channel name with unicode and set that channel name for example?

        debug = PluginHost.cfg.getboolean("general", "verbose")
        (error, name) = ts3lib.getChannelVariable(schid, channelID, ts3defines.ChannelProperties.CHANNEL_NAME)
        # name = name.encode("utf-8").decode(sys.stdout.encoding)
        if debug: print("err:", error, "CHANNEL_NAME", name)
        if not error and name: ts3lib.setChannelVariableAsString(schid, 0,ts3defines.ChannelProperties.CHANNEL_NAME,name)

5/18/2018 11:37:29  pyTSon.PluginHost.onMenuItemEvent   Error   Error calling onMenuItemEvent of python plugin Fake Anything: Traceback (most recent call last):
  File "C:/Users/blusc/AppData/Roaming/TS3Client/plugins/pyTSon/scripts\pluginhost.py", line 663, in onMenuItemEvent
    plugin.onMenuItemEvent(schid, atype, locid, selectedItemID)
  File "C:/Users/blusc/AppData/Roaming/TS3Client/plugins/pyTSon/scripts\faker\__init__.py", line 32, in onMenuItemEvent
    if atype == ts3defines.PluginMenuType.PLUGIN_MENU_TYPE_CHANNEL and menuItemID == 0: self.fakeChannel(schid, selectedItemID)
  File "C:/Users/blusc/AppData/Roaming/TS3Client/plugins/pyTSon/scripts\faker\__init__.py", line 42, in fakeChannel
    if not error and name: ts3lib.setChannelVariableAsString(schid, 0,ts3defines.ChannelProperties.CHANNEL_NAME,name)
TypeError: argument 4 must be str, not bytes

5/18/2018 11:37:54  pyTSon.PluginHost.onMenuItemEvent   Error   Error calling onMenuItemEvent of python plugin Fake Anything: Traceback (most recent call last):
  File "C:/Users/blusc/AppData/Roaming/TS3Client/plugins/pyTSon/scripts\pluginhost.py", line 663, in onMenuItemEvent
    plugin.onMenuItemEvent(schid, atype, locid, selectedItemID)
  File "C:/Users/blusc/AppData/Roaming/TS3Client/plugins/pyTSon/scripts\faker\__init__.py", line 32, in onMenuItemEvent
    if atype == ts3defines.PluginMenuType.PLUGIN_MENU_TYPE_CHANNEL and menuItemID == 0: self.fakeChannel(schid, selectedItemID)
  File "C:/Users/blusc/AppData/Roaming/TS3Client/plugins/pyTSon/scripts\faker\__init__.py", line 41, in fakeChannel
    if debug: print("err:", error, "CHANNEL_NAME", name.decode("utf-8"))
  File "C:/Users/blusc/AppData/Roaming/TS3Client/plugins/pyTSon/lib\encodings\cp1252.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode characters in position 0-2: character maps to <undefined>

I heard

#!/usr/bin/env python
# -*- coding: utf-8 -*-

would help but it doesn't :/

[pathmann]

The result of the getChannelVariable/getChannelVariableAsString function should be a utf-8 encoded (coming from the client) string.

This is what I get:

>>> schid = 1

>>> _, myid = ts3lib.getClientID(schid)

>>> _, mynick = ts3lib.getClientVariable(schid, myid, ts3defines.ClientProperties.CLIENT_NICKNAME)

>>> print(mynick)

てんㆁㅆㄤຣ

>>> type(mynick)

<class 'str'>

>>> ts3lib.setChannelVariableAsString(schid, 0, ts3defines.ChannelProperties.CHANNEL_NAME, mynick + "' Channel")

0

>>> ts3lib.flushChannelCreation(schid, 0)

0

>>> _, mychan = ts3lib.getChannelOfClient(schid, myid)

>>> _, channame = ts3lib.getChannelVariableAsString(schid, mychan, ts3defines.ChannelProperties.CHANNEL_NAME)
>>> print(channame)

てんㆁㅆㄤຣ' Channel

>>> type(channame)

<class 'str'>

If your result depends on specific unicode characters, you should post an example.

[Bluscream]

If you need an example: https://github.com/Bluscream/pyTSon_plugins/blob/master/scripts/faker/__init__.py#L35

I got this with more then just that one script, for example in https://github.com/Bluscream/pyTSon_plugins/blob/master/scripts/discordify/__init__.py#L100 i use the unidecode library to get it working, but without the unicode chars which is obviously not applicable to the first script

[pathmann]

That's neither a minimal example nor an example for a unicode channel name.

I won't use any "bloated" script to test this.

[Bluscream]

[*spacer5]═●═

╠-● Administrator | Blu

[cspacer56] ★ Temporäre Channel ★

[pathmann]

Provide a reproducable minimal example!

What are you trying to do here? This line is commented in your first post. I can't follow...

[Bluscream]

I uncommented it t try it out but it didn't help. Gonna provide a minimal example plugin this evening

[pathmann]

No plugin needed, a small piece of code is just enough.

[Bluscream]

But using the console could be a potential culprit, never had problems with utf8 there. Just want to make sure it works in a plugin

[pathmann]

Apart from some output redirecting it is the same environment.

Especially for the ts3lib wrapping there is no difference at all.

[Bluscream]

A new problem with utf-8:

mychan = ╠-● Administrator | Blu

cfg = cfg.read('config.ini', encoding='utf-8')
mychan = self.cfg.get("general", "mychan").split(",")
print(mychan) # causes the error below
mycid = ts3lib.getChannelIDFromChannelNames(schid, mychan) # crashes teamspeak

6/10/2018 10:57:23  pyTSon.PluginHost.start Error   Error starting python plugin my Support: Traceback (most recent call last):
  File "C:/Users/blusc/AppData/Roaming/TS3Client/plugins/pyTSon/scripts\pluginhost.py", line 147, in startPlugin
    cls.active[key] = cls.plugins[key]()
  File "C:/Users/blusc/AppData/Roaming/TS3Client/plugins/pyTSon/scripts\mySupport\__init__.py", line 59, in __init__
    self.checkServer(ts3lib.getCurrentServerConnectionHandlerID())
  File "C:/Users/blusc/AppData/Roaming/TS3Client/plugins/pyTSon/scripts\mySupport\__init__.py", line 133, in checkServer
    print(mychan)
  File "C:/Users/blusc/AppData/Roaming/TS3Client/plugins/pyTSon/lib\encodings\cp1252.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode character '\u2560' in position 0: character maps to <undefined>

[pathmann]

The crash is fixed with ae7e152.

[pathmann]

Can't reproduce this here:

>>> import configparser

>>> import io

>>> content = io.StringIO("[general]\nmychan = ╠-● Administrator | Blu")

>>> cfg = configparser.ConfigParser()

>>> cfg.readfp(content)

>>> cfg.get("general", "mychan")

'╠-● Administrator | Blu'

>>> type(cfg.get("general", "mychan"))

<class 'str'>
>>> cfg.get("general", "mychan").split(",")

['╠-● Administrator | Blu']

I need to check, if this is a windows only bug.

Another option is, that this is related to loading files from disk, but I doubt it. But to be extra secure: which ConfigParser class (I had to guess that!) do you use? Can you check if my example (using StringIO) works for you?

[Bluscream]

I need to check, if this is a windows only bug.

idk :/ have no other OS to test against

which ConfigParser class

    # I/O #
    def loadCfg(path, cfg):
        """

        :param path:
        :param cfg:
        """
        if not os.path.isfile(path) or os.path.getsize(path) < 1:
            saveCfg(path, cfg)
        cfg = cfg.read(path, encoding='utf-8')

    def saveCfg(path, cfg):
        """

        :param path:
        :param cfg:
        """
        with open(path, 'w') as cfgfile:
            cfg.write(cfgfile)

import ts3lib, ts3defines, datetime, pytson
from configparser import ConfigParser
from ts3plugin import ts3plugin, PluginHost
from PythonQt.QtCore import QTimer
from PythonQt.QtGui import QMessageBox
from bluscream import timestamp, getScriptPath, loadCfg
class mySupport(ts3plugin):
    perm = (0,"","")
    cfg = ConfigParser()
    # cfg.optionxform = str
    cfg["general"] = {
        "servername": "Mein Server",
        "myuid": "e3dvocUFTE1UWIvtW8qzulnWErI=",
        "mychan": "Vater Channel,Mein Channel"
    }
    def __init__(self):
        loadCfg(self.path+"/config.ini", self.cfg)

Can you check if my example (using StringIO) works for you?

6/12/2018 01:03:41  pyTSon.PluginHost.reload    Error   Error loading python plugin from C:/Users/blusc/AppData/Roaming/TS3Client/plugins/pyTSon\scripts\mySupport\: Traceback (most recent call last):
  File "C:/Users/blusc/AppData/Roaming/TS3Client/plugins/pyTSon/scripts\pluginhost.py", line 320, in reload
    cls.modules[base] = importlib.reload(cls.modules[base])
  File "C:/Users/blusc/AppData/Roaming/TS3Client/plugins/pyTSon/lib\importlib\__init__.py", line 166, in reload
    _bootstrap._exec(spec, module)
  File "<frozen importlib._bootstrap>", line 626, in _exec
  File "<frozen importlib._bootstrap_external>", line 661, in exec_module
  File "<frozen importlib._bootstrap_external>", line 767, in get_code
  File "<frozen importlib._bootstrap_external>", line 727, in source_to_code
  File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
  File "C:/Users/blusc/AppData/Roaming/TS3Client/plugins/pyTSon/scripts\mySupport\__init__.py", line 128
    content = StringIO("[general]\nmychan = ╠-● Administrator | Blu")
                                                                     ^
SyntaxError: invalid character in identifier

[pathmann]

That's what "# -- coding: utf-8 --" is for.

Nvm, I'll have a look when I have the time to do.

[pathmann]

Is the ini file written by any script? If not, try setting the encoding of the file with your favorite texteditor to utf8. This fixed it for me (on Windows).

[Bluscream]

The inifile is written by the saveCfg function from within the loadCfg function

[pathmann]

You mean here:

with open(path, 'w') as cfgfile:
            cfg.write(cfgfile)

Try adding the encoding keyword parameter.

[Bluscream]

add it to open() or .write() or both?

EDIT:

    with open(path, 'w') as cfgfile:
        cfg.write(cfgfile, encoding="utf-8")

shows no errors in PyCharm

EDIT 2:

7/19/2018 23:45:29  pyTSon  Error   Traceback (most recent call last):
  File "C:/Users/blusc/AppData/Roaming/TS3Client/plugins/pyTSon/scripts\customBadges\__init__.py", line 53, in __init__
    loadCfg(self.ini, self.cfg)
  File "C:/Users/blusc/AppData/Roaming/TS3Client/plugins/pyTSon/include\bluscream.py", line 306, in loadCfg
    saveCfg(path, cfg)
  File "C:/Users/blusc/AppData/Roaming/TS3Client/plugins/pyTSon/include\bluscream.py", line 316, in saveCfg
    cfg.write(cfgfile, encoding="utf-8")
TypeError: write() got an unexpected keyword argument 'encoding'

[Bluscream]

Is it possible that

7/19/2018 23:52:41  pyTSon.PluginHost.onMenuItemEvent   Error   Error calling onMenuItemEvent of python plugin Fake Anything: Traceback (most recent call last):
  File "C:/Users/blusc/AppData/Roaming/TS3Client/plugins/pyTSon/scripts\pluginhost.py", line 663, in onMenuItemEvent
    plugin.onMenuItemEvent(schid, atype, locid, selectedItemID)
  File "C:/Users/blusc/AppData/Roaming/TS3Client/plugins/pyTSon/scripts\faker\__init__.py", line 32, in onMenuItemEvent
    if atype == ts3defines.PluginMenuType.PLUGIN_MENU_TYPE_CHANNEL and menuItemID == 0: self.fakeChannel(schid, selectedItemID)
  File "C:/Users/blusc/AppData/Roaming/TS3Client/plugins/pyTSon/scripts\faker\__init__.py", line 41, in fakeChannel
    if debug: print("err:", error, "CHANNEL_NAME", name)
  File "C:/Users/blusc/AppData/Roaming/TS3Client/plugins/pyTSon/lib\encodings\cp1252.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode characters in position 10-11: character maps to <undefined>

appears because the codepage of the windows cmd is

C:\Users\blusc>chcp
Active code page: 850

?

When trying the stuff from https://stackoverflow.com/a/4027726/5494061 i get:

cp1252
True
cp1252
mbcs
7/19/2018 23:57:10  pyTSon.PluginHost.onMenuItemEvent   Error   Error calling onMenuItemEvent of python plugin Fake Anything: Traceback (most recent call last):
  File "C:/Users/blusc/AppData/Roaming/TS3Client/plugins/pyTSon/scripts\pluginhost.py", line 663, in onMenuItemEvent
    plugin.onMenuItemEvent(schid, atype, locid, selectedItemID)
  File "C:/Users/blusc/AppData/Roaming/TS3Client/plugins/pyTSon/scripts\faker\__init__.py", line 32, in onMenuItemEvent
    if atype == ts3defines.PluginMenuType.PLUGIN_MENU_TYPE_CHANNEL and menuItemID == 0: self.fakeChannel(schid, selectedItemID)
  File "C:/Users/blusc/AppData/Roaming/TS3Client/plugins/pyTSon/scripts\faker\__init__.py", line 41, in fakeChannel
    print(os.environ["PYTHONIOENCODING"])
  File "C:/Users/blusc/AppData/Roaming/TS3Client/plugins/pyTSon/lib\os.py", line 725, in __getitem__
    raise KeyError(key) from None
KeyError: 'PYTHONIOENCODING'

after setting the environment var i get

cp1252
True
cp1252
mbcs
utf_8
7/20/2018 00:00:10  pyTSon.PluginHost.onMenuItemEvent   Error   Error calling onMenuItemEvent of python plugin Fake Anything: Traceback (most recent call last):
  File "C:/Users/blusc/AppData/Roaming/TS3Client/plugins/pyTSon/scripts\pluginhost.py", line 663, in onMenuItemEvent
    plugin.onMenuItemEvent(schid, atype, locid, selectedItemID)
  File "C:/Users/blusc/AppData/Roaming/TS3Client/plugins/pyTSon/scripts\faker\__init__.py", line 32, in onMenuItemEvent
    if atype == ts3defines.PluginMenuType.PLUGIN_MENU_TYPE_CHANNEL and menuItemID == 0: self.fakeChannel(schid, selectedItemID)
  File "C:/Users/blusc/AppData/Roaming/TS3Client/plugins/pyTSon/scripts\faker\__init__.py", line 42, in fakeChannel
    print(chr(246), chr(9786), chr(9787))
  File "C:/Users/blusc/AppData/Roaming/TS3Client/plugins/pyTSon/lib\encodings\cp1252.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode character '\u263a' in position 0: character maps to <undefined>

[Bluscream]

I wonder if something like https://github.com/ReSpeak/TS3Hook/blob/master/TS3Hook/dllmain.cpp#L205 could fix it?

[pathmann]

Only open has an encoding parameter (https://docs.python.org/3.5/library/functions.html#open), so add it for both reading and saving the config.

[Bluscream]

Already figured that one out :)

[pathmann]

print(chr(9786))

This will throw an exception, because stdout's encoding is set to cp1252 (implicit print(chr(9786).encode("cp1252")).

print(chr(9786).encode("utf8"))

This should work.

Let's test if 9aae8d4 breaks anything (in fact I did not test it on any platform). But don't get me wrong, I don't consider this as a bug, if anything else breaks, I'll revert it and developers have to take care of encodings.

[Bluscream]

With the new nightly it seems to work

utf-8
True
cp1252
mbcs
utf_8
b'\xe2\x98\xba'
├ Ôÿ║ Ôÿ╗
err: 0 CHANNEL_NAME [lspacer]ÔòöÔùÅLaberecke I
err: 0 CHANNEL_NAME_PHONETIC
err: 0 CHANNEL_FLAG_PASSWORD 0
err: 0 CHANNEL_NEEDED_TALK_POWER 0
err: 0 CHANNEL_CODEC 4
err: 0 CHANNEL_CODEC_QUALITY 6
err: 0 CHANNEL_CODEC_LATENCY_FACTOR 1
err: 0 CHANNEL_CODEC_IS_UNENCRYPTED 0
err: 0 CHANNEL_MAXCLIENTS -1
err: 0 CHANNEL_MAXFAMILYCLIENTS -1
err: 0 CHANNEL_ICON_ID 0

even tho that's just glibberish atleast it doesnt error me out

EDIT: After using https://github.com/Bluscream/ts3-utf8-console the glibberish is gone now!

utf-8
True
cp1252
mbcs
utf_8
b'\xe2\x98\xba'
ö ☺ ☻
err: 0 CHANNEL_NAME [lspacer]╠●Laberecke III
err: 0 CHANNEL_NAME_PHONETIC
err: 0 CHANNEL_FLAG_PASSWORD 0
err: 0 CHANNEL_NEEDED_TALK_POWER 0
err: 0 CHANNEL_CODEC 4
err: 0 CHANNEL_CODEC_QUALITY 6
err: 0 CHANNEL_CODEC_LATENCY_FACTOR 1
err: 0 CHANNEL_CODEC_IS_UNENCRYPTED 0
err: 0 CHANNEL_MAXCLIENTS -1
err: 0 CHANNEL_MAXFAMILYCLIENTS -1
err: 0 CHANNEL_ICON_ID 0

[pathmann]

And that's the reason why I don't like it. These changes affect the complete process (the client and every other plugin).

[Bluscream]

btw, StringIO stuff still doesn't work but i'm not gonna use it anyway so :shrug:

  File "C:/Users/blusc/AppData/Roaming/TS3Client/plugins/pyTSon/scripts\pluginhost.py", line 322, in reload
    cls.modules[base] = importlib.__import__(base)
  File "<frozen importlib._bootstrap>", line 1055, in __import__
  File "<frozen importlib._bootstrap>", line 986, in _gcd_import
  File "<frozen importlib._bootstrap>", line 969, in _find_and_load
  File "<frozen importlib._bootstrap>", line 958, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 673, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 661, in exec_module
  File "<frozen importlib._bootstrap_external>", line 767, in get_code
  File "<frozen importlib._bootstrap_external>", line 727, in source_to_code
  File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
  File "C:/Users/blusc/AppData/Roaming/TS3Client/plugins/pyTSon/scripts\mySupport\__init__.py", line 128
    content = StringIO("[general]\nmychan = ╠-● Administrator | Blu")
                                                                     ^
SyntaxError: invalid character in identifier

[pathmann]

I'm pretty sure, there is a non-visible character at the end of the line. It's a syntax error, no encoding error! It's about an invalid character in some identifier, not in some string!

Consider (and try with your python interpreter):

#!/usr/bin/python
# -*- coding: latin-1 -*-

asd = "ä😀"
print(asd.encode("latin-1").decode("latin-1"))

#asd2😀 = "b"

Now uncomment the last line and try again. Quicktipp: Put this around your line of code: print(repr('content = StringIO...'))

P.S.: http://www.giyf.com/