UTF-8 encoding issues with Python 2/3 in action_set()

[dreibh]

The proxy charm code is run with Python 2, which causes UTF-8 encoding issues. Calling action_set with e.g. { 'output': u'This is a test! \u25cf äöüß æøå ÄÖÜ ÆØÅ' } leads to an exception: UnicodeEncodeError: 'ascii' codec can't encode character u'\u25cf' in position 16: ordinal not in range(128) This is particularly a problem when "output" comes from the stdout of an external program. Once it contains non-ASCII characters, setting the output in action_set fails.

To reproduce the issue: cmd = ['./action-set'] values = { 'output': u'This is a test! \u25cf äöüß æøå ÄÖÜ ÆØÅ' } for k, v in list(values.items()): cmd.append('{}={}'.format(k, v)) subprocess.check_call(cmd)

Using this "action-set" replacement script:

!/bin/bash

while [ $# -gt 0 ] ; do echo "Argument: $1" shift done

Output when run with Python 2: Traceback (most recent call last): File "./t3", line 74, in cmd.append('{}={}'.format(k, v)) UnicodeEncodeError: 'ascii' codec can't encode character u'\u25cf' in position 16: ordinal not in range(128)

Output when run with Python 3: Argument: output=This is a test! ● äöüß æøå ÄÖÜ ÆØÅ

Using '.encode("utf-8")' creates behaviour being incompatible between Python 2 and 3 (here: values = { 'output': u'This is a test! \u25cf äöüß æøå ÄÖÜ ÆØÅ'.encode('utf-8') }): Python 2: Argument: output=This is a test! ● äöüß æøå ÄÖÜ ÆØÅ Python 3: Argument: output=b'This is a test! \xe2\x97\x8f \xc3\xa4\xc3\xb6\xc3\xbc\xc3\x9f \xc3\xa6\xc3\xb8\xc3\xa5 \xc3\x84\xc3\x96\xc3\x9c \xc3\x86\xc3\x98\xc3\x85'

[AdamIsrael]

That's a little weird. The proxy charm runs python3, not python2.

That said, I'm sure we can do better validation of UTF-8 strings to make sure they work as expected.

[dreibh]

I am not absolutely sure whether Python 2 or 3 is used to run the charm code. But I was able to reproduce the issue with Python 2 (i.e. the UnicodeEncodeError exception) on my local Bionic desktop PC. I am using Xenial in a VM to run OSM.

[AdamIsrael]

The charm definitely runs as Python3. Have you experienced errors running this code when it's deployed through Juju?

If you're not familiar, you can execute the action outside of OSM via the juju run-action command. The debug-hooks command is also useful for interacting with and testing the python code.

I'm happy to write a charm to test this, but it'll be several days before I'll have time to.