runmanager under Python 2.7 cannot handle globals that contain unicode characters set under Python 3

[philipstarkey]

Original report (archived issue) by Philip Starkey (Bitbucket: pstarkey, GitHub: philipstarkey).

---

If you set a global (under Python 3) to 'asdf҂', then close runmanager, open runmanager under Python 2, load the globals file and then try to open the tab for the globals file, you get the below exception (and runmanager gets stuck in a broken state)

#!python

Traceback (most recent call last):
  File "C:\labscript_suite\labscript_py27\runmanager\__main__.py", line 2177, in on_treeView_groups_leftClicked
    self.open_group(globals_file, group_name)
  File "C:\labscript_suite\labscript_py27\runmanager\__main__.py", line 2780, in open_group
    group_tab = GroupTab(self.ui.tabWidget, globals_file, group_name)
  File "C:\labscript_suite\labscript_py27\runmanager\__main__.py", line 604, in __init__
    self.populate_model()
  File "C:\labscript_suite\labscript_py27\runmanager\__main__.py", line 655, in populate_model
    row = self.make_global_row(name, value, units, expansion)
  File "C:\labscript_suite\labscript_py27\runmanager\__main__.py", line 719, in make_global_row
    value_item.setData(str(value), self.GLOBALS_ROLE_PREVIOUS_TEXT)
UnicodeEncodeError: 'ascii' codec can't encode character u'\u0482' in position 6: ordinal not in range(128)

[philipstarkey]

Original comment by Chris Billington (Bitbucket: cbillington, GitHub: chrisjbillington).

---

str(value) -> value.decode('utf8') in __main__.py", line 719 should fix that

[philipstarkey]

Original comment by Chris Billington (Bitbucket: cbillington, GitHub: chrisjbillington).

---

Oh actually that will break on Python 3. We probably want to convert with runmanager._ensure_str() to decode the string conditional on whether it's already a unicode string or not.

[philipstarkey]

Original comment by Philip Starkey (Bitbucket: pstarkey, GitHub: philipstarkey).

---

I'm wondering if we just need if PY2: str=unicode and the top of __main__.py. PyQt should be able to handle unicode strings I think (probably should check carefully with PyQt4 and PyQt5)

[philipstarkey]

Original comment by Chris Billington (Bitbucket: cbillington, GitHub: chrisjbillington).

---

They can definitely handle unicode strings, but we want .decode('utf8'), not unicode(), as 'unicode' still assumes ascii codex for converting bytes:

>>> unicode('ü'.encode('utf8'))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 0: ordinal not in range(128)

if PY2: str=unicode is a good idea as well though to ensure both Python versions are using the same datatypes when str() is called in other contexts

[philipstarkey]

Original comment by Chris Billington (Bitbucket: cbillington, GitHub: chrisjbillington).

---

I am unable to reproduce this.

The strings are already unicode strings, as runmanager.get_globals() is already sufficiently full of calls to _ensure_str() on every string it reads from the globals file. so str() on them is a no-op. if PY2: str = unicode is already present at the top of __main__.py, but if i weren't str() would be equivalent to .encode('ascii'), causing the error you're seeing. I see the line numbers in your traceback don't match mine. That makes me think you're on a different version of the codebase that lacks the if PY2: str = unicode line and would explain why I can't reproduce.

[philipstarkey]

Original comment by Philip Starkey (Bitbucket: pstarkey, GitHub: philipstarkey).

---

• changed state from "new" to "invalid"

Can't reproduce with latest version