UnicodeDecodeError when sending stdout that is not strict utf-8

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/Volumes/data/b/swarm_slave/swarming_bot.1.zip/__main__.py", line 145, in <module>
  File "/Volumes/data/b/swarm_slave/swarming_bot.1.zip/__main__.py", line 133, in main
  File "/Volumes/data/b/swarm_slave/swarming_bot.1.zip/__main__.py", line 38, in CMDlocal_test_runner
  File "/Volumes/data/b/swarm_slave/swarming_bot.1.zip/task_runner.py", line 292, in main
  File "/Volumes/data/b/swarm_slave/swarming_bot.1.zip/task_runner.py", line 121, in load_and_run
  File "/Volumes/data/b/swarm_slave/swarming_bot.1.zip/task_runner.py", line 265, in run_command
  File "/Volumes/data/b/swarm_slave/swarming_bot.1.zip/task_runner.py", line 148, in post_update
  File "/Volumes/data/b/swarm_slave/swarming_bot.1.zip/xsrf_client.py", line 58, in url_read_json
  File "/Volumes/data/b/swarm_slave/swarming_bot.1.zip/xsrf_client.py", line 90, in _url_read_json_post
  File "/Volumes/data/b/swarm_slave/swarming_bot.1.zip/utils/net.py", line 188, in url_read_json
  File "/Volumes/data/b/swarm_slave/swarming_bot.1.zip/utils/net.py", line 553, in json_request
  File "/Volumes/data/b/swarm_slave/swarming_bot.1.zip/utils/net.py", line 449, in request
  File "/Volumes/data/b/swarm_slave/swarming_bot.1.zip/utils/net.py", line 368, in encode_request_body
  File "/Volumes/data/b/swarm_slave/swarming_bot.1.zip/utils/net.py", line 67, in <lambda>
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/__init__.py", line 250, in dumps
    sort_keys=sort_keys, **kw).encode(obj)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/encoder.py", line 209, in encode
    chunks = list(chunks)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/encoder.py", line 434, in _iterencode
    for chunk in _iterencode_dict(o, _current_indent_level):
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/encoder.py", line 390, in _iterencode_dict
    yield _encoder(value)
UnicodeDecodeError: 'utf8' codec can't decode byte 0x89 in position 53595: 
invalid start byte
Sending the crash report ... done.
Report URL:
Process exited due to exception
'utf8' codec can't decode byte 0x89 in position 53595: invalid start byte

What happened is that the stdout being sent was not valid utf-8. subprocess42 
yields bytes, so it's possible for it to send a partial utf-8 encoded rune and 
the remainder of the rune to be sent on the next call.

It's particularly problematic because this data is then passed as-is in the 
json call. The simplest fix is to convert the code to send the data as base64 
or any other related escaping form. Not super efficient but will work. It's 
better than a crash and an aborted task.
Original issue reported on code.google.com by maruel@chromium.org on 29 Oct 2014 at 7:49
madecoste / swarming

UnicodeDecodeError when sending stdout that is not strict utf-8 #169
