SqlReporter has problems with unicode

[mrtyler]

Given a test failure like this (from a sandbox_NN builder):

yelp.tests.xxx ValidatorTest.test_data_truncation ... fail: yelp.tests.xxx ValidatorTest.test_data_truncation
Traceback (most recent call last):
  File "./yelp/tests/xxx/validators_test.py", line 236, in test_data_truncation
    address[:64].strip()
AssertionError: assertion failed: l == r
l: u'M\u014drb\u0131 \u0131p\u015fum. D\u014d\u0144\u0119\u0107 ut p\u0119d\u0119 \u0144\u014d\u0144 \u015f\u0105p\u0131\u0119\u0144 \u0107\u014d\u0144d\u0131m\u0119\u0144tum d\u0131\u0107tum. M\u014drb\u0131 '
r: u'M\u014drb\u0131 \u0131p\u015fum. D\u014d\u0144\u0119\u0107 ut p\u0119d\u0119 \u0144\u014d\u0144 \u015f\u0105p\u0131\u0119\u0144 \u0107\u014d\u0144d\u0131m\u0119\u0144tum d\u0131\u0107tum. M\u014drb\u0131'

Diff:
l: Mōrbı ıpşum. Dōńęć ut pędę ńōń şąpıęń ćōńdımęńtum dıćtum. Mōrbı 
r: Mōrbı ıpşum. Dōńęć ut pędę ńōń şąpıęń ćōńdımęńtum dıćtum. Mōrbı<>

FAIL in 0.02s

When SqlReporter goes to stick this into the test results database, it throws an exception (reported in the stdio log for sandbox_0):

ERROR    Exception while reporting results: UnicodeEncodeError('latin-1', u"AssertionError: assertion failed: l == r\nl: u'M\\u014drb\\u0131 \\u0131p\\u015fum. D\\u014d\\u0144\\u0119\\u0107 ut p\\u0119d\\u0119 \\u0144\\u014d\\u0144 \\u015f\\u0105p\\u0131\\u0119\\u0144 \\u0107\\u014d\\u0144d\\u0131m\\u0119\\u0144tum d\\u0131\\u0107tum. M\\u014drb\\u0131 '\nr: u'M\\u014drb\\u0131 \\u0131p\\u015fum. D\\u014d\\u0144\\u0119\\u0107 ut p\\u0119d\\u0119 \\u0144\\u014d\\u0144 \\u015f\\u0105p\\u0131\\u0119\\u0144 \\u0107\\u014d\\u0144d\\u0131m\\u0119\\u0144tum d\\u0131\\u0107tum. M\\u014drb\\u0131'\n\nDiff:\nl: M\u014drb\u0131 \u0131p\u015fum. D\u014d\u0144\u0119\u0107 ut p\u0119d\u0119 \u0144\u014d\u0144 \u015f\u0105p\u0131\u0119\u0144 \u0107\u014d\u0144d\u0131m\u0119\u0144tum d\u0131\u0107tum. M\u014drb\u0131 \nr: M\u014drb\u0131 \u0131p\u015fum. D\u014d\u0144\u0119\u0107 ut p\u0119d\u0119 \u0144\u014d\u0144 \u015f\u0105p\u0131\u0119\u0144 \u0107\u014d\u0144d\u0131m\u0119\u0144tum d\u0131\u0107tum. M\u014drb\u0131<>", 473, 474, 'ordinal not in range(256)')

The result is that this test failure is not displayed in bbmanager (which relies on the work performed by SqlReporter), which is quite bad.

@bukzor thinks this is due to an errant str() in SqlReporter that fails on Unicode input.

Internal links:

• bb where this happened: http://buildbot.yelpcorp.com/rev/c8f04c774892e69c4f6f6d03a540d042fa1e109b?buildbot=12
• Clients that hit the test failure: http://buildbot12.yelpcorp.com/builders/sandbox_65/builds/1906/steps/test/logs/stdio http://buildbot12.yelpcorp.com/builders/sandbox_7/builds/1907/steps/test/logs/stdio
• Master that displayed the results-insertion Unicode error: http://buildbot12.yelpcorp.com/builders/sandbox_0/builds/1909/steps/test/logs/stdio

[bukzor]

Taking this.

[bukzor]

158 allows us to see the full traceback:

ERROR    Exception while reporting results: UnicodeEncodeError('latin-1', u'Traceback (most recent call last):\n  File "./yelp/tests/test_fake.py", line 13, in test_faile_unicode2\n    assert False, u\'\u0141\xf3d\u017a is wonderful this time of year.\'\nAssertionError: \\u0141\\xf3d\\u017a is wonderful this time of year.\n', 123, 124, 'ordinal not in range(256)')
Traceback (most recent call last):
  File "/nail/home/buck/trees/Testify/testify/plugins/sql_reporter.py", line 279, in report_results
    result['previous_run_id'] = insert_single_run(result['previous_run'])
  File "/nail/home/buck/trees/Testify/testify/plugins/sql_reporter.py", line 259, in insert_single_run
    results = conn.execute(self.TestResults.insert(create_row_to_insert(result, previous_run_id=previous_run_id)))
  File "/nail/home/buck/trees/Testify/testify/plugins/sql_reporter.py", line 192, in create_row_to_insert
    'failure' : get_failure_id(result['exception_info']),
  File "/nail/home/buck/trees/Testify/testify/plugins/sql_reporter.py", line 252, in get_failure_id
    'traceback': traceback,
  File "/usr/lib/pymodules/python2.6/sqlalchemy/engine/base.py", line 1399, in execute
    params)
  File "/usr/lib/pymodules/python2.6/sqlalchemy/engine/base.py", line 1532, in _execute_clauseelement
    compiled_sql, distilled_params
  File "/usr/lib/pymodules/python2.6/sqlalchemy/engine/base.py", line 1633, in _execute_context
    context)
  File "/usr/lib/pymodules/python2.6/sqlalchemy/engine/default.py", line 330, in do_execute
    cursor.execute(statement, parameters)
  File "/usr/lib/pymodules/python2.6/MySQLdb/cursors.py", line 172, in execute
    query = query % db.literal(args)
  File "/usr/lib/pymodules/python2.6/MySQLdb/connections.py", line 275, in literal
    return self.escape(o, self.encoders)
  File "/usr/lib/pymodules/python2.6/MySQLdb/connections.py", line 203, in unicode_literal
    return db.literal(u.encode(unicode_literal.charset))
UnicodeEncodeError: 'latin-1' codec can't encode character u'\u0141' in position 123: ordinal not in range(256)

[bukzor]

The solution here was somewhat external to the testify project. Adding 'query: charset: utf8' to the connection yaml causes the MySQLdb driver to use utf8 rather than latin1 in this case, and everything starts working correctly.

This change is demonstrated in #159

Resolved.

[baris]

This is resolved already. Closing.