UnicodeEncodeError when running search with utf8-only characters

[bendavis78]

Running the following line produces a UnicodeEncodeError:

search = "DisplayName = 'Kimberly O’Neil"
result = objects.Customer.where(search, qb=client)

The apostrophe in the string is character code U+2019 - RIGHT SINGLE QUOTATION MARK

Full exception below:

Traceback (most recent call last):
  ...
  File "/srv/hive/src/hive/billing/accounts.py", line 64, in get_customer
    result = objects.Customer.where(search, qb=client)
  File "/srv/hive/lib/python3.6/site-packages/quickbooks/mixins.py", line 250, in where
    return cls.query(select, qb=qb)
  File "/srv/hive/lib/python3.6/site-packages/quickbooks/mixins.py", line 262, in query
    json_data = qb.query(select)
  File "/srv/hive/lib/python3.6/site-packages/quickbooks/client.py", line 297, in query
    result = self.post(url, select, content_type='application/text')
  File "/srv/hive/lib/python3.6/site-packages/quickbooks/client.py", line 241, in post
    return self.make_request("POST", *args, **kwargs)
  File "/srv/hive/lib/python3.6/site-packages/quickbooks/client.py", line 219, in make_request
    req = self.process_request(request_type, url, headers=headers, params=params, data=request_body)
  File "/srv/hive/lib/python3.6/site-packages/quickbooks/client.py", line 250, in process_request
    request_type, url, headers=headers, params=params, data=data)
  File "/srv/hive/lib/python3.6/site-packages/rauth/session.py", line 358, in request
    return super(OAuth2Session, self).request(method, url, **req_kwargs)
  File "/srv/hive/lib/python3.6/site-packages/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/srv/hive/lib/python3.6/site-packages/requests/sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "/srv/hive/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/srv/hive/lib/python3.6/site-packages/urllib3/connectionpool.py", line 677, in urlopen
    chunked=chunked,
  File "/srv/hive/lib/python3.6/site-packages/urllib3/connectionpool.py", line 392, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib/python3.6/http/client.py", line 1239, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.6/http/client.py", line 1284, in _send_request
    body = _encode(body, 'body')
  File "/usr/lib/python3.6/http/client.py", line 161, in _encode
    (name.title(), data[err.start:err.end], name)) from None
UnicodeEncodeError: 'latin-1' codec can't encode character '\u2019' in position 54: Body ('’') is not valid Latin-1. Use body.encode('utf-8') if you want to send it encoded in UTF-8.

[TigerWalts]

I ran into the same problem, but when using the filter method. The workaround I found was to convert the string to latin-1 encoding.

Using where:

display_name = 'Kimberly O’Neil'.encode('utf-8').decode('latin-1') # Kimberly OâNeil
search = f"DisplayName = '{display_name.replace("'", r"\'")}'"
result = objects.Customer.where(search, qb=client)

Using filter (Useful if matching whole fields and can be build dynamically):

display_name = 'Kimberly O’Neil'.encode('utf-8').decode('latin-1')
search = {
    'DisplayName': display_name
}
result = objects.Customer.filter(**search, qb=client)

You could also use json to make the string safe, but I haven't tried that yet ([1:-1] removes the double quotes):

display_name = json.dumps('Kimberly O’Neil')[1:-1] # Kimberly O\u2019Neil