okigan / awscurl

curl-like access to AWS resources with AWS Signature Version 4 request signing.
MIT License
737 stars 91 forks source link

Add encode to support non-English text data #75

Closed muziyoshiz closed 4 years ago

muziyoshiz commented 4 years ago

When I call awscurl 0.17 with non-English text data, it fails with the following UnicodeEncodeError.

$ awscurl -XGET --service es "${ENDPOINT}/${INDEX}/_search" -d '日本語'
Traceback (most recent call last):
  File "/usr/bin/awscurl", line 11, in <module>
    load_entry_point('awscurl==0.17', 'console_scripts', 'awscurl')()
  File "/usr/lib/python3.6/dist-packages/awscurl/awscurl.py", line 395, in main
    args.insecure
  File "/usr/lib/python3.6/dist-packages/awscurl/awscurl.py", line 220, in make_request
    return __send_request(uri, data, headers, method, verify)
  File "/usr/lib/python3.6/dist-packages/awscurl/awscurl.py", line 244, in __send_request
    r = requests.request(method, uri, headers=headers, data=data, verify=verify)
  File "/usr/lib/python3.6/dist-packages/requests/api.py", line 60, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python3.6/dist-packages/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3.6/dist-packages/requests/sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3.6/dist-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/usr/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/usr/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 354, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib64/python3.6/http/client.py", line 1254, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib64/python3.6/http/client.py", line 1299, in _send_request
    body = _encode(body, 'body')
  File "/usr/lib64/python3.6/http/client.py", line 171, in _encode
    (name.title(), data[err.start:err.end], name)) from None
UnicodeEncodeError: 'latin-1' codec can't encode characters in position 126-127: Body ('日本語') is not valid Latin-1. Use body.encode('utf-8') if you want to send it encoded in UTF-8.

It is because the data passed to requests.request method is not encoded. Please check this pull request. Thanks.

okigan commented 4 years ago

Thanks -- Please add a unit test

muziyoshiz commented 4 years ago

@okigan Thanks for your review! I added the integration tests both for English data and non-English data. Please check.

muziyoshiz commented 4 years ago

Thank you!

muziyoshiz commented 4 years ago

@okigan Would you please let me know when the next release will be? Now I am using awscurl v0.17 with my patch. It is good to install the latest master version with pip for non-English users like me. Thanks.

okigan commented 4 years ago

next day or two, thanks!

okigan commented 4 years ago

"next day or two" did not account for the thanksgiving week -- hope everyone had a good break as v0.18 is now released and up on pypi, happy awscurling!

muziyoshiz commented 4 years ago

I see 😅 No problem. Thank you very much!