I am getting a UnicodeError exception when I try to post binary data through CoreRequestHandler Post method

[balaji1359]

When I try to send binary data from local file system to webserver through CoreRequestHandler post method, I am getting a UnicodeError Exception. Below code is my use case.

class Handler(CoreRequestHandler):

    def post(self):
        """
        :param: 
        :type: file.
        :return:
        :raise: HTTError 400
        """
       self.set_header("Content-type", 'application/octet-stream')
       for k, (info,) in self.request.files.items():
           name, body = info['filename'],  info['body'] 

Traceback:

Traceback (most recent call last):, "File \"/PycharmProjects/core4/core4/api/v1/request/main.py\", line 897, in prepare body_arguments = json_decode(self.request.body.decode(\"UTF-8\"))

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa3 in position 206: invalid start byte

pymongo.errors.DocumentTooLarge: BSON document too large (92181110 bytes) - the connected server supports BSON document sizes up to 16777216 bytes. File "/home/bnag/PycharmProjects/plannet4/.venv/lib/python3.5/site-packages/tornado/web.py", line 1678, in _execute result = await result

File "/home/bnag/PycharmProjects/core4/core4/api/v1/request/main.py", line 900, in prepare exc_info=True)

File "/usr/lib/python3.5/logging/init.py", line 1609, in warning self.log(WARNING, msg, *args, **kwargs)

File "/usr/lib/python3.5/logging/init.py", line 1641, in log self.logger._log(level, msg, args, **kwargs)

File "/usr/lib/python3.5/logging/init.py", line 1416, in _log self.handle(record)

File "/usr/lib/python3.5/logging/init.py", line 1426, in handle self.callHandlers(record)

File "/usr/lib/python3.5/logging/init.py", line 1488, in callHandlers hdlr.handle(record)

File "/home/bnag/PycharmProjects/core4/core4/logger/handler.py", line 102, in handle self._collection.insert_one(doc)

File "/home/bnag/PycharmProjects/plannet4/.venv/lib/python3.5/site-packages/pymongo/collection.py", line 693, in insert_one session=session),

File "/home/bnag/PycharmProjects/plannet4/.venv/lib/python3.5/site-packages/pymongo/collection.py", line 607, in _insert bypass_doc_val, session)

File "/home/bnag/PycharmProjects/plannet4/.venv/lib/python3.5/site-packages/pymongo/collection.py", line 595, in _insert_one acknowledged, _insert_command, session)

File "/home/bnag/PycharmProjects/plannet4/.venv/lib/python3.5/site-packages/pymongo/mongo_client.py", line 1248, in _retryable_write return self._retry_with_session(retryable, func, s, None)

File "/home/bnag/PycharmProjects/plannet4/.venv/lib/python3.5/site-packages/pymongo/mongo_client.py", line 1201, in _retry_with_session return func(session, sock_info, retryable)

File "/home/bnag/PycharmProjects/plannet4/.venv/lib/python3.5/site-packages/pymongo/collection.py", line 590, in _insert_command retryable_write=retryable_write)

File "/home/bnag/PycharmProjects/plannet4/.venv/lib/python3.5/site-packages/pymongo/pool.py", line 584, in command self._raise_connection_failure(error)

File "/home/bnag/PycharmProjects/plannet4/.venv/lib/python3.5/site-packages/pymongo/pool.py", line 745, in _raise_connection_failure raise error

File "/home/bnag/PycharmProjects/plannet4/.venv/lib/python3.5/site-packages/pymongo/pool.py", line 579, in command unacknowledged=unacknowledged)

File "/home/bnag/PycharmProjects/plannet4/.venv/lib/python3.5/site-packages/pymongo/network.py", line 119, in command message._raise_document_too_large(name, size, max_bson_size)

File "/home/bnag/PycharmProjects/plannet4/.venv/lib/python3.5/site-packages/pymongo/message.py", line 961, in _raise_document_too_large " bytes." % (doc_size, max_size))

[m-rau]

Issue is actually not so much the unicode/decode error. This is only a byproduct while parsing the parameters from the request body as json. If this cannot be decoded or is not valid json, then a warning is thrown but the request continues. The bigger issue is the logged warning. The exception UnicodeDecodeError adds the args to the exception record. Using mongo this fails if the arg is larger than 16mb. your suggested solution is pragmatic. Suggestion is to accept this PR.

[balaji1359]

You are right. UnicodeDecodeError is only as exception it is not the root cause of my error, as I am trying to send binary data, this cannot be decoded and as you said "The exception UnicodeDecodeError adds the args to the exception record" which is exactly happening in my case. Thank you for the quick support and accepting the suggestion. Finally, it works.

[m-rau]

has been merged into develop branch.