Sep52/client get requests (closes #69) (closes #70)

[sputney13]

@mlp6 @suyashkumar

I've come across a very unfortunate discovery while trying to write client requests for my Flask handles. First I discovered that when I post up an image to my database, it is posting the binary rather than the encoded image. I even printed off the result of our encoding function right before into was saved into the db, saw that it was base64, and it still posted as a binary to the database. (The Flask handle that deals with uploading images is lines 184-209 in final.py, the client function that calls on this POST is lines 41-56 in client.py).

I figured that the database must just hate base64, and decided to try and test the rest of the code by re-encoding the data to base64 every time I pulled it from the database with a GET. The Flask handle for GETting uploaded images is lines 229-250 in final.py, and the client function calling on it is lines 59-73 in client.py. Despite including the encoder, and then printing off the encoded image to ensure that it was base64 before jsonifying it (this print is included in the error message below), I STILL received this error (below) when running clienty.py that says I am trying to jsonify bytes rather than a base64 string.

Is there something that I am doing wrong with regards to encoding the image? Why will it not save to my database as a base64? And if it cannot save to the database as a base64, why does it not work when I re-encode it?

127.0.0.1 - - [10/Dec/2018 19:54:18] "POST /image/user HTTP/1.1" 200 - 127.0.0.1 - - [10/Dec/2018 19:54:18] "GET /image/user/sarah.putney@duke.edu HTTP/1.1" 200 - 127.0.0.1 - - [10/Dec/2018 19:54:18] "POST /image/upload HTTP/1.1" 200 - {'test_images/capy': b'/9j/4AAQSkZJRgABAQE ........ '} [2018-12-10 19:54:18,635] ERROR in app: Exception on /image/upload/sarah.putney@duke.edu [GET] Traceback (most recent call last): File "C:\Users\Student\AppData\Local\Programs\Python\Python37-32\lib\site-packages\flask\app.py", line 2292, in wsgi_app response = self.full_dispatch_request() File "C:\Users\Student\AppData\Local\Programs\Python\Python37-32\lib\site-packages\flask\app.py", line 1815, in full_dispatch_request rv = self.handle_user_exception(e) File "C:\Users\Student\AppData\Local\Programs\Python\Python37-32\lib\site-packages\flask\app.py", line 1718, in handle_user_exception reraise(exc_type, exc_value, tb) File "C:\Users\Student\AppData\Local\Programs\Python\Python37-32\lib\site-packages\flask_compat.py", line 35, in reraise raise value File "C:\Users\Student\AppData\Local\Programs\Python\Python37-32\lib\site-packages\flask\app.py", line 1813, in full_dispatch_request rv = self.dispatch_request() File "C:\Users\Student\AppData\Local\Programs\Python\Python37-32\lib\site-packages\flask\app.py", line 1799, in dispatch_request return self.view_functionsrule.endpoint File "C:/Users/Student/Documents/bme590final/final.py", line 250, in get_uploaded_images return jsonify(uploaded_images) File "C:\Users\Student\AppData\Local\Programs\Python\Python37-32\lib\site-packages\flask\json__init.py", line 321, in jsonify dumps(data, indent=indent, separators=separators) + '\n', File "C:\Users\Student\AppData\Local\Programs\Python\Python37-32\lib\site-packages\flask\json__init.py", line 179, in dumps rv = _json.dumps(obj, kwargs) File "C:\Users\Student\AppData\Local\Programs\Python\Python37-32\lib\json__init__.py", line 238, in dumps kw).encode(obj) File "C:\Users\Student\AppData\Local\Programs\Python\Python37-32\lib\json\encoder.py", line 199, in encode chunks = self.iterencode(o, _one_shot=True) File "C:\Users\Student\AppData\Local\Programs\Python\Python37-32\lib\json\encoder.py", line 257, in iterencode return _iterencode(o, 0) File "C:\Users\Student\AppData\Local\Programs\Python\Python37-32\lib\site-packages\flask\json\init__.py", line 81, in default return _json.JSONEncoder.default(self, o) File "C:\Users\Student\AppData\Local\Programs\Python\Python37-32\lib\json\encoder.py", line 179, in default raise TypeError(f'Object of type {o.class.name__} ' TypeError: Object of type bytes is not JSON serializable 127.0.0.1 - - [10/Dec/2018 19:54:18] "GET /image/upload/sarah.putney@duke.edu HTTP/1.1" 500 -

[mlp6]

https://github.com/EricaSkerrett/bme590final/blob/master/final.py#L184-L209

[mlp6]

@sputney13 is there a unit test that re-creates that error message above?

[sputney13]

@EricaSkerrett status on the image_encoder unit test? I guess the problem would most likely be that image_encoder doesn't work and that is where the problem lies.

... does a string that starts with b' indicate a bytes-type instead of a base64? I suppose I should have asked that first. Because in that case, our image_encoder is 100% the problem

[mlp6]

https://docs.python.org/3/reference/lexical_analysis.html#string-and-bytes-literals

[mlp6]

Given above URL, yes; byte literals.

[sputney13]

Wow...

I have no words. Guess I'll go fix the image encoder. Thanks for your help, Mark!

Except... I'm also not sure what's wrong with the image encoder. I'll push up a failing unit test in a minute so that you can see what's up.

[mlp6]

:) Base64 encoding needs explicit use of the base64 module (https://docs.python.org/3.7/library/base64.html).

[sputney13]

I think this just made me even more confused. So the base64 module returns bytes... which we can't use. So there is no point in encoding whatsoever, except for when we're doing the processing?

In which case... we needed to be storing the image as a binary the entire time? And encoding it was working but I didn't want to be encoding it?

[mlp6]

Well, remember that base64 is used as a data standard for the JSON data payload; that is why we're using that specific encoding. Otherwise, if you already have a byte-string, then you are just converting to a base64 symbol set when you do that encoding... it is still then of bytes type.

[sputney13]

Then why is it that I'm get a TypeError: Object of type bytes is not JSON serializable? If using base64.b64encode on an image gives back a bytes type... and that bytes type cannot be saved into a JSON... then what is happening here?

https://github.com/EricaSkerrett/bme590final/blob/beefa498a35ad3bc76096e8b03417d3f3e38d9ae/final.py#L138

It seems like the encoding processing should be fine?

[EricaSkerrett]

I haven't written the unit test for the encoder yet, only tested it's output on ipython. It returns a dictionary of image names as keys and their b64 strings.

[RoujiaW]

A couple resource might be relevant: https://stackoverflow.com/questions/44682018/typeerror-object-of-type-bytes-is-not-json-serializable https://stackoverflow.com/questions/13262125/how-to-convert-from-base64-to-string-python-3-2

summarized from slack conversation

[sputney13]

hallelujah