Response serialisation failing on Python 2.7

[adamlwgriffiths]

Using the example code in the README.md

original_response = requests.get("http://httpbin.org/get")
serialized_response = requestions.write_response(original_response)
response = requestions.read_response(serialized_response)

Receiving the following error

$ python test.py
Traceback (most recent call last):
  File "test.py", line 5, in <module>
    serialized_response = requestions.write_response(original_response)
  File "/media/sf_workspace/VirtualEnvs/sitebuilder/lib/python2.7/site-packages/requestions/io.py", line 78, in write_response
    return json.dumps(serialization)
  File "/home/adamgriffiths/.pythonbrew/pythons/Python-2.7.3/lib/python2.7/json/__init__.py", line 231, in dumps
    return _default_encoder.encode(obj)
  File "/home/adamgriffiths/.pythonbrew/pythons/Python-2.7.3/lib/python2.7/json/encoder.py", line 201, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/home/adamgriffiths/.pythonbrew/pythons/Python-2.7.3/lib/python2.7/json/encoder.py", line 264, in iterencode
    return _iterencode(o, 0)
  File "/home/adamgriffiths/.pythonbrew/pythons/Python-2.7.3/lib/python2.7/json/encoder.py", line 178, in default
    raise TypeError(repr(o) + " is not JSON serializable")
TypeError: CaseInsensitiveDict({'Accept-Encoding': 'gzip, deflate, compress', 'Accept': '*/*', 'User-Agent': 'python-requests/1.2.3 CPython/2.7.3 Linux/3.8.0-19-generic'}) is not JSON serializable

$ python --version
Python 2.7.3

[adamlwgriffiths]

test_requestions.py

(requestions)Vibur:src adamgriffiths$ python test_requestions.py
............
----------------------------------------------------------------------
Ran 12 tests in 0.045s

OK

If I add the following test the bug appears and the test fails.

class TestResponseSerialization(unittest.TestCase):

<snip>
    def test_string_response_simple(self):
        given_url = "http://httpbin.org/get"
        response = requests.models.Response()
        response.url = given_url
        sresponse = requestions.write_response(response)
        self.assertIn("url", sresponse.keys())
        self.assertEqual(given_url, sresponse["url"])

(requestions)Vibur:src adamgriffiths$ python test_requestions.py
.......E.....
======================================================================
ERROR: test_string_response_simple (__main__.TestResponseSerialization)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_requestions.py", line 61, in test_string_response_simple
    sresponse = requestions.write_response(response)
  File "/Users/adamgriffiths/Workspace/VirtualEnvs/requestions/src/requestions/io.py", line 78, in write_response
    return json.dumps(serialization)
  File "/Users/adamgriffiths/.pythonbrew/pythons/Python-2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/__init__.py", line 231, in dumps
    return _default_encoder.encode(obj)
  File "/Users/adamgriffiths/.pythonbrew/pythons/Python-2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/encoder.py", line 201, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/Users/adamgriffiths/.pythonbrew/pythons/Python-2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/encoder.py", line 264, in iterencode
    return _iterencode(o, 0)
  File "/Users/adamgriffiths/.pythonbrew/pythons/Python-2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/encoder.py", line 178, in default
    raise TypeError(repr(o) + " is not JSON serializable")
TypeError: CaseInsensitiveDict({}) is not JSON serializable

----------------------------------------------------------------------
Ran 13 tests in 0.013s

FAILED (errors=1)

The only difference is the return_string=False is removed.

Is this to be expected? Are there any issues from using return_string=False?

[adamlwgriffiths]

Ok, I understand why that revealed the issue. Obviously the test shouldn't be testing 'keys' against a string =P. Would be worth including the test but changing it to json.loads(...) the string back into the dict format for testing.

[kanzure]

So, the problem is that a CaseInsensitiveDict doesn't seem to be easily converted into a dictionary. Converting into a dictionary is required before it can be turned into a json string. I wanted a json string to be the default output of requestions, because it's not the responsibility of end users to write a serialization method-- that's supposed to be the responsibility of requestions.

I am still trying to think up a good solution. I wanted to use a method on CaseInsensitiveDict to convert to a regular dictionary before throwing it to the json module, but that doesn't work because it returns another CaseInsensitiveDict (as far as I can tell, but I haven't been very thorough when looking at this problem). I was thinking instead that I would have to write a function to crawl all of the key/value pairs and then convert anything that might be a CaseInsensitiveDict to a regular dictionary. After that, it would probably work.

I don't have time to write this code at the moment, I am gonna sleep.

[kanzure]

Hey, can you try the fix-case-insensitive-dict-serialization branch at commit ca81fb9d6e562cb852859a0d9d651a5371fdfc83? I'll merge this in tomorrow if this is working for you.

[adamlwgriffiths]

Seems to be working with what I throw at it =)

Using this as a test shows no problems. It's obviously not one you'd want to include verbatim as it does a proper 'get' and prints. But it proves the point.

    def test_write_string_response_simple(self):
        given_url = "http://httpbin.org/get"
        response = requests.get(given_url, headers={'User-Agent': 'test'})
        print response
        response.url = given_url
        sresponse = requestions.write_response(response)
        dresponse = json.loads(sresponse)
        self.assertIn("url", dresponse.keys())
        self.assertEqual(given_url, dresponse["url"])
        print dresponse['headers']
        nresponse = requestions.read_response(sresponse)
        print response
        print response.headers

[kanzure]

Cool, thanks. I have released v0.0.7 with this critical bugfix.