Closed wimglenn closed 2 years ago
Hi @wimglenn,
Sorry for the delayed response. I'm a bit sick so my response times will be kind of bad. I also haven't had a chance to toy with your reproduction test case but I sincerely appreciate you putting together a tiny one like that.
Could you also post the tracebacks you're seeing? Those would help me narrow this down and maybe guide you towards sending a pull request that fixes this. Would you be interested in doing that?
Sure, I can submit a PR. I just cloned and ran pytest
, all 132 existing tests are passing here. I'm a bit busy with other things now but I'll see if I can write a proper test and the fix when I get the chance to look into it more closely.
Traceback below:
TypeError Traceback (most recent call last)
<ipython-input-1-8769d5594677> in <module>()
17
18 with recorder.use_cassette('dump'):
---> 19 response1 = session.get(url, headers=headers)
20
21 with recorder.use_cassette('dump'):
/Users/wim/src/betamax/betamax/recorder.py in __exit__(self, *ex_args)
70
71 def __exit__(self, *ex_args):
---> 72 self.stop()
73 # ex_args comes through as the exception type, exception value and
74 # exception traceback. If any of them are not None, we should probably
/Users/wim/src/betamax/betamax/recorder.py in stop(self)
129 """Stop recording or replaying interactions."""
130 # No need to keep the cassette in memory any longer.
--> 131 self.betamax_adapter.eject_cassette()
132 # On exit, we no longer wish to use our adapter and we want the
133 # session to behave normally! Woooo!
/Users/wim/src/betamax/betamax/adapter.py in eject_cassette(self)
48 """Eject currently loaded cassette."""
49 if self.cassette:
---> 50 self.cassette.eject()
51 self.cassette = None # Allow self.cassette to be garbage-collected
52
/Users/wim/src/betamax/betamax/cassette/cassette.py in eject(self)
108
109 def eject(self):
--> 110 self._save_cassette()
111
112 def find_match(self, request):
/Users/wim/src/betamax/betamax/cassette/cassette.py in _save_cassette(self)
209 'recorded_with': 'betamax/{0}'.format(__version__)
210 }
--> 211 self.serializer.serialize(cassette_data)
212
213
/Users/wim/src/betamax/betamax/serializers/proxy.py in serialize(self, cassette_data)
66
67 with open(self.cassette_path, 'w') as fd:
---> 68 fd.write(self.proxied_serializer.serialize(cassette_data))
69
70 def deserialize(self):
/Users/wim/src/betamax/betamax/serializers/json_serializer.py in serialize(self, cassette_data)
15
16 def serialize(self, cassette_data):
---> 17 return json.dumps(cassette_data)
18
19 def deserialize(self, cassette_data):
/usr/local/Cellar/python3/3.5.2_1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/json/__init__.py in dumps(obj, skipkeys, ensure_ascii, check_circular, allow_nan, cls, indent, separators, default, sort_keys, **kw)
228 cls is None and indent is None and separators is None and
229 default is None and not sort_keys and not kw):
--> 230 return _default_encoder.encode(obj)
231 if cls is None:
232 cls = JSONEncoder
/usr/local/Cellar/python3/3.5.2_1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/json/encoder.py in encode(self, o)
196 # exceptions aren't as detailed. The list call should be roughly
197 # equivalent to the PySequence_Fast that ''.join() would do.
--> 198 chunks = self.iterencode(o, _one_shot=True)
199 if not isinstance(chunks, (list, tuple)):
200 chunks = list(chunks)
/usr/local/Cellar/python3/3.5.2_1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/json/encoder.py in iterencode(self, o, _one_shot)
254 self.key_separator, self.item_separator, self.sort_keys,
255 self.skipkeys, _one_shot)
--> 256 return _iterencode(o, 0)
257
258 def _make_iterencode(markers, _default, _encoder, _indent, _floatstr,
/usr/local/Cellar/python3/3.5.2_1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/json/encoder.py in default(self, o)
177
178 """
--> 179 raise TypeError(repr(o) + " is not JSON serializable")
180
181 def encode(self, o):
TypeError: b'xj8h' is not JSON serializable
I'm working on it in my fork
https://github.com/wimglenn/betamax/tree/bytes_vs_text_handling
Unfortunately the fix will have to be quite intrusive, because I think what needs to be done simply can't be done with JSONSerializer. That's because it's a format which doesn't correctly preserve the types of the strings inside a python dict.
on python2, it's data destructive:
>>> d = {'key1': 'val1', 'key2': u'val2'}
>>> d
{'key1': 'val1', 'key2': u'val2'}
>>> json.loads(json.dumps(d))
{u'key1': u'val1', u'key2': u'val2'}
on python3, it bails out (more correct, but makes betamax fail hard):
>>> d = {'key1': 'val1', 'key2': b'val2'}
>>> d
{'key1': 'val1', 'key2': b'val2'}
>>> json.loads(json.dumps(d))
TypeError: b'val2' is not JSON serializable
I ran in to a similar problem when using Requests, Betamax, and some third library such as suds-jurko or requests-oauthlib. Those extra libraries can introduce byte strings in to the flow, and the currently-available Betamax serializers can't handle bytes.
I wrote up some documentation about the problem, and a possible fix.
I think we have a partial fix but it's not yet been released. @hroncok would you have time to help prepare a release?
Sure. Not today, but sometimes during the weekend should be fine. Anything special, or just changelog + version bump + tag?
I don't think we need anything beyond what you mentioned. Thank you in advance. I'm traveling next week and won't have time for this any time soon
@sigmavirus24 and @christianmlong I wrote a test for one function that downloads a large zip file using requests module. I've found discrepancy in Content-Length when comparing test execution with betamax and without it. Using Betamax, the Content-Length of the binary string extracted is way larger. Besides that, I need to pass that binary string to BytesIO and then to zipfile.ZipFile
, but got zipfile.BadZipFile: Bad magic number for central directory
exception. Let me know if my problem is not related to this discussion.
My test setup:
import betamax
from betamax.fixtures import unittest
mode = os.getenv('BETAMAX_RECORD_MODE')
with betamax.Betamax.configure() as config:
config.cassette_library_dir = 'tests/test_funcs/cassettes'
config.default_cassette_options['record_mode'] = mode
print(f'Using record mode <{mode}>')
def the_function(session):
# session = requests.Session()
from io import BytesIO
from zipfile import ZipFile
response = session.get("https://ww2.stj.jus.br/docs_internet/processo/dje/xml/stj_dje_20211011_xml.zip")
zip_in_memory = BytesIO(response.content)
try:
my_zip = ZipFile(zip_in_memory, 'r')
my_zip.testzip()
result = True
except Exception:
result = False
return result
class BaseTest(unittest.BetamaxTestCase):
custom_headers = None
custom_proxies = None
_path_to_ignore = None
_no_generator_return_search = False
def setUp(self):
super(BaseTest, self).setUp()
if self.custom_headers:
self.session.headers.update(self.custom_headers)
if self.custom_proxies:
self.session.proxies.update(self.custom_proxies)
self.worker_under_test = self.worker_class()
self.worker_under_test._session = self.session
def test_search(self):
result = the_function(self.session)
assert result
I pass the self.session
to function under test and use it to get a endpoint. Through that endpoint, I get the zip file in the form of bytes string (response.content
). I found that test runs without errors if I don't use the Betamax session.
Related question: https://stackoverflow.com/questions/69653406/how-to-mock-a-function-that-downloads-a-large-binary-content-using-betamax
I don't believe your issue is related to this issue (and likely that we should close this one at this point). Please open an issue and include the headers from the response that you're recording.
I think there is something whack in the way non-ascii data is handled. I'm getting discrepancies in the response objects from live server and playback, and this is causing some issues in my app. I've tried to make a minimal example to reproduce the bug in a script below:
It fails in both python2 and python3, but with different failure modes.
In python2 it fails because the response1 and response2 have mutated.
In python3 it fails earlier, during the recording, whilst attempting to serialise a bytestring - an action which is a
TypeError
in python3 (and this gives a big hint at why the python2 version fails).