Closed Mr0grog closed 11 months ago
Well, this got a little complicated:
The actual updates for urllib3 v2 compatibility (seen in wayback/_client.py
) are a bit hacky, but relatively straightforward. Instead of replacing the from_httplib()
static method, which doesn’t have a replacement that is as overridable in v2, we patch the HTTPHeaderDict
class to repair the headers.
VCR became a really complicated configuration. When VCR added support for urllib3 v2, it did it in such a way that the same request and response generate different cassette files for v1 and v2 of urllib3. Namely, the bodies of gzipped responses are recorded decompressed for v2, but still have the headers and such declaring them as gzipped. These cassette files can be written and read safely with either v1 or v2 of urllib3, but cannot be written with one and read with the other. More info in this VCR issue: https://github.com/kevin1024/vcrpy/issues/719
The solution I wound up with here (see wayback/tests/support.py
) is to have a custom serializer that compresses the bodies when serializing to disk and decompresses them when deserializing from disk if urllib3 v2 is in use (it does nothing for v1, which records the actual raw bytes just fine). This is imperfect (we could wind up with bytes that don’t match the content-length
header, or other weird edge cases), but it lets the same cassette files be read or written regardless of which version of urllib3 you are testing or developing with.
I originally went for a simpler solution that just had separate cassette files for each urllib3 version (you can see that in a664a9533cb3cead7223454f69ad877059f68e4c). After thinking about it, though, I realized this puts a huge burden on contributors, who have to have two development environments set up (one with each urllib3 version) and then record cassettes in both environments when making new tests. Super complex! The custom serializer is way more code, but keeps development straightforward.
This includes a change to every cassette file, but in all but one case those are just adding .yaml
to the end of the file names (I noticed this was missing while messing with all this stuff).
Some recordings and test implementations have minor changes because stuff changed upstream (but in an innocuous way; if you are testing against a recording it would have kept working, but you wouldn’t have been able to re-generate the recording because some things we expected to fail on the Wayback Machine’s servers started succeeding, which is good. I replaced them with other things that still fail in the same way.).
Rebased on main
to mix correctly with new changes there.
Back in #118 we “fixed” things for urllib3 v2 by marking this package as only compatible with v1, so users wouldn't wind up with bad dependency combinations. However, we ultimately need to actually support urllib3 v2 and remove that limitation, which I intend to do here. Will fix #116.