kevin1024 / vcrpy

Automatically mock your HTTP interactions to simplify and speed up testing
MIT License
2.71k stars 388 forks source link

boto3 problems after updating to 2.1.x #472

Open maxcountryman opened 5 years ago

maxcountryman commented 5 years ago

It seems like I'm unable to get any of my cassettes to playback correctly after upgrading vcrpy from 2.0.1 to 2.1.0. In particular, this is the error I'm running into:

request = <SubRequest '_vcr_marker' for <Function test_datadog_failure_callback>>

    @pytest.fixture(autouse=True)
    def _vcr_marker(request):
        marker = request.node.get_closest_marker('vcr')
        if marker:
>           request.getfixturevalue('vcr_cassette')

../../../Library/Caches/pypoetry/virtualenvs/lattice-py3.7/lib/python3.7/site-packages/pytest_vcr.py:46:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../../../Library/Caches/pypoetry/virtualenvs/lattice-py3.7/lib/python3.7/site-packages/pytest_vcr.py:87: in vcr_cassette
    with vcr.use_cassette(vcr_cassette_name, **kwargs) as cassette:
../../../Library/Caches/pypoetry/virtualenvs/lattice-py3.7/lib/python3.7/site-packages/vcr/cassette.py:96: in __enter__
    return next(self.__finish)
../../../Library/Caches/pypoetry/virtualenvs/lattice-py3.7/lib/python3.7/site-packages/vcr/cassette.py:64: in _patch_generator
    for patcher in CassettePatcherBuilder(cassette).build():
../../../Library/Caches/pypoetry/virtualenvs/lattice-py3.7/lib/python3.7/site-packages/vcr/patch.py:123: in _build_patchers_from_mock_triples
    for args in mock_triples:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <vcr.patch.CassettePatcherBuilder object at 0x10780b5c0>

    @_build_patchers_from_mock_triples_decorator
    def _boto3(self):

        try:
            # botocore using awsrequest
            import botocore.awsrequest as cpool
        except ImportError:  # pragma: no cover
            try:
                # botocore using vendored requests
                import botocore.vendored.requests.packages.urllib3.connectionpool as cpool
            except ImportError:  # pragma: no cover
                pass
            else:
                from .stubs import boto3_stubs
                yield self._urllib3_patchers(cpool, boto3_stubs)
        else:
            from .stubs import boto3_stubs
            log.debug("Patching boto3 cpool with %s", cpool)
>           yield cpool.AWSHTTPConnectionPool, 'ConnectionCls', boto3_stubs.VCRRequestsHTTPConnection
E           AttributeError: module 'botocore.awsrequest' has no attribute 'AWSHTTPConnectionPool'

../../../Library/Caches/pypoetry/virtualenvs/lattice-py3.7/lib/python3.7/site-packages/vcr/patch.py:210: AttributeError
neozenith commented 5 years ago

Thank you for reporting this bug! Much appreciated.

That is disappointing an update caused a break. In the meantime, would pinning your version to v2.0.1 act as a work around?

Would you be open to contributing a pull request to address this? I'd be more than happy to collaborate getting the TravisCI to pass to get it over the line. Then we can put out a v2.1.1 fix shortly.

If you haven't got the bandwidth, that's fine too. Is there a toy repo that we could clone to replicate the issue at least?

fgblomqvist commented 5 years ago

Ran into this issue, but it was fixed by updating boto3 from 1.6.x to 1.9.x.

maxcountryman commented 5 years ago

Ran into this issue, but it was fixed by updating boto3 from 1.6.x to 1.9.x.

I’ll have to look into this but I believe boto3 is version locked by another dep in our case.

fgblomqvist commented 5 years ago

Alright. At the very least the >= specifier should probably be used since VCR breaks with too low of a version of boto3. But yeah thanks!

maxcountryman commented 5 years ago

Alright. At the very least the >= specifier should probably be used since VCR breaks with too low of a version of boto3. But yeah thanks!

Our lock is requiring this: boto3 = ">=1.7.0,<1.8.0" which is believe is tied directly to Airflow.

We're currently on Airflow ^1.10.

fgblomqvist commented 5 years ago

I can try some different versions of boto3 a little later, and see if one of the 1.7.x versions would work

fgblomqvist commented 5 years ago

Actually the issue here is just that VCR.py needs to specify that it depends on boto3>=1.8 (that's when it starts working). Then it's just up to @maxcountryman to update his Airflow version or something like that. @neozenith

maxcountryman commented 5 years ago

Then it's just up to @maxcountryman to update his Airflow version or something like that.

Well, no: Airflow is version-locking boto3, so we can't "just update it", Airflow is specifying it depends on a version of boto3 which is apparently not properly supported by this library.

We're also using the most recent release of Airflow, 1.10.5 so if you're suggesting we can simply update Airflow to fix this that won't work either.

fgblomqvist commented 5 years ago

Sure, but what I'm arguing is that this is not vcrpy's fault. You should open an issue with Airflow and tell them to update their boto3 (1.7.x is quite old), and until they do, you're just gonna have to run an older version of vcrpy. If all Python libraries had to support really old versions of dependencies just because some libraries refuse to update, the ecosystem would never get anywhere.

neozenith commented 5 years ago

I've exhausted all the bandwidth my employer had allocated me to get VCRpy updated. Unless it is approving PRs I have no bandwidth on issues anymore sorry.

Just leaving a comment here so others are aware and I'm not ghosting the project but setting expectations.

If there is a change to setup.py dependencies that can help, feel free to fork this project, make the change to setup.py then use the information in this link to pip install your fork for testing.

eg,

pip install https://github.com/neozenith/vcrpy/archive/master.zip

https://stackoverflow.com/a/24811490/622276

Then open a PR to bring it back into the main branch of VCRpy.