search

getmoto / moto

A library that allows you to easily mock out tests based on AWS infrastructure.
http://docs.getmoto.org/en/latest/
Apache License 2.0
7.66k stars 2.05k forks source link

moto not compatible with mock 4.0.3 #3535

Closed etiennechabert closed 3 years ago

etiennechabert commented 3 years ago

Versions Using moto-1.3.16 (boto-2.49.0 boto3-1.16.20 botocore-1.19.33)

Problem Mock has released 5h ago a new version: 4.0.3 Since then our tests are failing with the exception below.

Fix Forcing the version of mock to 4.0.2 is solving our issue

Traceback (most recent call last):
  File "/usr/local/Cellar/python@3.9/3.9.0_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/local/Cellar/python@3.9/3.9.0_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/usr/local/Cellar/python@3.9/3.9.0_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/unittest/__main__.py", line 18, in <module>
    main(module=None)
  File "/usr/local/Cellar/python@3.9/3.9.0_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/unittest/main.py", line 101, in __init__
    self.runTests()
  File "/usr/local/Cellar/python@3.9/3.9.0_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/unittest/main.py", line 271, in runTests
    self.result = testRunner.run(self.test)
  File "/usr/local/Cellar/python@3.9/3.9.0_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/unittest/runner.py", line 176, in run
    test(result)
  File "/usr/local/Cellar/python@3.9/3.9.0_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/unittest/suite.py", line 84, in __call__
    return self.run(*args, **kwds)
  File "/usr/local/Cellar/python@3.9/3.9.0_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/unittest/suite.py", line 122, in run
    test(result)
  File "/usr/local/Cellar/python@3.9/3.9.0_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/unittest/suite.py", line 84, in __call__
    return self.run(*args, **kwds)
  File "/usr/local/Cellar/python@3.9/3.9.0_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/unittest/suite.py", line 122, in run
    test(result)
  File "/usr/local/Cellar/python@3.9/3.9.0_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/unittest/suite.py", line 84, in __call__
    return self.run(*args, **kwds)
  File "/usr/local/Cellar/python@3.9/3.9.0_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/unittest/suite.py", line 122, in run
    test(result)
  File "/usr/local/Cellar/python@3.9/3.9.0_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/unittest/case.py", line 653, in __call__
    return self.run(*args, **kwds)
  File "../.venv/lib/python3.9/site-packages/moto/core/models.py", line 102, in wrapper
    self.stop()
  File "../.venv/lib/python3.9/site-packages/moto/core/models.py", line 86, in stop
    self.default_session_mock.stop()
  File "../.venv/lib/python3.9/site-packages/mock/mock.py", line 1563, in stop
    return self.__exit__(None, None, None)
  File "../.venv/lib/python3.9/site-packages/mock/mock.py", line 1529, in __exit__
    if self.is_local and self.temp_original is not DEFAULT:
bblommers commented 3 years ago

Thanks for raising it @etiennechabert - will investigate

bblommers commented 3 years ago

Do you have an example test case? I can't reproduce this with a bare-bones scenario:

pip install moto
pip show mock
Version: 4.0.3
@mock_s3
def test_x():
  print("working")
etiennechabert commented 3 years ago

Can you please add to yours:

With @mock_s3 these are all the mock we are using in the test failing with mock 4.0.3.

Additionally, we are also using @set_initial_no_auth_action_count(1) (moto.core) inside of a @mock_sts

Let me know if this helps, otherwise I come back with more tomorrow.

bblommers commented 3 years ago

Still no luck with this test case, @etiennechabert:

import boto3
from moto import mock_cloudwatch, mock_s3, mock_sts
from moto.core import set_initial_no_auth_action_count

@mock_cloudwatch
@mock_s3
@mock_sts
def test_x():
    s3 = boto3.client("s3", region_name="us-east-1")

    @set_initial_no_auth_action_count(1)
    def inside():
        print("inside")
    print(s3)
    inside()

I'm testing this all on Linux (Ubuntu 20.04.1), although I'm assuming that doesn't make a difference..

ostashkevych commented 3 years ago

can you try this?

import unittest

import boto3
from moto import mock_s3

@mock_s3
class MyTestCase(unittest.TestCase):

    def test_x(self):
        s3 = boto3.client("s3", region_name="us-east-1")

        def inside():
            print("inside")

        print(s3)
        inside()

if __name__ == "__main__":
    unittest.main()
bblommers commented 3 years ago

Thanks @ostashkevych - didn't think about the possibility of a class-decorator. i can reproduce it with that.

Attached a PR that should fix this issue.

etiennechabert commented 3 years ago

Thanks a lot for your help @ostashkevych

kzkv commented 3 years ago

To add search visibility, in my case the tests were failing with 

AttributeError: '_patch' object has no attribute 'is_local'

While we wait on the PR, I just threw in the mock==4.0.2 requirement.

marcelloromani commented 3 years ago

I'm having the same issue, my versions are:

boto3==1.16.35
moto==1.3.16
pynamodb==4.3.3
mock==4.0.2

was automatically installed (found out via $ pip freeze|grep mock )

@kzkv did mock==4.0.2 fix the issue for you? It didn't for me.

I'm trying to find which version of moto/mock doesn't show the bug.

kzkv commented 3 years ago

Yes, it did! But in my case I had mock==4.0.3 in pip list before the change.

marcelloromani commented 3 years ago

Thanks!

I have now mock==4.0.3 (which wasn't in my requirements.txt before) and I'm testing different moto versions.

moto==1.3.16 ==> FAIL (as expected) moto==1.3.15 ==> FAIL moto==1.3.14 ==> PASS

So in summary this version combination worked for me:

boto3==1.16.35
moto==1.3.14
mock==4.0.3
hadan commented 3 years ago

I have this problem as well. Using boto3==1.16.28 my current workaround is either moto==1.3.14 mock==4.0.3 or moto==1.3.16 mock==4.0.2

jgbreezer commented 3 years ago

The version 0.12.1 of responses in use with moto 1.3.16 (whether mock 4.0.2/3) also had a separate issue when using to test fetching presigned urls that I resolved by reverting to 0.12.0

rubenverhack commented 3 years ago

For reference. I was receiving the following TypeError:

  File "/usr/local/lib/python3.6/dist-packages/requests/api.py", line 76, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.6/dist-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/responses.py", line 733, in unbound_on_send
    return self._on_request(adapter, request, *a, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/responses.py", line 680, in _on_request
    match, match_failed_reasons = self._find_match(request)
TypeError: 'CallbackResponse' object is not iterable

When executing the following code:

    @mock_sqs
    def test_36(self):
        print(requests.get('http://localhost:3000/businesses/6c213885-7b2c-425a-8280-9af507d9ff07/sites'))

It was fixed as suggested by @marcelloromani by setting

boto3==1.16.28 # AWS library
moto==1.3.14  # Mock boto3 client in unittests
mock==4.0.3
estraviz commented 3 years ago

Downgrading moto to 1.3.14 in my project as v1.3.16 and mock v4.0.3 didn't work (mocking s3), my tests had stopped working after recreating the virtual env (using pyenv now). All tests in green before with v1.3.14. Now everything is fine again.

bblommers commented 3 years ago

We've just released a fix, and Moto >= 1.3.17.dev264 should be compatible with mock == 4.0.3.

Please let us know if you're still experiencing issues.

jordimares commented 3 years ago

Thanks!

lkev commented 3 years ago

Just to add context, I was failing with

boto3==1.17.9
moto==1.3.16
mock==4.0.3

and also had to downgrade to moto==1.3.14 - now working again

potiuk commented 3 years ago

Is there a chance 1.3.17 will be released soon ?

bblommers commented 3 years ago

Hi @potiuk, we've gone with a major release instead - see https://github.com/spulec/moto/issues/3722 Every 2.x release contains this fix - the latest release is 2.0.2 atm

potiuk commented 3 years ago

I see. We will have to likely upgrade to never version of boto3 as well. Is there any relation between moto and boto version? I'e currently in airflow we have 'boto3>=1.15.0,<1.18.0'. Do you think we have to increase the lower boundary if we use moto>=2 ?

bblommers commented 3 years ago

That boundary should be fine afaik, @potiuk. Moto only requires boto3>=1.9.201, because they do a good job of keeping things backward compatible. There's no relation between versions, though. There are best efforts to keep it working with the latest version of boto3, and to only require a boto3-update if absolutely necessary.

potiuk commented 3 years ago

Thanks. I am already finishing the migration to moto2 . We had a couple of tests that started to fail with moto2 and worked before, but I already fixed them.

potiuk commented 3 years ago

PR in Apache Airflow: https://github.com/apache/airflow/pull/15051