Test cases written in moto using Latest version of boto3 fails

[krishnamee2004]

Test cases written in moto makes actual AWS API calls to botocore instead of mocking them. This happens with the latest version of boto3 (1.8.). It used to work fine without issues with 1.7. versions.

Sample code to reproduce error

import boto3
import json
from moto import mock_s3

@mock_s3
def test_mock_s3():
    client = boto3.client('s3', region_name='us-east-1')
    client.create_bucket(Bucket='testbucket')
    response = client.list_buckets()
    print json.dumps(response, default=str)

if __name__ == "__main__":
    test_mock_s3()

Expected result

Method should return the ListBuckets response. It should look something like:

{"Owner": {"DisplayName": "webfile", "ID": "bcaf1ffd86f41161ca5fb16fd081034f"}, "Buckets": [{"CreationDate": "2006-02-03 16:45:09+00:00", "Name": "testbucket"}], "ResponseMetadata": {"RetryAttempts": 0, "HTTPStatusCode": 200, "HTTPHeaders": {"Content-Type": "text/plain"}}}

Actual error

botocore.errorfactory.BucketAlreadyExists: An error occurred (BucketAlreadyExists) when calling the CreateBucket operation: The requested bucket name is not available. The bucket namespace is shared by all users of the system. Please select a different name and try again.

Full stack trace

Traceback (most recent call last):
  File "testcases.py", line 14, in <module>
    test_mock_s3()
  File "/private/tmp/virtualenv2/lib/python2.7/site-packages/moto/core/models.py", line 71, in wrapper
    result = func(*args, **kwargs)
  File "testcases.py", line 8, in test_mock_s3
    client.create_bucket(Bucket='testbucket')
  File "/private/tmp/virtualenv2/lib/python2.7/site-packages/botocore/client.py", line 314, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/private/tmp/virtualenv2/lib/python2.7/site-packages/botocore/client.py", line 612, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.errorfactory.BucketAlreadyExists: An error occurred (BucketAlreadyExists) when calling the CreateBucket operation: The requested bucket name is not available. The bucket namespace is shared by all users of the system. Please select a different name and try again.

Library versions

moto : 1.3.4
boto3 : 1.8.1 - fails
boto3 : 1.7.84 - succeeds

[yingyi-nauto]

Similar problem with mock_sqs. Got the following error: ClientError: An error occurred (AccessDenied) when calling the CreateQueue operation: Access to the resource https://queue.amazonaws.com/ is denied.

[rouge8]

It looks like the issue is actually with botocore >= 1.11.0 which no longer uses requests and instead directly uses urllib3: https://github.com/boto/botocore/pull/1495. This means moto probably can't use responses anymore...

[garyd203]

Obviously we want to update moto so that it can work with the new implementation of botocore.

Whilst we figure out what this looks like, is it worthwhile restricting moto's install_requires in setup.py to require only compatible versions of botocore? Clearly this isn't a perfect solution, but if it saves some pain for some of our users then it could be worthwhile.

EDIT: I have added a PR for this as a discussion starter

[grudelsud]

pinning to boto3<1.8 is not really a solution as it would outdate the library pretty soon and doesn't solve when installing dependencies relying on newer versions either, as this will cause incompatibilities.
I appreciate this comment doesn't bring much to the table, but can we have an idea of how big of an effort would be to update moto to reflect the current status of libraries?

[garyd203]

@grudelsud I agree that pinning boto3 is not a viable long-term solution, and introduces problems of its own. But given that users are currently being affected by tests silently falling back to making real boto3 calls (which can have significant unexpected side-effects if you happen to have valid AWS credentials available), it seems worthwhile to consider a quick solution as well a permanent solution. I feel that, for now, the problems from pinning boto3 are lower severity than doing nothing.

That said, I would be a million times happier if someone put forward a genuine fix. But I lack the time and knowledge to do that myself at this point, or even to guess how much work would be involved.

[joguSD]

@garyd203 I've been driving a lot of the changes to get botocore tracking upstream dependencies instead of using our vendored versions. What does moto require so that you guys wouldn't need to monkey patch our dependencies?

[garyd203]

Hi @joguSD , thanks for asking. I'm just an occasional contributor to moto myself, and I'm not very familiar with the internals of moto. That said, I will try to provide some comments...

The goal of moto is to provide a fake implementation of specified AWS services which are accessed via boto3, as if they were the live services. FWIW, my understanding of the current implementation is that we have achieved this by mocking out HTTPAdapter.send in botocore's vendored version of requests,so that we can inspect each request and either pass it off to our internal handler for the fake service, or pass it through to the original send. You can see this in moto.core.models, with botocore_mock and ResponsesMockAWS

Moving forward, there seem to be a few choices for how we could do this better. I suspect we are constrained to work with the HTTP request rather than some other part of the boto3/botocore stack. So just to start the discussion, here's a couple of options:

1. Use a pluggable HTTP client backend for botocore, so that moto can wrap the standard HTTP backend with it's own interceptor functionality
2. Add a filter/interceptor chain in botocore's HTTP request handling, where moto can inject it's own filter early on and modify the behaviour based on what request is being made.

Do you have any opinion on these (or other) implementation choices, from botocore's perspective? My personal preference would be a pluggable backend with a standard implementation that is extensible and/or wrappable.

Again, don't take any of this discussion as definitive (I can't speak for the moto project maintainers), but I hope it helps.

[jbaum-cmcrc]

Can we have a release for the temporary work-around, please?

Everyone has to pin to boto3<1.8 until the proper fix is made in any case; it would be better to have that pin in one place (moto) rather than in every project that depends on it.

[spulec]

As a temporary measure, I've merged #1794 and released version 1.3.5.

If anyone does still want to use the newer boto3, it should work properly with the deprecated decorators (@mock_ecs_deprecated, etc). Those decorators mock at the socket level. We had been wanting to move away from that because they are fragile, but we may need to reconsider now.

Another option is to do a comparable thing to what we were doing before, but for urllib3. There looks to be at least one library out there: https://github.com/florentx/urllib3-mock

Finally another option is to stop patching all together and switch to moto only supporting standalone mode. I don't particularly love this, but if it means avoiding hacky patching issues like this in the future, I think we seriously need to consider it.

So, we've released a temporary fix for now, but we need to think through some bigger picture questions in the near-term so people aren't locked to old boto3/botocore. Since this is probably as good a place as any, feel free to comment here with thoughts.

[joguSD]

@spulec I'd rather we reach some sort of compromise that allows you to continue doing what you are now without having to patch anything.

@garyd203 I personally really like option 1, and is a large reason I've begun working on refactoring how botocore makes HTTP requests. I'd like to eventually get to a place where it's relatively easy to provide alternative HTTP clients, but I'm not sure how far out that is as we'll have to solidify a lot of interfaces we've considered internal to botocore.

As for option 2, that seems a little more reasonable in the short term. We've been playing with the idea of a before-send event in botocore that would allow custom handlers to return an HTTP response instead of going through botocore's default HTTP layer.

[joguSD]

I can't make any promises yet but something like this is what I was thinking for option 2:

from botocore.session import Session
from botocore.awsrequest import AWSResponse

class RawResponse(object):
    # a stubbed out urllib3-like response object
    def stream(self):
        yield b''

def stub_http(event_name, request, **kwargs):
    # return None here to let the botocore actually send the request
    return AWSResponse(request.url, 200, {}, RawResponse())

client = Session().create_client('s3')
# NOTE: this event doesn't exist
client.meta.events.register('before-send', stub_http)
# gets a stubbed response
print(client.list_buckets())
client.meta.events.unregister('before-send', stub_http)
# gets a real response
print(client.list_buckets())

To actually get something like this pushed through we would need to finalize and make the AWSRequest and AWSResponse interfaces public, and make what we expect from a raw response solid.

I think the last blocker would be injecting the event handler without patching which might not be possible given moto's current interface (no direct reference to the client/session).

[will-ockmore]

Just ran into this issue with an internal project, the dependencies were not pinned so when running on the CI server the newer version was installed and ran against our infrastructure. Thankfully nothing serious was changed by the tests.

The sub-dependency pinning fixes (#1794 , #1800 ) are possibly not sufficient - and I think for such a potentially damaging / expensive problem further steps should be taken (error on import if the versions are not correct perhaps?)

Obviously best practice is to pin everything, but sadly this doesn't happen all the time!

With the latest releases available on PyPI, steps to access your real AWS resources with no errors / pip install warnings (using python 3.6.4 on my mac):

# set up environment
$ python -m venv env && source env/bin/activate && pip install boto3 moto
$ python
Python 3.6.4 (default, Mar 12 2018, 13:30:09)
[GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.39.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import boto3
>>> import moto
>>> with moto.mock_s3():
...     s3 = boto3.resource('s3')
...     print([x for x in s3.buckets.all()])

# see the list of all your real s3 buckets

Install order matters, if you install moto first (eg. pip install moto boto3) it correctly pins botocore; but as this can't be depended on the problem remains.

[michaelPotter]

I agree with @will-ockmore; dependency management can be a tricky thing to get right. Checking the botocore version and raising an error would not only be more reliable in preventing access to real systems, but it would also inform users as to why their tests might have suddenly stopped working.

[michael-k]

more reliable in preventing access to real systems

You should never rely on moto or any other library to prevent (accidental) access to your production resources. The only way to prevent this is to not use production credentials outside of production (dev, test, stage, …).

[will-ockmore]

You should never rely on moto or any other library to prevent (accidental) access to your production resources. The only way to prevent this is to not use production credentials outside of production (dev, test, stage, …).

Totally agreed; but you have to account for the human factor (like junior developers!).

Another thing to bear in mind is that moto doesn't require credentials in the same way as boto, so if you have your local aws credentials set up (eg. in ~/.aws) and you run the tests locally on a project with no credentials override, you will encounter the same problem.

Thinking out loud - but perhaps moto should require credentials - and warn that they shouldn't be your real creds?

[2rs2ts]

I mean, we don't do development in production environments, but it's still expensive when your test suite spins up a couple dozen VPCs and EC2 instances.

[luyun-aa]

I ran into this issue when using moto with aws-xray-sdk. I'm working on a fix: https://github.com/spulec/moto/pull/1808

[luyun-aa]

Hi there,

I was rushing to a working implementation and wasn't aware of the discussion above until I've completed most part of the patch, in PR #1808. Basically it took an intuitive approach to mock botocore.httpsession.URLLib3Session.send instead of requests. To make it work, I need to:

• introduce a modified version of the responses library, tailored for URLLib3Session and AWSRequest
• test the vendored responses library
• update tests to send http request by URLLib3Session
• wrap request and response object to make them similar to that of requests

I've also tried another two approaches:

• patching requests and URLLib3Session at the same time: doesn't work because their request objects are way too different
• patching at urllib3 level:
  • it's not too much different from patching URLLib3Session because URLLib3Session did little to massage the response of urllib3
  • as commented in the docstring, URLLib3Session is equivalent to HTTPAdapter in requests, their send method are comparable

For a longer term solution, I'd prefer @garyd203's option 2 as it requires less engineering effort from both botocore and moto. It would be something like wsgi application that moto can return a response of its own.

[joguSD]

@luyun-aa As you found out, the new classes in botocore are similar to requests but not quite the same so I don't think switching the patch target will work completely. If the short term solution is just "switch the patch target" then using the deprecated decorators will work.

Also, to reiterate, I strongly suggest not implementing moto's new mechanism by patching botocore's HTTP client library as we cannot guarantee that urllib3 (or even httplib) will be used to carry out all HTTP requests in the future.

As an update, I've been playing with @garyd203's option 2 and have a proof of concept that mostly works. A few issues I'm running into are:

• It's difficult to be as dynamic as patching is in terms of when the mock is enabled / disabled and when clients are created. With the event approach, clients created before the mock is enabled are unaffected.
• The InstanceMetadataFetcher and ContainerMetadataFetcher were previously implemented using requests but now use botocore's URLLib3Session, meaning you won't pick up on the mocked credentials if you were relying on that. And the way things are currently structured it would be fairly difficult to add an event to stub these requests in similar manner.

[bernardgardner]

The fix applied in moto 1.3.5 appears to be incomplete - when using Pipenv on a clean environment to install moto, dependency resolution fails, because moto wants a version of botocore < 1.11 while the current version of boto3 (1.9.4) has a requirement for botocore >= 1.12.4 (so you still have to pin boto3 < 1.8 - which wanted botocore 1.11.0)

I think you probably need to also pin boto3 in your install_requires.

Steps to repro:

• Install pipenv
• mkdir moto-repro && cd moto-repro
• pipenv install moto # installs packages and errors out trying to generate Pipfile.lock
• pipenv graph # shows installed versions and dependency tree

Note pipenv install 'boto3<1.8.0' moto works

This might (also) be considered a failure in pipenv's dependency resolution but at the moment, moto isn't installable using pipenv without doing a bunch of troubleshooting

[yan12125]

@bernardgardner That is indeed a moto issue and a fix is proposed in https://github.com/spulec/moto/pull/1801

[4lph4-Ph4un]

What's the current situation with this? :) The long term solution, to be exact!

[joguSD]

@4lph4-Ph4un I've opened a PR that begins the work to move moto off of patching libraries to implement the core functionality and instead use the before-send event we recently added in botocore. There's still a few kinks that need to be worked out, however.

@spulec Have you had a chance to look at #1847?

[4lph4-Ph4un]

@joguSD, you have no idea how happy I'm to hear that it's been worked on! 👍 😄

[lhufnagel]

It looks like one can get the test-suite to pass when mocking out the responses calls (like done before), on top of @joguSD's fixes https://github.com/joguSD/moto/pull/1

[MartinThoma]

What is the currently recommended solution for this problem?

I have the following installed:

boto==2.48.0
boto3==1.9.27
botocore==1.12.27
moto==1.3.6

and still mock_s3 is not mocking s3:

>           raise error_class(parsed_response, operation_name)
E           botocore.exceptions.ClientError: An error occurred (InvalidAccessKeyId) when calling the CreateBucket operation: The AWS Access Key Id you provided does not exist in our records.

/home/math/.local/lib/python3.5/site-packages/botocore/client.py:623: ClientError

If I set the version of boto3 to 1.7.84, I get:

E           requests.exceptions.ConnectionError: Connection refused: POST https://sts.amazonaws.com/

/home/math/.local/lib/python3.5/site-packages/responses.py:543: ConnectionError

which looks similar to https://github.com/spulec/moto/issues/1026

Other combinations which DONT work for me:

boto==2.49.0
boto3==1.9.27
botocore==1.12.27
moto==1.3.6

boto==2.49.0
boto3==1.7.84
botocore==1.10.84
moto==1.3.6

[danmaas]

This Pip spec works for me at the moment (for SQS - S3 not tested):

boto3>=1.7.84,<1.8
moto>=1.3.4

As mentioned above, it might be necessary to have some AWS credentials in the environment, even if they are just "dummy" ones, e.g.

AWS_ACCESS_KEY_ID=dummy-access-key
AWS_SECRET_ACCESS_KEY=dummy-access-key-secret
AWS_DEFAULT_REGION=us-east-1

[mbelang]

Same here with

boto==2.49.0
boto3==1.7.84
botocore==1.10.84
moto==1.3.6

Have to set environment variables for AWS otherwise it doesn't work.

[spulec]

This is fixed with https://github.com/spulec/moto/pull/1907

[gdoron]

@spulec Thanks a lot, for the fix and for this amazing package! When can we expect it to be on pypi?

[gilbsgilbs]

@gdoron it's shipped with moto 1.3.7 which is already available on PyPi. You might have forgotten to specify dummy credentials in your environment variables (like me). See https://github.com/spulec/moto/pull/1907/files#diff-354f30a63fb0907d4ad57269548329e3R26 .

[mikegrima]

Noticing an issue with PynamoDB (https://github.com/pynamodb/PynamoDB).

It appears to be making use of the botocore vendored requests library: https://github.com/pynamodb/PynamoDB/blob/master/pynamodb/connection/base.py#L19-L20

Unfortunately, with the latest moto I'm getting:

 pynamodb.exceptions.VerboseClientError: An error occurred (UnrecognizedClientException) on request (...) on table (...) when calling the DescribeTable operation: The security token included in the request is invalid.

@joguSD: what happens now for libraries that are still utilizing the vendored requests in botocore? Are these easy fixes?

EDIT: I just tried overriding their Requests session library as documented here: https://github.com/pynamodb/PynamoDB/issues/558, and it now works 👏 .

In the meantime, I am going to submit a PR to PynamoDB that swaps out the vendored Requests for the normal Requests library and hopefully that will work 🤞 .

[alexcasalboni]

Any updates on this?

I've updated to

boto==2.49.0
boto3==1.9.47
botocore==1.12.47
moto==1.3.7

and I am still seeing @mikegrima's error when using @mock_ssm:

ClientError: An error occurred (UnrecognizedClientException) when calling the PutParameter operation: The security token included in the request is invalid.

Plus, I get another error when using @mock_secretsmanager:

ConnectionClosedError: Connection was closed before we received a valid response from endpoint URL: "https://secretsmanager.eu-west-1.amazonaws.com/".

[johnnyplaydrums]

Also checking in on this, I thought the 1.3.7 release fixed this, maybe I'm missing something. Thank you in advance for the help! I have a file test.py and a fresh virtualenv where I just installed moto with pip install moto (1.3.7):

import boto3

from moto import mock_ec2

@mock_ec2
def test_mock_ec2():
    client = boto3.client('ec2', region_name='us-east-1')
    print(client.describe_images()['Images'][0]['ImageId'])

if __name__ == "__main__":
    test_mock_ec2()

Running this gives me:

(venv) ~/sandbox $ python test.py
botocore.exceptions.NoCredentialsError: Unable to locate credential

With fake credentials, I am able to get the correct response:

(venv) ~/sandbox $ AWS_ACCESS_KEY_ID=fake AWS_SECRET_ACCESS_KEY=fake python test.py
ami-03cf127a

Relevant dependencies from my virtualenv:

boto3==1.9.48
botocore==1.12.48
moto==1.3.7

I got confused by some of the discussion above: Is supposed supposed to need fake credentials in order to run? I figured it would mock all that out. Thanks again!

[lhufnagel]

Yeah, currently you need to export fake credentials - I opened https://github.com/spulec/moto/pull/1952 to do that for you, but am hoping someone has a better fix

[johnnyplaydrums]

thanks for the prompt response @lhufnagel. I'll keep my eye on it and lend a hand if I can free up some time.

[jethrolam]

Just reporting that I can still reproduce the error with boto3==1.9.70 and moto==1.3.7 by the following:

import boto3
boto3.client('s3')  # ***By having this line, mock_s3 failed and real S3 was created***
import json
from moto import mock_s3

@mock_s3
def test_mock_s3():
    client = boto3.client('s3', region_name='us-east-1')
    client.create_bucket(Bucket='testbucket')
    response = client.list_buckets()
    print json.dumps(response, default=str)

if __name__ == "__main__":
    test_mock_s3()

This runs on

boto3==1.9.70
botocore==1.12.40
moto==1.3.7

Hope this helps.

[luisdemarchi]

Hello,

Just registering that with all libraries in the latest version Dynamodb in other regions works only with @mock_dynamodb2_deprecated

[luisdemarchi]

I discovered what it is, the problem is apparently in Pynamodb, but with the deprecated function it works. Here's my solution for @mock_dynamodb2:

model > config.py

from pynamodb.connection import Connection
_conn = Connection(region='us-west-2')
session_cls = _conn.session_cls

model > user.py

from model.config import session_cls
...

class User(Model):
    class Meta:
        table_name = '{}-user'.format(PROJECT_PREFIX)
        region = 'us-west-2'
        session_cls = session_cls

    user_id = UnicodeAttribute(hash_key=True, default=utils.generate_uuid)

test.py

from model import config
config.session_cls = requests.Session
from moto import mock_sqs, mock_dynamodb2
from model.user import User

@mock_dynamodb2 def setUp(): User.create_table(read_capacity_units=1, write_capacity_units=1, wait=True)

[mikegrima]

@luisdemarchi This might be related to the requests version that is being used by PynamoDB. Can you try testing the fix outlined here: https://github.com/pynamodb/PynamoDB/issues/558 ?

[nebi-frame]

Any solution so far?

[mikegrima]

@nebi-frame Have you tried the solution outlined in pynamodb/PynamoDB#558 (assuming you are talking about the Dynamo issue)

[nebi-frame]

no not tried yet @mikegrima what about u?

[halfdan]

Just wanted to report that the issue @jethrolam posted is still valid: https://github.com/spulec/moto/issues/1793#issuecomment-449475823

This has to do with import order! We fixed all of our issues by upgrading moto to latest master and running from moto import mock_s3 as early as possibly (before running any boto3.client/boto3.resource calls).

import boto3
import json
from moto import mock_s3
boto3.client('s3')  # Moving this line to _after_ the import works. Whatever boto3.client('s3') does internally is creating an instance that is not mocked by moto

@mock_s3
def test_mock_s3():
    client = boto3.client('s3', region_name='us-east-1')
    client.create_bucket(Bucket='testbucket')
    response = client.list_buckets()
    print json.dumps(response, default=str)

if __name__ == "__main__":
    test_mock_s3()

[mikegrima]

@halfdan Just curious, what is the purpose of having a boto3.client in the same file defined outside of the test cases? It would make more sense to me to have the client defined within each unit test itself (of course after the from moto import ... at the top of the file) (or even through a pytest fixture).

[jethrolam]

@mikegrima In my case, the line boto3.client resides in an external dependency foo that I don't have immediate control. So import foo broke mock_s3 silently.

[mikegrima]

@jethrolam Would you be able to import your dependency locally in the unit test first?

I also wonder if you can mock.patch() the boto3.client in your unit test with the moto mocked one.

[jethrolam]

Once we discovered import foo was the culprit, we resolved the problem easily by modifying foo. We did not attempt other solutions.

With that said, I want to highlight that it was the discovery of the culprit (import foo in my case) that is the most difficult.

[chris-erickson]

Can you or anyone highlight some preferred strategies to structuring project with Boto involved (isolating it and consistently importing it or something)? Seems like I never find a structure that doesn't cause me these sorts of troubles at some point.

[mikegrima]

Personally, I always use local imports for the code I want to test in my test functions. This helps to isolate package import issues. I'm not totally sure how "proper Pythonic" it is, but it works really well without giving me any headaches.