search

getmoto / moto

A library that allows you to easily mock out tests based on AWS infrastructure.
http://docs.getmoto.org/en/latest/
Apache License 2.0
7.57k stars 2.02k forks source link

Can't replay Cognito Steps #6531

Closed leandromonaco closed 10 months ago

leandromonaco commented 1 year ago

I'm using motoserver/moto:4.1.13 on Docker

Reproduction Steps

  1. Start the recording http://localhost:52009/moto-api/recorder/start-recording
  2. Run aws --endpoint-url=http://localhost:52009 cognito-idp create-user-pool --pool-name "LocalDev"
  3. Download the recording http://localhost:52009/moto-api/recorder/download-recording
  4. Copy the content (see content below)
  5. Run http://localhost:52009/moto-api/recorder/upload-recording using the content from step 3 as body
  6. Run http://localhost:52009/moto-api/recorder/replay-recording

Result

The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.

Docker log

Error on request:
2023-07-17 11:38:35 Traceback (most recent call last):
2023-07-17 11:38:35   File "/usr/local/lib/python3.7/site-packages/werkzeug/serving.py", line 333, in run_wsgi
2023-07-17 11:38:35     execute(self.server.app)
2023-07-17 11:38:35   File "/usr/local/lib/python3.7/site-packages/werkzeug/serving.py", line 320, in execute
2023-07-17 11:38:35     application_iter = app(environ, start_response)
2023-07-17 11:38:35   File "/usr/local/lib/python3.7/site-packages/moto/moto_server/werkzeug_app.py", line 250, in __call__
2023-07-17 11:38:35     return backend_app(environ, start_response)
2023-07-17 11:38:35   File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 2552, in __call__
2023-07-17 11:38:35     return self.wsgi_app(environ, start_response)
2023-07-17 11:38:35   File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 2532, in wsgi_app
2023-07-17 11:38:35     response = self.handle_exception(e)
2023-07-17 11:38:35   File "/usr/local/lib/python3.7/site-packages/flask_cors/extension.py", line 176, in wrapped_function
2023-07-17 11:38:35     return cors_after_request(app.make_response(f(*args, **kwargs)))
2023-07-17 11:38:35   File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 2529, in wsgi_app
2023-07-17 11:38:35     response = self.full_dispatch_request()
2023-07-17 11:38:35   File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1825, in full_dispatch_request
2023-07-17 11:38:35     rv = self.handle_user_exception(e)
2023-07-17 11:38:35   File "/usr/local/lib/python3.7/site-packages/flask_cors/extension.py", line 176, in wrapped_function
2023-07-17 11:38:35     return cors_after_request(app.make_response(f(*args, **kwargs)))
2023-07-17 11:38:35   File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1823, in full_dispatch_request
2023-07-17 11:38:35     rv = self.dispatch_request()
2023-07-17 11:38:35   File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1799, in dispatch_request
2023-07-17 11:38:35     return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
2023-07-17 11:38:35   File "/usr/local/lib/python3.7/site-packages/moto/core/utils.py", line 106, in __call__
2023-07-17 11:38:35     result = self.callback(request, request.url, dict(request.headers))
2023-07-17 11:38:35   File "/usr/local/lib/python3.7/site-packages/moto/moto_api/_internal/recorder/responses.py", line 44, in replay_recording
2023-07-17 11:38:35     recorder.replay_recording(target_host=url)
2023-07-17 11:38:35   File "/usr/local/lib/python3.7/site-packages/moto/moto_api/_internal/recorder/models.py", line 122, in replay_recording
2023-07-17 11:38:35     row_loaded = json.loads(row)
2023-07-17 11:38:35   File "/usr/local/lib/python3.7/json/__init__.py", line 348, in loads
2023-07-17 11:38:35     return _default_decoder.decode(s)
2023-07-17 11:38:35   File "/usr/local/lib/python3.7/json/decoder.py", line 337, in decode
2023-07-17 11:38:35     obj, end = self.raw_decode(s, idx=_w(s, 0).end())
2023-07-17 11:38:35   File "/usr/local/lib/python3.7/json/decoder.py", line 353, in raw_decode
2023-07-17 11:38:35     obj, end = self.scan_once(s, idx)
2023-07-17 11:38:35 json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 2 column 1 (char 2)

Recording Content

{
    "headers": {
        "Host": "localhost:52009",
        "Accept-Encoding": "identity",
        "X-Amz-Target": "AWSCognitoIdentityProviderService.CreateUserPool",
        "Content-Type": "application/x-amz-json-1.1",
        "User-Agent": "aws-cli/2.11.3 Python/3.11.2 Windows/10 exe/AMD64 prompt/off command/cognito-idp.create-user-pool",
        "X-Amz-Date": "20230717T013638Z",
        "X-Amz-Security-Token": "IQoJb3JpZ2luX2VjEIj//////////wEaDmFwLXNvdXRoZWFzdC0yIkcwRQIgCGSNlDzUHlvhAQRbh4tokV8o0PbZcUo+eMrVUvdhq84CIQCOWbtafrkDvrGRYpN609w/v5DdcQvIIqJdQIHGEUw3FiqhAwix//////////8BEAMaDDI0MTU3MTkzNjE3NyIMtsl7nYrdM0h5N58sKvUCFzVBI+2t/JTNadzWWkAIlwb+tJdtaeH3uyACeXtyN+7YPdlm3OBTTT83Bk3Vc06FFqFCwp8uKIv3McF8iTxlzl7Lng4s5Qsw7u7HhzJJk/t08IdONwJxxOPQSALJ79cIF9Ix5FJVRAPm/ni3wNq9eMUtLpISxiqGGvUkfLk0q9Dacuw6rmvHPrnJyltWpI8rfwVXVnyaKxGarbsIikL5nr09M30uDevZ3Jd5nUoBwJOrORhevw1WX+L51A9h2czNhz5E3qU9+rpb+ePpjwD0GQGGXgG1AmopG+WBVz2uHt8wunYGe/+b5tnALkZMB/DAkIhZEPFOuPKIatATyU6fhX2hA0xqTTXylk+NxMVt6dY7E/vwVCclqVAP+PAyZEDKXeD+JxSgqNEy5L+NBIB2CTiE0mXZVYpwyyuetrZFzI1MlVtdXZa0HecQrrrbunY9UYENxIxg8s+Y1iAYFk6kyq2OZGPco0gSBDsCL9kh6C3uPBLsbjC82qqjBjqmAd6ysVeSFFuTVEa6clL6flwuY/T968WOJ0ZcqjUi2uuOj06R9TZuZUrOZnP6WvbDE/S0w7sQ8gl2VxxrXP/n/zJFsgSJ3aocEBg1yCwd6lwFPx+O74PRUtLLVXuPdl26fWAy/JufPAQpjysEro/jMkW+dXGvNx25Eplid1hQDQfI6POOiH1oIl5COafPoj0fZ24du5bgHI7f+LOlIy5SQwjepmI+d7o=",
        "Authorization": "AWS4-HMAC-SHA256 Credential=ASIATQPWPG6YZOY6SSXV/20230717/ap-southeast-2/cognito-idp/aws4_request, SignedHeaders=content-type;host;x-amz-date;x-amz-security-token;x-amz-target, Signature=305a40050aee707dcf2c0ddbd450b075ba864ef14a7549b0fdf43e4edbdf27a6",
        "Content-Length": "24"
    },
    "method": "POST",
    "url": "http://localhost:52009/",
    "body": "eyJQb29sTmFtZSI6ICJMb2NhbERldiJ9",
    "body_encoded": true
}
bblommers commented 1 year ago

Hi @leandromonaco, I can't reproduce this, I'm afraid. The steps I took:

docker run --expose=5000 motoserver/moto:4.1.13
curl http://172.17.0.2:5000/moto-api/recorder/start-recording
aws --endpoint-url=http://172.17.0.2:5000 cognito-idp create-user-pool --pool-name "LocalDev"
curl http://172.17.0.2:5000/moto-api/recorder/stop-recording
curl http://172.17.0.2:5000/moto-api/recorder/download-recording > recording.txt

I then restarted the Docker container. Uploading the recording to an empty MotoServer works fine:

curl -X POST http://172.17.0.2:5000/moto-api/recorder/upload-recording -T ~/recording.txt
curl http://172.17.0.2:5000/moto-api/recorder/replay-recording
aws --endpoint-url=http://172.17.0.2:5000 cognito-idp list-user-pools --max-result 10

The Recording Content that you posted looks valid JSON as well, so I don't understand where that error comes from.

As a debugging step: what happens when you upload the recording file to Moto, and then immediately download it again? Is the content the same?

leandromonaco commented 1 year ago

Thanks @bblommers, I'll check if there is another condition triggering this issue. I'll get back as soon as I can.

bblommers commented 1 year ago

Hi @leandromonaco, any update on this?

leandromonaco commented 1 year ago

hello @bblommers thanks for following up on this. I'm still facing the issue, but I could not pinpoint any specific reason. The only difference is the port number. I'm not sure if the OS and Docker version could also affect this behavior.

bblommers commented 1 year ago

Are you able to create an open source project with an example, and a CI that demonstrates the failing test, @leandromonaco? I realize that's a bit of work, but that would be the easiest way for us to debug what's going on.

bblommers commented 10 months ago

I'm going to close this for now - happy to have another look if you can show a repo that reproduces this problem.