search

Miserlou / Zappa

Serverless Python
https://blog.zappa.io/
MIT License
11.89k stars 1.2k forks source link

Zappa doesn't work with case insensetive headers. #1188

Open digitaldjango opened 7 years ago

digitaldjango commented 7 years ago

Edit: Figured it out if the header is "content-type" zappa doesn't pass it to flask. "Content-Type" gets passed. According to the specs headers should be case insensetive.

Context

Request with lowercase content-type header

[1508701474917] [DEBUG] 2017-10-22T19:44:34.917Z 6e53dd5a-b761-11e7-9c1b-affcee69afd7 Zappa Event: {'resource': '/{proxy+}', 'path': '/auth/', 'httpMethod': 'POST', 'headers': {'Accept': 'application/json', 'Accept-Encoding': 'gzip;q=1.0, compress;q=0.5', 'Accept-Language': 'en-NL;q=1.0', 'CloudFront-Forwarded-Proto': 'https', 'CloudFront-Is-Desktop-Viewer': 'true', 'CloudFront-Is-Mobile-Viewer': 'false', 'CloudFront-Is-SmartTV-Viewer': 'false', 'CloudFront-Is-Tablet-Viewer': 'false', 'CloudFront-Viewer-Country': 'NL', 'content-type': 'application/json', 'Host': 'api.urlyapp.io', 'User-Agent': 'Urly/1.0 (com.esvaru.urlyapp; build:2; iOS 10.3.2) Alamofire/4.5.1', 'Via': '2.0 25d8d373b361f7af9e59da6c842223d0.cloudfront.net (CloudFront)', 'X-Amz-Cf-Id': 'G7kmFCmIMOU_3QdJ3dEDwVhHDCMeQ33hS-m_WjLC4PRVk7EM3k4iEA==', 'X-Amzn-Trace-Id': 'Root=1-59ecf522-084dee24514d78a36554bebe', 'X-Forwarded-For': '86.80.153.197, 216.137.58.67', 'X-Forwarded-Port': '443', 'X-Forwarded-Proto': 'https'}, 'queryStringParameters': None, 'pathParameters': {'proxy': 'auth'}, 'stageVariables': None, 'requestContext': {'path': '/auth/', 'accountId': '338913649781', 'resourceId': 'pkmwck', 'stage': 'production', 'requestId': '6e538fad-b761-11e7-8981-e5887a02dba4', 'identity': {'cognitoIdentityPoolId': None, 'accountId': None, 'cognitoIdentityId': None, 'caller': None, 'apiKey': '', 'sourceIp': '86.80.153.197', 'accessKey': None, 'cognitoAuthenticationType': None, 'cognitoAuthenticationProvider': None, 'userArn': None, 'userAgent': 'Urly/1.0 (com.esvaru.urlyapp; build:2; iOS 10.3.2) Alamofire/4.5.1', 'user': None}, 'resourcePath': '/{proxy+}', 'httpMethod': 'POST', 'apiId': 'pdbjoebwd7'}, 'body': '{"title":"test","type":2,"username":"admin","password":"*******"}', 'isBase64Encoded': False}
[1508701879629] ========= Printing request.headers:
[1508701474918] Content-Length: 63
[1508701474918] Accept: application/json
[1508701474918] Accept-Encoding: gzip;q=1.0, compress;q=0.5
[1508701474918] Accept-Language: en-NL;q=1.0
[1508701474918] Cloudfront-Is-Tablet-Viewer: false
[1508701474918] Cloudfront-Viewer-Country: NL
[1508701474918] Host: api.urlyapp.io
[1508701474918] User-Agent: Urly/1.0 (com.esvaru.urlyapp; build:2; iOS 10.3.2) Alamofire/4.5.1
[1508701474918] Via: 2.0 25d8d373b361f7af9e59da6c842223d0.cloudfront.net (CloudFront)
[1508701474918] X-Amz-Cf-Id: G7kmFCmIMOU_3QdJ3dEDwVhHDCMeQ33hS-m_WjLC4PRVk7EM3k4iEA==
[1508701474918] X-Amzn-Trace-Id: Root=1-59ecf522-084dee24514d78a36554bebe
[1508701474918] X-Forwarded-For: 86.80.153.197, 216.137.58.67
[1508701474918] X-Forwarded-Port: 443
[1508701474918] X-Forwarded-Proto: https
[1508701474918] Cloudfront-Forwarded-Proto: https
[1508701474918] Cloudfront-Is-Desktop-Viewer: true
[1508701474918] Cloudfront-Is-Mobile-Viewer: false
[1508701474918] Cloudfront-Is-Smarttv-Viewer: false
[1508701879629] ========= Printing request.headers.get('Content-type'):
[1508701474918] None
[1508701474918] None
[1508701879629] ========= Printing raw and parsed body:
[1508701474918] b'{"title":"test","type":2,"username":"admin","password":"******"}'
[1508701474918] b'{"title":"test","type":2,"username":"admin","password":"******"}'
[1508701474919] ImmutableMultiDict([])
[1508701474919] None
[1508701474920] None

As you can see the content type header is printed in zappa event but not present in the headers.

Here are the logs for a "seemingly" same request made with a Content-Type header

[1508701879629] [DEBUG] 2017-10-22T19:51:19.614Z 5f8a0bf8-b762-11e7-ad60-29f5fbcdec28 Zappa Event: {'resource': '/{proxy+}', 'path': '/auth/', 'httpMethod': 'POST', 'headers': {'Accept': 'application/json', 'Accept-Encoding': 'gzip, deflate', 'cache-control': 'no-cache', 'CloudFront-Forwarded-Proto': 'https', 'CloudFront-Is-Desktop-Viewer': 'true', 'CloudFront-Is-Mobile-Viewer': 'false', 'CloudFront-Is-SmartTV-Viewer': 'false', 'CloudFront-Is-Tablet-Viewer': 'false', 'CloudFront-Viewer-Country': 'NL', 'Content-Type': 'application/json', 'Host': 'api.urlyapp.io', 'Postman-Token': '529c1777-5861-41cd-8edd-39b1b6ecd3c2', 'User-Agent': 'PostmanRuntime/6.3.2', 'Via': '1.1 48e3cf3ee71856983cda4c8805113c56.cloudfront.net (CloudFront)', 'X-Amz-Cf-Id': 'NjwfT3tNZNMSdt-Vu4L9oImqLkpwO_EFarNwkTA6-T7rGuT3VcCmxQ==', 'X-Amzn-Trace-Id': 'Root=1-59ecf6b7-7a14636508fe1f6d65b89905', 'X-Forwarded-For': '86.80.153.197, 216.137.58.67', 'X-Forwarded-Port': '443', 'X-Forwarded-Proto': 'https'}, 'queryStringParameters': None, 'pathParameters': {'proxy': 'auth'}, 'stageVariables': None, 'requestContext': {'path': '/auth/', 'accountId': '338913649781', 'resourceId': 'pkmwck', 'stage': 'production', 'requestId': '5f83f1f2-b762-11e7-b6b8-b16418e5696d', 'identity': {'cognitoIdentityPoolId': None, 'accountId': None, 'cognitoIdentityId': None, 'caller': None, 'apiKey': '', 'sourceIp': '86.80.153.197', 'accessKey': None, 'cognitoAuthenticationType': None, 'cognitoAuthenticationProvider': None, 'userArn': None, 'userAgent': 'PostmanRuntime/6.3.2', 'user': None}, 'resourcePath': '/{proxy+}', 'httpMethod': 'POST', 'apiId': 'pdbjoebwd7'}, 'body': '{\n    "title": "test",\n    "type": 2,\n    "username": "admin",\n    "password": "*******"\n  }', 'isBase64Encoded': False}
[1508701879629] ========= Printing request.headers:
[1508701879629] Content-Type: application/json
[1508701879629] Content-Length: 108
[1508701879629] Accept: application/json
[1508701879629] Accept-Encoding: gzip, deflate
[1508701879629] Cloudfront-Is-Mobile-Viewer: false
[1508701879629] Cloudfront-Is-Smarttv-Viewer: false
[1508701879629] Cloudfront-Is-Tablet-Viewer: false
[1508701879629] Host: api.urlyapp.io
[1508701879629] Postman-Token: 529c1777-5861-41cd-8edd-39b1b6ecd3c2
[1508701879629] User-Agent: PostmanRuntime/6.3.2
[1508701879629] Via: 1.1 48e3cf3ee71856983cda4c8805113c56.cloudfront.net (CloudFront)
[1508701879629] X-Amz-Cf-Id: NjwfT3tNZNMSdt-Vu4L9oImqLkpwO_EFarNwkTA6-T7rGuT3VcCmxQ==
[1508701879629] X-Amzn-Trace-Id: Root=1-59ecf6b7-7a14636508fe1f6d65b89905
[1508701879629] X-Forwarded-For: 86.80.153.197, 216.137.58.67
[1508701879629] X-Forwarded-Port: 443
[1508701879629] X-Forwarded-Proto: https
[1508701879629] Cache-Control: no-cache
[1508701879629] Cloudfront-Forwarded-Proto: https
[1508701879629] Cloudfront-Is-Desktop-Viewer: true
[1508701879629] Cloudfront-Viewer-Country: NL
[1508701879629] ========= Printing request.headers.get('Content-type'):
[1508701879629] application/json
[1508701879629] application/json
[1508701879629] ========= Printing raw and parsed body:
[1508701879629] b'{"title": "test","type": 2,"username": "admin","password": "*******"}'
[1508701879629] b'{"title": "test","type": 2,"username": "admin","password": "*******"}'
[1508701879629] ImmutableMultiDict([])
[1508701879629] {'title': 'test', 'type': 2, 'username': 'admin', 'password': '*******'}
[1508701879629] {'title': 'test', 'type': 2, 'username': 'admin', 'password': '*******'}

I tried:

Expected Behavior

The content type header should be available and the request body parsed.

Actual Behavior

Content type header is None and the request body is not parsed.

Steps to Reproduce

Works: curl -i -X POST -H "Content-Type: application/x-www-form-urlencoded" -d "password=tester&title=Django%E2%80%99s%20MacBook%20Pro&type=2&username=tester" https://api.urlyapp.io/auth/ Doesn't work: curl -i -X POST -H "content-type: application/x-www-form-urlencoded" -d "password=tester&title=Django%E2%80%99s%20MacBook%20Pro&type=2&username=tester" https://api.urlyapp.io/auth/

Your Environment

bxm156 commented 6 years ago

@Speedrockracer Can you post your settings file? Do you happen to have CORS enabled? Doe the following setting make a difference:

"cors": {"allowed_headers": ["content-type", "Content-Type", "X-Amz-Date", "Authorization", "X-Api-Key", X-Amz-Security-Token"]}
digitaldjango commented 6 years ago

I had cors enabled and tried your suggestion but it didn't help.. Also note that the header in question is visible in the zappa tail call leading me to believe that it is passed correctly to the lambda.

I tried with a new config without cors and minimal settings but the problem still remains. curl: 

curl -i -X POST -H "content-type: application/x-www-form-urlencoded" -d "password=tester&title=Django%E2%80%99s%20MacBook%20Pro&type=2&username=tester" https://1wbn5kovxh.execute-api.eu-west-1.amazonaws.com/temp/auth/

Zappa settings:

"temp": {
        "app_function": "app.__init__.app",
        "aws_region": "eu-west-1",
        "profile_name": "default",
        "s3_bucket": "zappa-pe8osfm9",
        "binary_support": false,
        "memory_size": 1536,
        "extra_permissions": [
            {
                "Effect": "Allow",
                "Action": "s3:PutObject",
                "Resource": "arn:aws:s3:::images.urlyapp.io/media/*"
            },
            {
                "Effect": "Allow",
                "Action": "s3:GetObject",
                "Resource": "arn:aws:s3:::images.urlyapp.io/uploads/*"
            }
        ],
        "vpc_config": {
            "SubnetIds": [*********],
            "SecurityGroupIds": [ ********]
        }
    }
digitaldjango commented 6 years ago

So I checked the source of zappa and it seems the wsgi.py file has some code that should do it. It looks like it should work. I tried some stuff but don't have much experience with zappa or python in general so I didn't get far debugging. Any help would be appriciated :)

Miserlou commented 6 years ago

Hey Speed - is your problem that it's changing? Or that it isn't getting passed through at all? Should application be able to handle any case?

digitaldjango commented 6 years ago

According to the logs the header is coming in to zappa (first line of the log posted above contains json with the headers among other things.). When the header is "Content-Type" flask receives it but when it is "content-type" it's not passed trough to flask at all. The code at around line 85 of wsgi.py should convert the header to "Content-Type" I think but maybe something goes wrong there.

ajw0100 commented 6 years ago

This bit me as well. I think the issue is here. The headers dictionary is modified while it is being iterated over. This causes some unexpected behavior. In particular, some of the headers don't get canonicalized. What makes it even trickier is that it's not deterministic as to which headers get skipped. It depends on the contents of the headers dictionary itself. You can read more about the dangers of modifying dictionaries while iterating over them here.

If the Content-Type header does not get canonicalized then it fails this test and instead of being placed into environ['CONTENT_TYPE'] it gets placed into environ['HTTP_CONTENT_TYPE'] down here where it may go unrecognized by your wsgi app.

Changing the canonicalization code to this seems to fix it for me: https://github.com/ajw0100/Zappa/commit/90acabd8117e15e91de066ca85ff0315f1a98f63

If this works for others I can submit a PR.

digitaldjango commented 6 years ago

I am currently using a temp fix to make my project run.

json = request.get_json(force=True)

I'll have some time to test your commit somewhere next week!

lispmeister commented 6 years ago

Tested patch ajw0100/Zappa@90acabd and it fixes the content-type issue #1188. Tested directly and with python-github-webhook where the content-type bug shows for POST events. Can we get this merged? Do we need a separate PR?

AartGoossens commented 6 years ago

I ran into the same issue (in combination with Apistar). I could not find any information about pending releases for Zappa, is there already one planned that contains the fix by @ajw0100?

bhcopeland commented 4 years ago

Was this issue fixed?

I am running the latest version of Zappa (0.48.2), and Content-Type header is not being received. Any tips what I can do?

Edit: Ah, in wsgi.py you need to send ["POST", "PUT", "PATCH", "DELETE"] for content-type to be set.