search

Miserlou / Zappa

Serverless Python
https://blog.zappa.io/
MIT License
11.89k stars 1.2k forks source link

AWS permission Zappa execution role in example code #1194

Open robomotic opened 6 years ago

robomotic commented 6 years ago

Context

Running the examples from the Zappa repository. Aws credentials are configured as full administrator user.

Expected Behavior

When running the zappa deploy for all the entries: dev_api, dev_event, production no extra configuration is necessary.

Actual Behavior

When I do zappa deploy dev_event, the following error occurs:

(botocore 1.7.35 (/Users/robomotic/.pyenv/versions/AwsFlask/lib/python3.6/site-packages), Requirement.parse('botocore==1.7.14'), {'zappa'}) Calling deploy for stage dev_event.. Creating dev-event-zappa-test-flask-app-dev-event-ZappaLambdaExecutionRole IAM Role.. Error: Failed to manage IAM roles! You may lack the necessary AWS permissions to automatically manage a Zappa execution role. To fix this, see here: https://github.com/Miserlou/Zappa#using-custom-aws-iam-roles-and-policies

Possible Fix

Not sure!

Steps to Reproduce

Git clone Zappa main branch, cd into examples, configure aws with full permission, run the following: zappa deploy dev_api zappa deploy production zappa deploy dev_event

This is when it fails.

Your Environment

{ "dev_event": { "project_name": "dev_event_zappa_test_flask_app", "keep_warm": false, "debug": true, "log_level": "DEBUG", "events": [{ "function": "mymodule.myfunc", "expression": "rate(1 minute)" }], "aws_region": "eu-west-2", "s3_bucket": "zappa-ciao-dev-us-west-2", "lambda_handler": "app.lambda_handler", "use_apigateway": false, "http_methods": ["GET"], "parameter_depth": 1, "timeout_seconds": 300, "memory_size": 128, "use_precompiled_packages": true }, "dev_api": { "project_name": "dev_api_zappa_test_flask_app", "keep_warm": false, "debug": true, "log_level": "DEBUG", "aws_region": "eu-west-2", "s3_bucket": "zappa-ciao-dev-us-west-2", "app_function": "app.app", "http_methods": ["GET"], "parameter_depth": 1, "timeout_seconds": 300, "memory_size": 128, "use_precompiled_packages": true }, "prod": { "project_name": "dev_zappa_test_flask_app", "events": [{ "function": "mymodule.myfunc", "expression": "rate(5 minutes)" }], "authorizer": { "function": "authmodule.lambda_handler" }, "aws_region": "eu-west-2", "s3_bucket": "zappa-test-bucket", "app_function": "app.app", "http_methods": ["GET"], "parameter_depth": 1, "memory_size": 512, "use_precompiled_packages": true } }

bxm156 commented 6 years ago

Out of curiosity, does switching to us-west-2 work?

It works for me when using Full Admin privileges. The only other thing that comes to my mind right now is maybe its using different credentials than your expecting, or maybe you have multiple creds defined in ~/.aws/credentials?

If you do have multiple credentials defined, you can give them a profile name and specify "profile_name" in zappa_settings.

robomotic commented 6 years ago

Hello, thanks for looking into this. I have one set of credentials and they are full admin privileges. I have switched to "aws_region": "us-west-2" and the issue is still there. What is really odd is that the other dev_api and production works, is there a way to create the privilege manually from the AWS dashboard or get more info about the problem?

Cheers.

robomotic commented 6 years ago

I have also deleted all previous IAM roles, lambdas, buckets and created a new full administrator account this is the Role dump: { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "", "Resource": "" } ] }

robomotic commented 6 years ago

I got a bit further by using the same project name so that it will reuse the rule generated by dev_api:


    "dev_event": {
       "project_name": "dev_api_zappa_test_flask_app",
       "keep_warm": false,
       "debug": true,
       "log_level": "DEBUG",
       "events": [{
         "function": "mymodule.myfunc",
         "expression": "rate(1 minute)"
         }],
       "aws_region": "us-west-2",
       "s3_bucket": "zappa-ciao-dev-us-west-2",
       "lambda_handler": "app.lambda_handler",
       "use_apigateway": false,
       "http_methods": ["GET"],
       "parameter_depth": 1,
       "timeout_seconds": 300,
       "memory_size": 128,
       "use_precompiled_packages": true
    },
    "dev_api": {
       "project_name": "dev_api_zappa_test_flask_app",
       "keep_warm": false,
       "debug": true,
       "log_level": "DEBUG",
       "aws_region": "eu-west-2",
       "s3_bucket": "zappa-ciao-dev-us-west-2",
       "app_function": "app.app",
       "http_methods": ["GET"],
       "parameter_depth": 1,
       "timeout_seconds": 300,
       "memory_size": 128,
       "use_precompiled_packages": true
    },

However I now get another error: 

(AwsFlask) PaoloMacbook:example robomotic$ zappa deploy dev_event
(botocore 1.7.35 (/Users/robomotic/.pyenv/versions/AwsFlask/lib/python3.6/site-packages), Requirement.parse('botocore==1.7.14'), {'zappa'})
Calling deploy for stage dev_event..
Creating dev-api-zappa-test-flask-app-dev-event-ZappaLambdaExecutionRole IAM Role..
Creating zappa-permissions policy on dev-api-zappa-test-flask-app-dev-event-ZappaLambdaExecutionRole IAM Role.
Downloading and installing dependencies..
 - psycopg2==2.7.3.1: Using locally cached manylinux wheel
Packaging project as zip.
Uploading dev-api-zappa-test-flask-app-dev-event-1509013080.zip (15.6MiB)..
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 16.4M/16.4M [00:04<00:00, 3.75MB/s]
Oh no! An error occurred! :(

==============

Traceback (most recent call last):
  File "/Users/robomotic/.pyenv/versions/AwsFlask/lib/python3.6/site-packages/zappa/cli.py", line 718, in deploy
    function_name=self.lambda_name)
  File "/Users/robomotic/.pyenv/versions/AwsFlask/lib/python3.6/site-packages/zappa/core.py", line 1106, in get_lambda_function
    FunctionName=function_name)
  File "/Users/robomotic/.pyenv/versions/AwsFlask/lib/python3.6/site-packages/botocore/client.py", line 312, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/Users/robomotic/.pyenv/versions/AwsFlask/lib/python3.6/site-packages/botocore/client.py", line 605, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.errorfactory.ResourceNotFoundException: An error occurred (ResourceNotFoundException) when calling the GetFunction operation: Function not found: arn:aws:lambda:us-west-2:017092884887:function:dev-api-zappa-test-flask-app-dev-event

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/robomotic/.pyenv/versions/AwsFlask/lib/python3.6/site-packages/zappa/cli.py", line 2525, in handle
    sys.exit(cli.handle())
  File "/Users/robomotic/.pyenv/versions/AwsFlask/lib/python3.6/site-packages/zappa/cli.py", line 484, in handle
    self.dispatch_command(self.command, stage)
  File "/Users/robomotic/.pyenv/versions/AwsFlask/lib/python3.6/site-packages/zappa/cli.py", line 518, in dispatch_command
    self.deploy()
  File "/Users/robomotic/.pyenv/versions/AwsFlask/lib/python3.6/site-packages/zappa/cli.py", line 734, in deploy
    aws_kms_key_arn=self.aws_kms_key_arn
  File "/Users/robomotic/.pyenv/versions/AwsFlask/lib/python3.6/site-packages/zappa/core.py", line 972, in create_lambda_function
    'Mode': 'Active' if self.xray_tracing else 'PassThrough'
  File "/Users/robomotic/.pyenv/versions/AwsFlask/lib/python3.6/site-packages/botocore/client.py", line 312, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/Users/robomotic/.pyenv/versions/AwsFlask/lib/python3.6/site-packages/botocore/client.py", line 605, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.errorfactory.InvalidParameterValueException: An error occurred (InvalidParameterValueException) when calling the CreateFunction operation: Error occurred while GetObject. S3 Error Code: PermanentRedirect. S3 Error Message: The bucket is in this region: eu-west-2. Please use this region to retry the request

==============

Need help? Found a bug? Let us know! :D
File bug reports on GitHub here: https://github.com/Miserlou/Zappa
And join our Slack channel here: https://slack.zappa.io
Love!,
 ~ Team Zappa!
robomotic commented 6 years ago

I then tried to set both to EU zones:

    "dev_event": {
       "project_name": "dev_api_zappa_test_flask_app",
       "keep_warm": false,
       "debug": true,
       "log_level": "DEBUG",
       "events": [{
         "function": "mymodule.myfunc",
         "expression": "rate(1 minute)"
         }],
       "aws_region": "eu-west-2",
       "s3_bucket": "zappa-ciao-dev-us-west-2",
       "lambda_handler": "app.lambda_handler",
       "use_apigateway": false,
       "http_methods": ["GET"],
       "parameter_depth": 1,
       "timeout_seconds": 300,
       "memory_size": 128,
       "use_precompiled_packages": true
    },
    "dev_api": {
       "project_name": "dev_api_zappa_test_flask_app",
       "keep_warm": false,
       "debug": true,
       "log_level": "DEBUG",
       "aws_region": "eu-west-2",
       "s3_bucket": "zappa-ciao-dev-us-west-2",
       "app_function": "app.app",
       "http_methods": ["GET"],
       "parameter_depth": 1,
       "timeout_seconds": 300,
       "memory_size": 128,
       "use_precompiled_packages": true
    },

This is the Zappa status:

Status for dev-api-zappa-test-flask-app-dev-event: 
    Lambda Versions:      2
    Lambda Name:          dev-api-zappa-test-flask-app-dev-event
    Lambda ARN:           arn:aws:lambda:eu-west-2:017092884887:function:dev-api-zappa-test-flask-app-dev-event
    Lambda Role ARN:      arn:aws:iam::017092884887:role/dev-api-zappa-test-flask-app-dev-event-ZappaLambdaExecutionRole
    Lambda Handler:       app.lambda_handler
    Lambda Code Size:     16387862
    Lambda Version:       $LATEST
    Lambda Last Modified: 2017-10-26T10:23:26.152+0000
    Lambda Memory Size:   128
    Lambda Timeout:       300
    Lambda Runtime:       python3.6
    Lambda VPC ID:        None
    Invocations (24h):    6
    Errors (24h):         0
    Error Rate (24h):     0.00%
    Num. Event Rules:     1
    Event Rule Name:      dev-api-zappa-test-flask-app-dev-event-mymodule.myfunc
    Event Rule Schedule:  rate(1 minute)
    Event Rule State:     Enabled
    Event Rule ARN:       arn:aws:events:eu-west-2:017092884887:rule/dev-api-zappa-test-flask-app-dev-event-mymodule.myfunc

And

Status for dev-api-zappa-test-flask-app-dev-api: 
    Lambda Versions:      2
    Lambda Name:          dev-api-zappa-test-flask-app-dev-api
    Lambda ARN:           arn:aws:lambda:eu-west-2:017092884887:function:dev-api-zappa-test-flask-app-dev-api
    Lambda Role ARN:      arn:aws:iam::017092884887:role/dev-api-zappa-test-flask-app-dev-api-ZappaLambdaExecutionRole
    Lambda Handler:       handler.lambda_handler
    Lambda Code Size:     16387901
    Lambda Version:       $LATEST
    Lambda Last Modified: 2017-10-26T09:34:04.937+0000
    Lambda Memory Size:   128
    Lambda Timeout:       300
    Lambda Runtime:       python3.6
    Lambda VPC ID:        None
    Invocations (24h):    1
    Errors (24h):         0
    Error Rate (24h):     0.00%
    API Gateway URL:      https://5ze1p3tg50.execute-api.eu-west-2.amazonaws.com/dev_api
    Domain URL:           None Supplied
    Num. Event Rules:     0

However now it does trigger every 1 minutes but it calls the lambda of the dev_api and not dev_event.

Is there a way maybe to create a role manually in AWS? I feel pretty close to solve it.

robomotic commented 6 years ago

Huuu huuuuu I think I what is happening, this is when I try to create the IAM role manually:

aws iam create-role --role-name dev-event-zappa-test-flask-app-dev-event-ZappaLambdaExecutionRole --assume-role-policy-document file://policy.json

An error occurred (ValidationError) when calling the CreateRole operation: 1 validation error detected: Value 'dev-event-zappa-test-flask-app-dev-event-ZappaLambdaExecutionRole' at 'roleName' failed to satisfy constraint: Member must have length less than or equal to 64

And in fact, by reducing the project name to less than 64 it goes beautifully!

Creating event-dev-event-ZappaLambdaExecutionRole IAM Role.. Creating zappa-permissions policy on event-dev-event-ZappaLambdaExecutionRole IAM Role.

Therefore I would suggest:

I think my other issue might be related but let me check. This was tough to spot! Cheers!

jonwils24 commented 6 years ago

I was having the same issue. Shortened the project name and it works now. Good spot @robomotic!

lenards commented 6 years ago

Curious, I'm totally new to Zappa, where is the project-name stored so I can shorten it? I've looked for a .zappa style hidden directory, it's not in "zappa_settings.json" for the example I'm using. When I do an undeploy it seems to remain derived from the original s3 bucket name I gave.