Have to reset ALB target group lambda:InvokeFunction permissions on each update

Context

We use Zappa in our Jenkins CI pipelines to deploy/update Lambdas. We're using target groups in ALBs to invoke Lambdas in a hybrid environment where some rules forward requests to Lambdas, and other requests go to EC2 or k8s hosted services. It's entirely internal, and we use ALBs rather than the API Gateway to declaratively manage our hybrid routing rules. We have had to add a step to all our pipelines to reset permissions to allow InvokeFunction from ALB target groups after every update:

aws lambda add-permission --function-name ${lambda-ARN} \
--action lambda:InvokeFunction --statement-id elb1 --principal elasticloadbalancing.amazonaws.com

Is there some configuration we could use to preserve this permission from update to update? Seems related to https://github.com/Miserlou/Zappa/issues/980 , but triggering from ALB Target Groups is somewhat different than the API Gateway.

We're also not using Zappa to configure an ALB ( using an existing ALB configured externally ) so this may also be related to https://github.com/Miserlou/Zappa/issues/1832 . And permissions may get overwritten with the assumption that Zappa would be handling any ALB config.

Expected Behavior

We were expecting the InvokeFunction permissions to be preserved on update.

Actual Behavior

Any existing permissions are overwritten on update

Possible Fix

We may be unaware of an existing configuration which would preserve permissions on update, but if it doesn't exist a flag to preserve existing permissions on update would be desired.

Steps to Reproduce

Deploy a simple zappa lambda (hello world)
Call aws lambda add-permission --function-name ${lambda-ARN} --action lambda:InvokeFunction --statement-id elb1 --principal elasticloadbalancing.amazonaws.com
configure a simple ALB with one target group to trigger the lambda
Test that you can trigger the lambda by calling the endpoint through the ALB
Use zappa update $VERSION_CODE -s settings.json to update the Lambda
Call the endpoint through the ALB and get a 502, and lambda does not trigger
Call the AWS add-permission call from step 2 again
call the endpoint and get a 200 and lambda triggers

Your Environment

Zappa version used: 0.48.2
Operating System and Python version: Python 3.6.4, Ubuntu 16.04

The output of pip freeze:

argcomplete==1.9.3
boto3==1.5.8
botocore==1.8.22
certifi==2019.3.9
cfn-flip==1.2.0
chardet==3.0.4
Click==7.0
cx-Oracle==7.1.3
docopt==0.6.2
docutils==0.14
durationpy==0.5
ecdsa==0.13.2
elasticsearch==7.0.1
et-xmlfile==1.0.1
Flask==1.0.2
Flask-Cors==3.0.6
future==0.16.0
hjson==3.0.1
idna==2.7
itsdangerous==1.1.0
jdcal==1.3
Jinja2==2.10.1
jmespath==0.9.3
kappa==0.6.0
lambda-packages==0.20.0
MarkupSafe==1.1.1
mysql-connector-python==8.0.16
odfpy==1.3.6
openpyxl==2.4.9
placebo==0.9.0
protobuf==3.7.1
pyasn1==0.4.5
PyMySQL==0.8.0
python-dateutil==2.6.1
python-jose==3.0.1
python-slugify==1.2.4
PyYAML==3.13
requests==2.20.0
rsa==4.0
s3transfer==0.1.12
six==1.11.0
tablib==0.12.1
toml==0.10.0
tqdm==4.19.1
troposphere==2.4.6
unicodecsv==0.14.1
Unidecode==1.0.23
urllib3==1.24.3
Werkzeug==0.14
wsgi-request-logger==0.4.6
xlrd==1.1.0
xlwt==1.3.0
zappa==0.48.2

Your zappa_settings.py:

{
"v0": {
    "project_name": "name",
    "lambda_description": "desc",
    "app_function": "APP",
    "memory_size": 3008,
    "slim_handler": false,
    "role_name": "role",
    "role_arn": "REDACTED",
    "keep_warm_expression": "rate(45 minutes)",
    "keep_warm": true,
    "timeout_seconds": 30,
    "touch_path": "/",
    "aws_region": "us-west-2",
    "profile_name": "default",
    "runtime": "python3.6",
    "s3_bucket": "REDACTED",
    "manage_roles": false,
    "delete_local_zip": false,
    "delete_s3_zip": false,
    "exclude": [
        "jenkins_zappa_settings.json",
        "zappa_settings.json",
        "boto3*",
        "botocore*"
    ],
    "cloudwatch_log_level": "INFO",
    "cloudwatch_data_trace": true,
    "vpc_config": {
        "SubnetIds": [
            REDACTED
        ],
        "SecurityGroupIds": [
            "REDACTED"
        ]
    },
    "layers": [
        "arn:aws:lambda:us-west-2:703076520410:layer:oracle-libs-layer:1"
    ],
    "aws_environment_variables": {
        REDACTED
    }
}
}

I got halfway through opening a PR to fix this, but I think we're just going to migrate off Zappa instead. Still, dropping the change I hacked in that fixed this in case someone wants to pick it up. I changed deploy_lambda_alb in core.py to have:

        kwargs = dict(
            Action="lambda:InvokeFunction",
            FunctionName=lambda_name,
            Principal="elasticloadbalancing.amazonaws.com",
            SourceArn=target_group_arn,
            StatementId=lambda_name,
        )
        response = self.lambda_client.add_permission(**kwargs)

        # Register target group to lambda association.
        # https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/elbv2.html#ElasticLoadBalancingv2.Client.register_targets
        kwargs = dict(
            TargetGroupArn=target_group_arn,
            Targets=[{"Id": lambda_arn}],
        )
        response = self.elbv2_client.register_targets(**kwargs)

The key parts being the FunctionName on the add_permission call, and then the Id on the register_targets call. For reasons I don't fully understand, the code already there wasn't adding any permission (confirmed by adding a call to get the policy in there), even though it was returning a 201. With the above change it works. The part I hadn't figured out was how to include Alias in there.

Hope this helps someone.

Miserlou / Zappa