Current AWS Boto version doesn't support S3 v4 Signatures causing failing remote env

anateus commented 7 years ago

Context

I'm not certain under what conditions S3 requires v4 signatures, but that's the direction things are moving. Zappa currently uses the AWS-provided boto and botocore packages which default to using HMACv1 signatures. There appears to be no way to instruct those packages to use s3v4 as the signature method.

Expected Behavior

Uploading a JSON file to s3 and setting its path to the remote_env field in zappa_settings.json should result in the env vars being loaded as if they were provided in the environment_variables field.

Actual Behavior

The app deploys fine, but then fails to load. In the logs we can see that accessing the remote json file fails due to signature error. The presence of Calculating signature using hmacv1 auth. in the logs indicates it's not using the v4 signature method.

Possible Fix

The botocore changlog indicates that starting from version 1.5.71, the v4 signature method becomes the default. AWS provides older versions for both Python 2.7 and 3.6.

Was able to fix this by manually modifying zappa/cli.py to remove boto3 and botocore from the list of excluded packages and having the latest versions of both in the virtual environment when the app gets deployed.

Thus, a proposed fix is to add configuration parameter that allows overriding the default excludes.

Steps to Reproduce

As mentioned above, v4 seems to be the default signature method now, but some regions may allow older ones as well. If the following steps aren't reproducible this may be why.

Place a remote_env.json file on S3
Add it zappa_settings.json
Deploy the app
Attempt to load the app

Your Environment

Zappa version used: 0.42.2
Operating System and Python version: OS X 11.6 Python 3.6.0

The output of pip freeze:

alembic==0.9.2
argcomplete==1.8.2
base58==0.2.4
bcrypt==3.1.3
boto3==1.4.4
botocore==1.5.75
certifi==2017.4.17
cffi==1.10.0
chardet==3.0.4
click==6.7
dataset==0.8.0
docutils==0.13.1
durationpy==0.4
Flask==0.12
Flask-Caching==1.2.0
Flask-Migrate==2.0.3
Flask-S3==0.3.3
Flask-Script==2.0.5
Flask-Session==0.3.1
Flask-SQLAlchemy==2.2
future==0.16.0
futures==3.1.1
gunicorn==19.7.0
hjson==2.0.7
idna==2.5
itsdangerous==0.24
Jinja2==2.9.6
jmespath==0.9.3
kappa==0.6.0
lambda-packages==0.15.1
Mako==1.0.6
MarkupSafe==1.0
mistune==0.7.4
netaddr==0.7.19
normality==0.4.4
passlib==1.7.1
placebo==0.8.1
psycopg2==2.7.1
pycparser==2.17
PyMySQL==0.7.10
python-dateutil==2.6.0
python-editor==1.0.3
python-slugify==1.2.4
PyYAML==3.12
redis==2.10.5
requests==2.18.1
s3transfer==0.1.10
six==1.10.0
SQLAlchemy==1.1.6
SQLAlchemy-Utils==0.32.12
toml==0.9.2
tqdm==4.14.0
troposphere==1.9.4
Unidecode==0.4.20
urllib3==1.21.1
Werkzeug==0.12
wsgi-request-logger==0.4.6
zappa==0.42.2

Your zappa_settings.py:

{
"dev": {
    "project_name": "zappa-foo",
    "app_function": "Foo.__init__.app",
    "profile_name": "zappafoo",
    "s3_bucket": "zappa-foo-zappa-bucket",
    "timeout_seconds": 60,
    "remote_env": "s3://zappa-foo-zappa-bucket/remote_env.json",
    "aws_region": "us-east-1",
    "cloudwatch_log_level": "INFO",
    "keep_warm": false,
    "vpc_config": {
        "SubnetIds": [
            ...
        ],
        "SecurityGroupIds": ["..."] // Lambda security group
    }
}
}

Miserlou commented 7 years ago

I'll bet you a dollar this only occurs in us-east-1, if you want to try in another region..

Miserlou commented 7 years ago

Suggest we add an optional excludes_default to replace/override the internal excludes.

Miserlou commented 7 years ago

Related is to bump the version of boto the Zappa ships with.

ghost commented 7 years ago

"I'm not certain under what conditions S3 requires v4 signatures"

I've had this error when trying to pass headers dict to boto.S3.Key.set_contents_from_filename() The headers were s3 KMS encryption related e.g. {x-amz-server-side-encryption: "kms", ...} and the error returned is "Requests specifying Server Side Encryption with AWS KMS managed keys require AWS Signature Version 4" I'm using ap regions not us.

mjgorman commented 6 years ago

The work around did not work for me. This is really blocking me from properly deploying my app. Any other suggestions?

What i am seeing when i don't rely on the values for important things, not sure if it is related:

[1515611587391] [DEBUG] 2018-01-10T19:13:07.391Z 4a48f26f-f63a-11e7-a554-e5677c554d11 The s3 config key is not a dictionary type, ign
oring its value of: None

Miserlou / Zappa