Having an issue deploying to AWS account - subnet step

[olearytd]

I have my acli configured to my account and I pass my password with the command

./osmstats.py deploy --name osmstats-deploy --password myPassword

It looks like the deploy script gets past creating the security group and then errors

Gumbiis-MacBook-Pro:deploy toleary$ ./osmstats.py deploy --name osmstats-deploy --password APBZvffsuFfJ 09:29:46: Starting deployment of osmstats-deploy 09:29:46: Fetching latest osm-stats-workers repository 09:29:54: Kinesis stream osmstats-deploy exists 09:29:56: Creating RDS database osmstats-deploy 09:41:43: Migrating database 09:41:54: IAM role osmstats-deploy_lambda exists 09:41:55: Creating osmstats-deploy lambda function 09:41:56: Creating security group osmstats-deploy_lambda Traceback (most recent call last): File "./osmstats.py", line 130, in func = create_function(args.name, zfile, lsize=int(args.lsize), timeout=int(args.ltimeout)) File "/Users/toleary/Documents/git/osm-stats/deploy/boto3utils.py", line 100, in create_function VpcConfig={'SubnetIds': subnets, 'SecurityGroupIds': [group.group_id]} File "/Users/toleary/anaconda/lib/python2.7/site-packages/botocore/client.py", line 258, in _api_call return self._make_api_call(operation_name, kwargs) File "/Users/toleary/anaconda/lib/python2.7/site-packages/botocore/client.py", line 548, in _make_api_call raise ClientError(parsed_response, operation_name) botocore.exceptions.ClientError: An error occurred (InvalidParameterValueException) when calling the CreateFunction operation: Subnets are required to be in the same VPC.

An error occurred (InvalidParameterValueException) when calling the CreateFunction operation: Subnets are required to be in the same VPC.

My VPC has 4 subnets and 1 of them isn't being used so should be free if the RDS database needs its own subnet. I've been debugging it but feel like I'm missing something.

Running on Mac OS X 10.11.5 Python 2.7.11 :: Anaconda 4.0.0 (x86_64)

[kriscarle]

I just encountered this as well. Seems to happen when you have more than one VPC in your account. The call to describe_subnets gets all the subnets in your account so the script ends up passing subnets from multiple VPCs into lamba create_function() http://boto3.readthedocs.io/en/latest/reference/services/ec2.html#EC2.Client.describe_subnets

I got around this by adding updating the code here https://github.com/AmericanRedCross/osm-stats/blob/master/deploy/boto3utils.py#L89 to filter to just the VPC ID that I'm using for osmstats

subnets = [s['SubnetId'] for s in ec2.describe_subnets(
                Filters=[
                    {
                        'Name': 'vpc-id',
                        'Values': ['vpc-12345a']
                    }
                ]
            )['Subnets']]

I keep running into VPC related errors, I think it has something to do with older AWS accounts and what they have enabled by default. Maybe this script works better on a brand new empty AWS account?

[olearytd]

Thanks for getting back to me, I'll try this out first thing and post here! I do have multiple VPCs in my account so hopefully this does he trick

[olearytd]

@kriscarle I set the filter name to the name of my vpc but there is a new error saying that the vpc name is invalid. I tried changing this to the vpc id as well but same error.

Did you just put the name of your vpc in 'Name': 'vpc-id' ?

[kriscarle]

@olearytd

Oh no, you just need to update the values section

subnets = [s['SubnetId'] for s in ec2.describe_subnets(
                Filters=[
                    {
                        'Name': 'vpc-id',
                        'Values': ['PUT_YOUR_VPC_ID_HERE']
                    }
                ]
            )['Subnets']]

I think the correct way to fix this is to look up the default VPC ID automatically using http://boto3.readthedocs.io/en/latest/reference/services/ec2.html#EC2.Client.describe_vpcs

Something like this (haven't tested this code yet)

default_vpc = ec2.describe_vpcs(
    Filters=[
        {
            'Name': 'IsDefault',
            'Values': [
                True
            ]
        }
    ]
)['Vpcs'][0]
subnets = [s['SubnetId'] for s in ec2.describe_subnets(
                Filters=[
                    {
                        'Name': 'vpc-id',
                        'Values': [default_vpc.VpcId]
                    }
                ]
            )['Subnets']]

I also tried to get this working in a custom VPC instead of the default, but that is another issue. I eventually gave up and just setup a brand new AWS account for my project. The script runs a lot cleaner that way.

[olearytd]

Ah that makes more sense, I had my vpc id set, I'll try this out! Thanks :)

[olearytd]

That worked setting it to the default vpc

Gumbiis-MacBook-Pro:deploy toleary$ ./osmstats.py deploy --name osmstats-deploy --password myPassword 08:35:24: Starting deployment of osmstats-deploy 08:35:24: Fetching latest osm-stats-workers repository 08:35:39: Kinesis stream osmstats-deploy exists 08:35:40: RDS Database osmstats-deploy exists 08:35:40: Migrating database 08:35:49: IAM role osmstats-deploy_lambda exists 08:35:51: Creating osmstats-deploy lambda function 08:36:05: Creating security group osmstats-deploy_ec2 08:36:05: Creating IAM role osmstats-deploy_ec2 08:36:10: Creating EC2 instance osmstats-deploy 08:37:42: Deploying to EC2 08:44:11: Completed deployment of osmstats-deploy

I'll probably have the same issue if I need to deploy to another vpc in our account but for now I'll mark this as closed since this is fine for testing. Thanks for all your help @kriscarle !