Launching VPC cluster throws the error: "Exception: Error authorizing cluster ingress to self"

[dhulse]

When I try to launch a cluster into a vpc. I get this error: "Exception: Error authorizing cluster ingress to self"

See https://github.com/nchammas/flintrock/issues/85 for more details

[nchammas]

I wonder if the reason I don't see this issue is because I have a default VPC, and you guys don't.

Will investigate.

[nchammas]

OK, if I create a VPC (i.e. a non-default VPC) and use it, I believe I get the same error you're seeing:

Traceback (most recent call last):
  File ".../flintrock/flintrock/ec2.py", line 323, in get_or_create_ec2_security_groups
    SourceSecurityGroupName=cluster_group.group_name)
  File ".../flintrock/venv/lib/python3.5/site-packages/boto3/resources/factory.py", line 481, in do_action
    response = action(self, *args, **kwargs)
  File ".../flintrock/venv/lib/python3.5/site-packages/boto3/resources/action.py", line 83, in __call__
    response = getattr(parent.meta.client, operation_name)(**params)
  File ".../flintrock/venv/lib/python3.5/site-packages/botocore/client.py", line 228, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File ".../flintrock/venv/lib/python3.5/site-packages/botocore/client.py", line 486, in _make_api_call
    raise ClientError(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (InvalidGroup.NotFound) when calling the AuthorizeSecurityGroupIngress operation: The security group 'flintrock-nick' does not exist in default VPC 'vpc-fd4af498'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File ".../flintrock/venv/bin/flintrock", line 9, in <module>
    load_entry_point('Flintrock', 'console_scripts', 'flintrock')()
  File ".../flintrock/flintrock/flintrock.py", line 803, in main
    cli(obj={})
  File ".../flintrock/venv/lib/python3.5/site-packages/click/core.py", line 716, in __call__
    return self.main(*args, **kwargs)
  File ".../flintrock/venv/lib/python3.5/site-packages/click/core.py", line 696, in main
    rv = self.invoke(ctx)
  File ".../flintrock/venv/lib/python3.5/site-packages/click/core.py", line 1060, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File ".../flintrock/venv/lib/python3.5/site-packages/click/core.py", line 889, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File ".../flintrock/venv/lib/python3.5/site-packages/click/core.py", line 534, in invoke
    return callback(*args, **kwargs)
  File ".../flintrock/venv/lib/python3.5/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File ".../flintrock/flintrock/flintrock.py", line 295, in launch
    instance_initiated_shutdown_behavior=ec2_instance_initiated_shutdown_behavior)
  File ".../flintrock/flintrock/ec2.py", line 30, in wrapper
    res = func(*args, **kwargs)
  File ".../flintrock/flintrock/ec2.py", line 409, in launch
    region=region)
  File ".../flintrock/flintrock/ec2.py", line 326, in get_or_create_ec2_security_groups
    raise Exception("Error authorizing cluster ingress to self.") from e
Exception: Error authorizing cluster ingress to self.

So that's why I haven't seen this issue before--I have a default VPC. (I've sporadically seen an unrelated issue that results in a similar error, so that threw me off at first.)

[nchammas]

Related upstream issue: https://github.com/boto/boto3/issues/158

[nchammas]

@dhulse @marcuscollins I believe I've found the issue and am working on a fix.

In theory, the best workaround until I get a fix out would be to create a VPC and mark it as the default VPC for your account.

Unfortunately, that's not possible at this time:

Q. Can I specify which VPC is my default VPC?

Not at this time.

According to that same FAQ, I'm guessing both of you created your AWS accounts in 2013 or earlier, which is why your accounts don't have default VPCs in the regions you most frequently use.

Another workaround which should actually work is to use Flintrock in a region you've never used before. AWS will automatically create a default VPC for you in that region and Flintrock should work fine. Comment out vpc-id in your config when you do this.

Let me know if this works for you!

[marcuscollins]

​Unfortunately, we already have VPCs in all the regions I have access to... But yes, our account was created prior to 2013. I'll see if I can try it out in some other region.​