search

steder / aws-snippets

Helpful scripts for working with AWS services
MIT License
1 stars 1 forks source link

aws_sg_recipe.py - fails when modifying groups, #1

Open gyoza opened 9 years ago

gyoza commented 9 years ago

Hi,

I found this python script and we wanted to use it to assist with our automatic deployment of systems and I've found that it works great creating the initial security groups but fails if you try to modify the security group.


Current Rules:
[SecurityGroupRule(ip_protocol=u'tcp', from_port=u'9160', to_port=u'9160', cidr_ip=u'0.0.0.0/0', src_group_name=None),
 SecurityGroupRule(ip_protocol=u'tcp', from_port=u'8983', to_port=u'8983', cidr_ip=u'0.0.0.0/0', src_group_name=None),
 SecurityGroupRule(ip_protocol=u'tcp', from_port=u'8888', to_port=u'8888', cidr_ip=u'0.0.0.0/0', src_group_name=None),
 SecurityGroupRule(ip_protocol=u'tcp', from_port=u'22', to_port=u'22', cidr_ip=u'0.0.0.0/0', src_group_name=None),
 SecurityGroupRule(ip_protocol=u'tcp', from_port=u'7199', to_port=u'7199', cidr_ip=u'0.0.0.0/0', src_group_name=None)]
Authorizing missing rule SecurityGroupRule(ip_protocol='tcp', from_port='1024', to_port='65535', cidr_ip='0.0.0.0/0', src_group_name='Cassandra Cluster')...
Traceback (most recent call last):
  File "./aws_sg_recipe-orig.py", line 132, in <module>
    create_security_groups()
  File "./aws_sg_recipe-orig.py", line 128, in create_security_groups
    update_security_group(c, group, rules)
  File "./aws_sg_recipe-orig.py", line 114, in update_security_group
    authorize(c, group, rule)
  File "./aws_sg_recipe-orig.py", line 73, in authorize
    return modify_sg(c, group, rule, authorize=True)
  File "./aws_sg_recipe-orig.py", line 61, in modify_sg
    src_group=src_group)
  File "/Library/Python/2.7/site-packages/boto/ec2/securitygroup.py", line 203, in authorize
    dry_run=dry_run)
  File "/Library/Python/2.7/site-packages/boto/ec2/connection.py", line 3192, in authorize_security_group
    params, verb='POST')
  File "/Library/Python/2.7/site-packages/boto/connection.py", line 1223, in get_status
    raise self.ResponseError(response.status, response.reason, body)
boto.exception.EC2ResponseError: EC2ResponseError: 400 Bad Request
<?xml version="1.0" encoding="UTF-8"?>
<Response><Errors><Error><Code>InvalidPermission.Duplicate</Code><Message>the specified rule "peer: sg-da510bbf, TCP, from port: 1024, to port: 65535, ALLOW" already exists</Message></Error></Errors></Response>

Any assistance with this matter would be seriously appreciated!!

gyoza commented 9 years ago

Also I noticed that it tries to revoke rules that should be allowed. 

Revoking unexpected rule SecurityGroupRule(ip_protocol=u'tcp', from_port=u'7000', to_port=u'7000', cidr_ip='0.0.0.0/0', src_group_name=None)...
Revoking unexpected rule SecurityGroupRule(ip_protocol=u'tcp', from_port=u'1024', to_port=u'65535', cidr_ip='0.0.0.0/0', src_group_name=None)...
Revoking unexpected rule SecurityGroupRule(ip_protocol=u'tcp', from_port=u'61620', to_port=u'61621', cidr_ip='0.0.0.0/0', src_group_name=None)...
gyoza commented 9 years ago

any chance this could get looked at?

steder commented 9 years ago

If you're running the script as is it should work if executed with power user permissions in EC2 and IAM.

Have you made any modifications to the script? Do you have permissions on these security groups?

I've tested this snippet with my personal account and have not seen any permission error messages or unexpected revocation. On Tue, Feb 24, 2015 at 12:40 PM gyoza notifications@github.com wrote:

any chance this could get looked at?

— Reply to this email directly or view it on GitHub https://github.com/steder/aws-snippets/issues/1#issuecomment-75817838.

gyoza commented 9 years ago

Strange, I got a git copy of the latest revision made a copy, changed port 22 to port 23,

Ran the first copy, no error.

Ran the second copy, received error.

I am using a Administrator IAM user.

Here is a diff of the scripts.

http://i.gyazo.com/172d728ccbfc776fce6becaf579c22f1.png

Error from second run below.

Updating group "Cassandra Cluster"...
Expected Rules:
[SecurityGroupRule(ip_protocol='tcp', from_port='23', to_port='23', cidr_ip='0.0.0.0/0', src_group_name=None),
 SecurityGroupRule(ip_protocol='tcp', from_port='1024', to_port='65535', cidr_ip='0.0.0.0/0', src_group_name='Cassandra Cluster'),
 SecurityGroupRule(ip_protocol='tcp', from_port='7000', to_port='7000', cidr_ip='0.0.0.0/0', src_group_name='Cassandra Cluster'),
 SecurityGroupRule(ip_protocol='tcp', from_port='61620', to_port='61621', cidr_ip='0.0.0.0/0', src_group_name='Cassandra Cluster'),
 SecurityGroupRule(ip_protocol='tcp', from_port='7199', to_port='7199', cidr_ip='0.0.0.0/0', src_group_name=None),
 SecurityGroupRule(ip_protocol='tcp', from_port='8888', to_port='8888', cidr_ip='0.0.0.0/0', src_group_name=None),
 SecurityGroupRule(ip_protocol='tcp', from_port='8983', to_port='8983', cidr_ip='0.0.0.0/0', src_group_name=None),
 SecurityGroupRule(ip_protocol='tcp', from_port='8981', to_port='8982', cidr_ip='0.0.0.0/0', src_group_name=None),
 SecurityGroupRule(ip_protocol='tcp', from_port='666', to_port='667', cidr_ip='0.0.0.0/0', src_group_name=None),
 SecurityGroupRule(ip_protocol='tcp', from_port='9160', to_port='9160', cidr_ip='0.0.0.0/0', src_group_name=None)]
Revoking unexpected rule SecurityGroupRule(ip_protocol=u'tcp', from_port=u'61620', to_port=u'61621', cidr_ip='0.0.0.0/0', src_group_name=None)...
Revoking unexpected rule SecurityGroupRule(ip_protocol=u'tcp', from_port=u'7000', to_port=u'7000', cidr_ip='0.0.0.0/0', src_group_name=None)...
Revoking unexpected rule SecurityGroupRule(ip_protocol=u'tcp', from_port=u'22', to_port=u'22', cidr_ip=u'0.0.0.0/0', src_group_name=None)...
Current Rules:
[SecurityGroupRule(ip_protocol=u'tcp', from_port=u'9160', to_port=u'9160', cidr_ip=u'0.0.0.0/0', src_group_name=None),
 SecurityGroupRule(ip_protocol=u'tcp', from_port=u'8981', to_port=u'8982', cidr_ip=u'0.0.0.0/0', src_group_name=None),
 SecurityGroupRule(ip_protocol=u'tcp', from_port=u'8983', to_port=u'8983', cidr_ip=u'0.0.0.0/0', src_group_name=None),
 SecurityGroupRule(ip_protocol=u'tcp', from_port=u'8888', to_port=u'8888', cidr_ip=u'0.0.0.0/0', src_group_name=None),
 SecurityGroupRule(ip_protocol=u'tcp', from_port=u'7199', to_port=u'7199', cidr_ip=u'0.0.0.0/0', src_group_name=None),
 SecurityGroupRule(ip_protocol=u'tcp', from_port=u'666', to_port=u'667', cidr_ip=u'0.0.0.0/0', src_group_name=None)]
Authorizing missing rule SecurityGroupRule(ip_protocol='tcp', from_port='23', to_port='23', cidr_ip='0.0.0.0/0', src_group_name=None)...
Authorizing missing rule SecurityGroupRule(ip_protocol='tcp', from_port='1024', to_port='65535', cidr_ip='0.0.0.0/0', src_group_name='Cassandra Cluster')...
Traceback (most recent call last):
  File "./aws_sg_recipe-orig-2.py", line 133, in <module>
    create_security_groups()
  File "./aws_sg_recipe-orig-2.py", line 129, in create_security_groups
    update_security_group(c, group, rules)
  File "./aws_sg_recipe-orig-2.py", line 115, in update_security_group
    authorize(c, group, rule)
  File "./aws_sg_recipe-orig-2.py", line 74, in authorize
    return modify_sg(c, group, rule, authorize=True)
  File "./aws_sg_recipe-orig-2.py", line 62, in modify_sg
    src_group=src_group)
  File "/Library/Python/2.7/site-packages/boto/ec2/securitygroup.py", line 203, in authorize
    dry_run=dry_run)
  File "/Library/Python/2.7/site-packages/boto/ec2/connection.py", line 3192, in authorize_security_group
    params, verb='POST')
  File "/Library/Python/2.7/site-packages/boto/connection.py", line 1223, in get_status
    raise self.ResponseError(response.status, response.reason, body)
boto.exception.EC2ResponseError: EC2ResponseError: 400 Bad Request
<?xml version="1.0" encoding="UTF-8"?>
<Response><Errors><Error><Code>InvalidPermission.Duplicate</Code><Message>the specified rule "peer: sg-7bd3fe1e, TCP, from port: 1024, to port: 65535, ALLOW" already exists</Message></Error></Errors> *** remove request ID due to paranoia ***

Its trying to revoke rules that should exist from what I can see. Any help would be super appreciated, I really do not see what I am doing wrong.. heh.

gyoza commented 9 years ago

Did some further testing and found that there are issues with the script, not sure what is causing them but here is a way to reproduce my error.

Mail group

secgroup = [
  #pop3
  SecurityGroupRule("tcp", "110", "110", "0.0.0.0/0", None),
  SecurityGroupRule("tcp", "995", "995", "0.0.0.0/0", None),
  #imap
  SecurityGroupRule("tcp", "143", "143", "0.0.0.0/0", None),
  SecurityGroupRule("tcp", "993", "993", "0.0.0.0/0", None),
  #smtp
  SecurityGroupRule("tcp", "587", "587", "0.0.0.0/0", None),
  SecurityGroupRule("tcp", "465", "465", "0.0.0.0/0", None),
  SecurityGroupRule("tcp", "25", "25", "0.0.0.0/0", None),
  ]

SECURITY_GROUPS = [("CUSTID - MAIL PORTS FROM 0.0.0.0/0", secgroup)]

Run the python script with the above it works. (I am only capturing STDERR)

$ ./build_secgrp.sh -k key -s secret -i derp -az us-west-2 -file groups/mail.py
----------------------------------------------------------------------
AUTOMATE SECURITY GROUP

FILE FOUND! - groups/mail.py
----------------------------------------------------------------------

Change port 25 to port 22.

secgroup = [
  #pop3
  SecurityGroupRule("tcp", "110", "110", "0.0.0.0/0", None),
  SecurityGroupRule("tcp", "995", "995", "0.0.0.0/0", None),
  #imap
  SecurityGroupRule("tcp", "143", "143", "0.0.0.0/0", None),
  SecurityGroupRule("tcp", "993", "993", "0.0.0.0/0", None),
  #smtp
  SecurityGroupRule("tcp", "587", "587", "0.0.0.0/0", None),
  SecurityGroupRule("tcp", "465", "465", "0.0.0.0/0", None),
  SecurityGroupRule("tcp", "22", "22", "0.0.0.0/0", None),
  ]

SECURITY_GROUPS = [("CUSTID - MAIL PORTS FROM 0.0.0.0/0", secgroup)]

Run the script 

$ ./build_secgrp.sh -k key -s secret -i derp -az us-west-2 -file groups/mail.py
----------------------------------------------------------------------
AUTOMATE SECURITY GROUP

FILE FOUND! - groups/mail.py

----------------------------------------------------------------------

Everything worked, made sure port 25 was changed to 22.

Change port 22 back to port 25

$ ./build_secgrp.sh -k key -s secret -i derp -az us-west-2 -file groups/mail.py
----------------------------------------------------------------------
AUTOMATE SECURITY GROUP

FILE FOUND! - groups/mail.py

----------------------------------------------------------------------
    raise self.ResponseError(response.status, response.reason, body)
boto.exception.EC2ResponseError: EC2ResponseError: 400 Bad Request
<Response><Errors><Error><Code>InvalidPermission.Duplicate</Code><Message>the specified rule "peer: 0.0.0.0/0, TCP, from port: 143, to port: 143, ALLOW" already exists</Message></Error></Errors>

Hope this info helps..

steder commented 9 years ago

Day job is preventing me from looking at this now but I'll try to repro next chance I get.

On Mon Mar 02 2015 at 4:49:09 PM gyoza notifications@github.com wrote:

Did some further testing and found that there are issues with the script, not sure what is causing them but here is a way to reproduce my error.

Mail group

secgroup = [

pop3

SecurityGroupRule("tcp", "110", "110", "0.0.0.0/0", None), SecurityGroupRule("tcp", "995", "995", "0.0.0.0/0", None),

imap

SecurityGroupRule("tcp", "143", "143", "0.0.0.0/0", None), SecurityGroupRule("tcp", "993", "993", "0.0.0.0/0", None),

smtp

SecurityGroupRule("tcp", "587", "587", "0.0.0.0/0", None), SecurityGroupRule("tcp", "465", "465", "0.0.0.0/0", None), SecurityGroupRule("tcp", "25", "25", "0.0.0.0/0", None), ]

SECURITY_GROUPS = [("CUSTID - MAIL PORTS FROM 0.0.0.0/0", secgroup)]

Run the python script with the above it works. (I am only capturing STDERR)

$ ./build_secgrp.sh -k key -s secret -i derp -az us-west-2 -file groups/mail.py

AUTOMATE SECURITY GROUP

FILE FOUND! - groups/mail.py

Change port 25 to port 22.

secgroup = [

pop3

SecurityGroupRule("tcp", "110", "110", "0.0.0.0/0", None), SecurityGroupRule("tcp", "995", "995", "0.0.0.0/0", None),

imap

SecurityGroupRule("tcp", "143", "143", "0.0.0.0/0", None), SecurityGroupRule("tcp", "993", "993", "0.0.0.0/0", None),

smtp

SecurityGroupRule("tcp", "587", "587", "0.0.0.0/0", None), SecurityGroupRule("tcp", "465", "465", "0.0.0.0/0", None), SecurityGroupRule("tcp", "22", "22", "0.0.0.0/0", None), ]

SECURITY_GROUPS = [("CUSTID - MAIL PORTS FROM 0.0.0.0/0", secgroup)]

Run the script

$ ./build_secgrp.sh -k key -s secret -i derp -az us-west-2 -file groups/mail.py

AUTOMATE SECURITY GROUP

FILE FOUND! - groups/mail.py

Everything worked, made sure port 25 was changed to 22.

Change port 22 back to port 25

$ ./build_secgrp.sh -k key -s secret -i derp -az us-west-2 -file groups/mail.py

AUTOMATE SECURITY GROUP

FILE FOUND! - groups/mail.py

raise self.ResponseError(response.status, response.reason, body)

boto.exception.EC2ResponseError: EC2ResponseError: 400 Bad Request

InvalidPermission.Duplicatethe specified rule "peer: 0.0.0.0/0, TCP, from port: 143, to port: 143, ALLOW" already exists Hope this info helps.. — Reply to this email directly or view it on GitHub https://github.com/steder/aws-snippets/issues/1#issuecomment-76844344.
gyoza commented 9 years ago

No worries man, I sincerely appreciate any work you do on this. We're getting around the issue now. :) Thanks!