Security groups in facet misbehaving

[hustonhoburg]

The following definition results in an instance with the sparky and sparky-web security groups, but no ssh security group.

Ironfan.cluster "sparky" do
    cloud(:ec2).security_group(:ssh).authorize_port_range 22..22
    facet :web do
        cloud(:ec2).security_group("sparky-web").authorize_port_range(80)
    end
end

However, removing the security group definition from the web facet, creates an instance with the ssh group as well as the sparky and sparky-web groups. Facet security group definitions seem to be clobbering those defined in the cluster?

[nickmarden]

+1 on the critical tag for this one :-)

[meekmichael]

I think this is somewhat related, the first time we launch a server that creates a new security group, we get this sort of error:

  WARNING: Error running #<Ironfan::Broker::Computer(server=#<Ironfan::Dsl::Server(name="0", run_list_items=c{ xfs, role[volumes], role[gsfn_baseline], role[gsfn_dlb], role[eastdb_cluster], role[eastdb_dlb] }, clouds=c{ ec2 }, volumes=c{  }, environment=:drdev, cluster_role=#<Ironfan::Dsl::Role>, facet_role=#<Ironfan::Dsl::Role>, cluster_name="eastdb", facet_name="dlb")>, resources=c{ client, node, role__eastdb_cluster, role__eastdb_dlb, key_pair }, drives=c{ root, ephemeral0 }, providers=c{ chef, iaas })>:
  WARNING: undefined method `name' for nil:NilClass
  ERROR: undefined method `name' for nil:NilClass (NoMethodError)
  /Users/mmittelstadt/.rvm/gems/ruby-1.9.3-p194@ironfan-homebase/bundler/gems/ironfan-f6623865a712/lib/ironfan/provider/ec2/security_group.rb:27:in `load!'
  /Users/mmittelstadt/.rvm/gems/ruby-1.9.3-p194@ironfan-homebase/bundler/gems/ironfan-f6623865a712/lib/ironfan/provider/ec2/security_group.rb:70:in `create!'
  /Users/mmittelstadt/.rvm/gems/ruby-1.9.3-p194@ironfan-homebase/bundler/gems/ironfan-f6623865a712/lib/ironfan/broker/computer.rb:119:in `block in ensure_dependencies' 
  /Users/mmittelstadt/.rvm/gems/ruby-1.9.3-p194@ironfan-homebase/bundler/gems/ironfan-f6623865a712/lib/ironfan/broker/computer.rb:117:in `each'
  /Users/mmittelstadt/.rvm/gems/ruby-1.9.3-p194@ironfan-homebase/bundler/gems/ironfan-f6623865a712/lib/ironfan/broker/computer.rb:117:in `ensure_dependencies'
  /Users/mmittelstadt/.rvm/gems/ruby-1.9.3-p194@ironfan-homebase/bundler/gems/ironfan-f6623865a712/lib/ironfan/broker/computer.rb:75:in `launch'
  /Users/mmittelstadt/.rvm/gems/ruby-1.9.3-p194@ironfan-homebase/bundler/gems/ironfan-f6623865a712/lib/ironfan/broker/computer.rb:291:in `block in launch'
  /Users/mmittelstadt/.rvm/gems/ruby-1.9.3-p194@ironfan-homebase/bundler/gems/ironfan-f6623865a712/lib/ironfan.rb:35:in `block (3 levels) in parallel'
  /Users/mmittelstadt/.rvm/gems/ruby-1.9.3-p194@ironfan-homebase/bundler/gems/ironfan-f6623865a712/lib/ironfan.rb:131:in `safely'
  /Users/mmittelstadt/.rvm/gems/ruby-1.9.3-p194@ironfan-homebase/bundler/gems/ironfan-f6623865a712/lib/ironfan.rb:34:in `block (2 levels) in parallel'

[nickmarden]

Actually @meekmichael, I think the stack trace you've reported is unrelated. The bug you're talking about can probably be resolved with something like this:

diff --git a/lib/ironfan/provider/ec2/security_group.rb b/lib/ironfan/provider/ec2/security_group.rb
index f6929e1..0b3c68b 100644
--- a/lib/ironfan/provider/ec2/security_group.rb
+++ b/lib/ironfan/provider/ec2/security_group.rb
@@ -24,7 +24,7 @@ module Ironfan
         # Discovery
         #
         def self.load!(cluster=nil)
-          Ironfan.substep(cluster.name, "security groups")
+          Ironfan.substep(cluster && cluster.name, "security groups")

           Ec2.connection.security_groups.each do |raw|
             next if raw.blank?

[nickmarden]

I'm hot on the trail of this bug, but I could use some help from @mrflip or @temujin9 because the root cause appears to be deep in the bowels of the Ironfan code related to cluster resolution. Specifically if you expand the example above to a two-facet cluster:

Ironfan.cluster 'tiny' do
  cloud(:ec2) do
    flavor              'm1.small'
    security_group(:ssh).authorize_port_range(22)
    mount_ephemerals
  end

  facet :web do
    instances     1
    cloud(:ec2) do
      flavor 't1.micro'
      security_group(:web).authorize_port_range(80)
    end
  end

  facet :mysql do
    instances 1
    cloud(:ec2) do
      flavor 'm1.xlarge'
      security_group(:mysql).authorize_port_range(3306)
    end
  end
end

then a few curious things happen:

1. The flavor of the last facet in the file overrides the flavor of all other facets in the file

2. The security group(s) in the last facet in the file overrides the security groups of all other facets in the file, and overrides the explicit security groups of the cluster-level cloud(:ec2) declaration:

   Inventorying servers in tiny cluster, all facets, all servers tiny: Loading chef tiny: Loading ec2 tiny: Reconciling DSL and provider information +--------------+-------+-------------+-----------+------------+----------+ | Name | Chef? | State | Flavor | AZ | Env | +--------------+-------+-------------+-----------+------------+----------+ | tiny-web-0 | no | not running | m1.xlarge | us-east-1d | _default | | tiny-mysql-0 | no | not running | m1.xlarge | us-east-1d | _default | +--------------+-------+-------------+-----------+------------+----------+

This is happening in the call in cluster.resolve here:

https://github.com/infochimps-labs/ironfan/blob/master/lib/ironfan/broker.rb#L16

Specifically, the actual corruption appears in

https://github.com/infochimps-labs/ironfan/blob/master/lib/gorillib/resolution.rb#L47

at the moment when field_names == :facets. At that moment, the read_resolved_attribute call causes some internal pollution effects which can be noticed when

Ironfan.load_cluster('sparky').cloud(:ec2)

suddenly takes on the same "flavor" and "security_groups" attribute values that one would find in

Ironfan.load_cluster('sparky').facets[:mysql].cloud(:ec2)

Any help here would be greatly appreciated!

[fractaloop]

Specs added for this issue when we get it nailed down. https://github.com/infochimps-labs/ironfan/commit/0ca688489754276fa272c4bfec5f27387b1f4140

[nickmarden]

Sorry for the reference noise! I need to look at my github settings to see how to prevent mentions in my own github account from appearing in the Infochimps issue tracking history.

@hustonhoburg, do you mind letting me know if my pull request fixes the issue you're having?

[mrflip]

note: https://github.com/infochimps-labs/ironfan/tree/cloud_merge_borkage has the code from #175 merged on top of my recent changes

Nick -- I don't know if saying 'WIP' in the commit does anything for that, but I know that github does understand it to be 'work in progress'. I also know a force push will not create duplicates (only the latest will show) but that of course requires nobody else noticing the branch you're forcing to.

[temujin9]

@nickmarden FYI, we tested this internally, and it fixes the issue. Thanks again.