acl order incorrect. elasticsearch not finding other cluster nodes

[lmunro]

The ACLs for iptables at least are appended to the bottom of the list. This means a REJECT all rule cancels out the additional rules and nodes cannot find each other.

ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED ACCEPT icmp -- 0.0.0.0/0 0.0.0.0/0 ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 ctstate NEW tcp dpt:22 REJECT all -- 0.0.0.0/0 0.0.0.0/0 reject-with icmp-host-prohibited ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp multiport dports 5959 /* allow syslog entries inbound / ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp multiport dports 5960 / allow lumberjack entries / ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp multiport dports 5961 / allow tcp entries inbound / ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp multiport dports 9200 / allow esearch entries inbound / ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp multiport dports 9300 / allow esearch2 entries inbound / ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp multiport dports 443 / allow nginx SSL entries inbound / ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp multiport dports 80 / allow nginx entries inbound/ ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 multiport dports 5962 / allow udp entries inbound / ACCEPT tcp -- 10.181.166.219 0.0.0.0/0 tcp multiport dports 9300 / allow ES host 10.181.166.219 to connect / ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 /_ allow services on loopback to talk to any interface */

[martinb3]

I think this is an issue with the wrapper/caller cookbook. It shouldn't apply a reject/drop rule until it runs elkstack. I don't see anything in elkstack itself that is applying REJECT rules.

[lmunro]

Yes indeed, not really a elkstack bug, but something we can still help avoid. The REJECT i think was from the cloud image by default. So we can just put a position 1 for all the added permits and it should be fine for this type of scenario.

[martinb3]

So we can just put a position 1

On every chef run, chef will see that each rule is not in position 1, and remove and re-add it. This will eventually create hundreds of the same rule. It uses iptables' built in position values, so it's not really 'relative positioning'. In short, you have to get the relative order of the rule resources correct, to get the final relative order correct.

[jujugrrr]

The REJECT i think was from the cloud image by default

It might come from vagrant as well, when I was testing ELKstack in the cloud I noticed some rules, and they were coming from Vagrant set up itself

[lmunro]

On every chef run, chef will see that each rule is not in position 1, and remove and re-add it. This will eventually create hundreds of the same rule. It uses iptables' built in position values, so it's not really 'relative positioning'. In short, you have to get the relative order of the rule resources correct, to get the final relative order correct.

I see, not so simple to fix then :(

It might come from vagrant as well

I don't get it from vagrant, but definitely get it bootstrapping a cloud node node manually with knife rackspace. It's coming from the Rakspace cloud image in this case.

[lmunro]

@jujugrrr made a good point. All firewall rules on a chef node should be managed by Chef, so you should really handle it in your recipe and flush any rules that already exist on your first Chef run. Then add in the rules you need.