iptables issues with testing

[ianmiell]

I have a seemingly new problem with testing and iptables.

Formerly, downloading the certs was fine, but I spent the weekend tracking down an issue which meant that the 9999 port was getting an ICMP 'no route to host error'.

After many hours unpicking vagrant landrush+dnsmasq (host and VM) and iptables on the host, I realised that switching iptables off on the master1 VM resolved the problem. It seems to be something to do with the DOCKER-ISOLATION chain, but I haven't looked in depth. It might be related to this? https://github.com/moby/moby/issues/29312

Has something changed that I'm not aware of? Is this ringing any bells for anyone?

I have worked around the problem in the tests by switching off iptables every minute until the certs are successfully downloaded and the hosts join ok.

[ianmiell]

@jperville @IshentRas

the better news is that using the recent changes I made to ShutIt to allow pipelining of requests, the cluster test time has been reduced by 1/3rd!

Cluster tests now take about 2 hours, compared to the single-host kitchen tests at 1 hour.

[ianmiell]

On the original issue, it looks like it happens on:

• 1.3.1

but not

• 1.5.0

will check 1.4

[ianmiell]

This appears to be a feature of the process of installation. After Docker is installed some iptables rules are created which prevent the access to :9999. Origin appears to improve matters, after which the certificate download works.

I just needed to be more patient (but I had assumed this was a DNS/landrush problem) I think.

Will re-test, then close.

[ianmiell]

This has recurred in another test.

I had to flush the iptables rules by hand on the first master to progress the build.

The deadlock was: the master1 build was blocked waiting on etcd to start, while master2 was blocked because it couldn't download the certs for etcd.

These were the iptables rules:

# Generated by iptables-save v1.4.21 on Wed Jan 23 09:16:41 2019
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [122592:5783639]
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
COMMIT
# Completed on Wed Jan 23 09:16:41 2019
# Generated by iptables-save v1.4.21 on Wed Jan 23 09:16:41 2019
*nat
:PREROUTING ACCEPT [839:63012]
:INPUT ACCEPT [6:1380]
:OUTPUT ACCEPT [29697:1784060]
:POSTROUTING ACCEPT [30669:1852450]
:DOCKER - [0:0]
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT -d 10.0.2.2/32 -p tcp -m tcp --dport 53 -j DNAT --to-destination 10.0.2.2:10053
-A OUTPUT -d 10.0.2.2/32 -p udp -m udp --dport 53 -j DNAT --to-destination 10.0.2.2:10053
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A DOCKER -i docker0 -j RETURN
COMMIT
# Completed on Wed Jan 23 09:16:41 2019

[ianmiell]

There was no DOCKER-ISOLATION chain this time, so I think this may be different to the problem seen previously. The resolution appears to be the same, however.

[ianmiell]

This may only occur when the certserver == master1 and there exists a master2, as I've not seen this (yet) with other configurations

[ianmiell]

This is the delete command to remove the 'bad' rule - where does it come from?

iptables -D INPUT -j REJECT --reject-with icmp-host-prohibited

[ianmiell]

[root@master1 sysconfig]# vi iptables
[root@master1 sysconfig]# cat !$
cat iptables
# sample configuration for iptables service
# you can edit this manually or use system-config-firewall
# please do not ask us to add additional ports/services to this default configuration
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
COMMIT

What installs this?

[ianmiell]

[root@master1 sysconfig]# repoquery -l iptables-services
/etc/sysconfig/ip6tables
/etc/sysconfig/iptables
/usr/lib/systemd/system/ip6tables.service
/usr/lib/systemd/system/iptables.service
/usr/libexec/initscripts/legacy-actions/ip6tables
/usr/libexec/initscripts/legacy-actions/ip6tables/panic
/usr/libexec/initscripts/legacy-actions/ip6tables/save
/usr/libexec/initscripts/legacy-actions/iptables
/usr/libexec/initscripts/legacy-actions/iptables/panic
/usr/libexec/initscripts/legacy-actions/iptables/save
/usr/libexec/iptables
/usr/libexec/iptables/ip6tables.init
/usr/libexec/iptables/iptables.init

[ianmiell]

iptables cookbook installs that package.

[ianmiell]

Perhaps solution to this is to add a rule that allows 9999 on the certificate server.

[ianmiell]

Hang on - this already exists:

[root@master1 sysconfig]# cat !$
cat /etc/iptables.d/firewall_certificate
-A OS_FIREWALL_ALLOW -m state --state NEW,ESTABLISHED -m comment --comment "OpenShift HTTPD" -m tcp -p tcp --dport 9999 -j ACCEPT

[ianmiell]

iptables needs to be restarted for it to take effect.

[ianmiell]

We also need to open up other ports:

default['cookbook-openshift3']['etcd_client_port'] = '2379'
default['cookbook-openshift3']['etcd_peer_port'] = '2380' default['cookbook-openshift3']['openshift_master_etcd_port'] = '4001'

on all the etcd servers.