Not an issue, just suggestion

[axmetishe]

Hello,

We also use saltstack for service provisioning, and of course we test formulas before, I was wonder that you use bash in testing. Simple suggestion is to use test-kitchen with salt provisioner and serverspec, it solve many problems for us, for example testing on different platform, right now we support at least two platforms centos 7, debian 8, on some formulas we extend list of platform for up to 4 platforms and all integration testing jobs is just a 'kitchen test'.

Short example of kitchen configuration, based on our iptables-formula, which apply rules from predefined roles via list in grains:

---
driver:
  name: docker
  use_sudo: false
  require_chef_omnibus: false
  remove_images: true
  socket: tcp://
  privileged: false
  volume: /sys/fs/cgroup
  cap_add:
    - SYS_ADMIN

provisioner:
  name: salt_solo
  salt_bootstrap_options: 'stable'
  formula: iptables
  grains:
    roles:
      - ssh-server
      - dns-client
      - os-updates
  state_top:
    base:
      '*':
        - iptables
  pillars:
    top.sls:
      base:
        '*':
          - iptables
    iptables.sls:
      ipt:
        ifaces:
          - eth0
        roles:
          dns-client:
            out:
              destination:
                - 0.0.0.0/0
          os-updates:
            out:
              destination:
                - 0.0.0.0/0

platforms:
  - name: debian
    driver_config:
      image: debian:latest
      run_command: /sbin/init
      provision_command:
        - apt-get install -y apt-utils apt-transport-https locales curl tar iptables iproute kmod
        - localedef  --no-archive -c -i en_US -f UTF-8 en_US.UTF-8 &&
          localedef  --no-archive -c -i ru_RU -f UTF-8 ru_RU.UTF-8

  - name: centos
    driver_config:
      image: sperasoft:centos-7-systemd
      run_command: /sbin/init
      provision_command:
        - localedef  --no-archive -c -i en_US -f UTF-8 en_US.UTF-8 &&
          localedef  --no-archive -c -i ru_RU -f UTF-8 ru_RU.UTF-8
        - sed -i -r 's/^(.*pam_nologin.so)/#\1/' /etc/pam.d/sshd
        - yum install -y -q curl tar iptables iproute

suites:
  - name: clean
  - name: ssh-server
    provisioner:
      pillars:
        top.sls:
          base:
            '*':
              - iptables
              - ssh
        ssh.sls:
          ipt:
            roles:
              ssh-server:
                in:
                  source:
                    - 123.45.56.0/26
                    - 78.90.123.0/28
                    - 45.67.89.0/28
                    - 172.17.0.0/16

in this configuration we get 4 containers - 2 with default rules, 2 with allowed ssh from network list

Tests example:

require 'serverspec'
set :backend, :exec

case os[:family]
when 'debian'
  servicename = 'netfilter-persistent'
  servicepkg = 'iptables-persistent'
when 'redhat'
  servicename = 'iptables'
  servicepkg = 'iptables-services'
end

modules = [
  'ip_tables',
  'iptable_filter',
  'nf_conntrack',
  'nf_conntrack_ftp',
  'nf_conntrack_tftp',
]

describe package(servicepkg) do
  it { should be_installed }
end

describe service(servicename) do
  it { should be_enabled }
end

modules.each do |mod|
  describe kernel_module(mod) do
    it { should be_loaded }
  end
end

describe iptables do
  it { should have_rule('-P INPUT DROP') }
  it { should have_rule('-P OUTPUT DROP') }
  it { should have_rule('-P FORWARD DROP') }
  it { should have_rule('-N bad') }
  it { should have_rule('-N bad_tcp') }
  it { should have_rule('-N bad_udp') }
  it { should have_rule('-N bad_icmp') }
  it { should have_rule('-N icmp_in') }
  it { should have_rule('-N icmp_out') }
  it { should have_rule('-N udp_in') }
  it { should have_rule('-N udp_out') }
  it { should have_rule('-N tcp_in') }
  it { should have_rule('-N tcp_out') }
  it { should have_rule('-A INPUT -s 127.0.0.1/32 -i lo -j ACCEPT') }
  it { should have_rule('-A INPUT -p tcp -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT') }
  it { should have_rule('-A INPUT -p udp -m conntrack --ctstate ESTABLISHED -j ACCEPT') }
  it { should have_rule('-A INPUT -m pkttype --pkt-type broadcast -j DROP') }
  it { should have_rule('-A INPUT -m pkttype --pkt-type multicast -j DROP') }
  it { should have_rule('-A INPUT -j bad') }
  it { should have_rule('-A INPUT -i eth0 -p tcp -j tcp_in') }
  it { should have_rule('-A INPUT -i eth0 -p udp -j udp_in') }
  it { should have_rule('-A INPUT -i eth0 -p icmp -j icmp_in') }
  it { should have_rule('-A INPUT -j LOG --log-prefix "IN:DIED:DROP "') }
  it { should have_rule('-A FORWARD -j LOG --log-prefix "FWD:DIED:DROP "') }
  it { should have_rule('-A OUTPUT -s 127.0.0.1/32 -o lo -j ACCEPT') }
  it { should have_rule('-A OUTPUT -p tcp -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT') }
  it { should have_rule('-A OUTPUT -p udp -m conntrack --ctstate ESTABLISHED -j ACCEPT') }
  it { should have_rule('-A OUTPUT -j bad') }
  it { should have_rule('-A OUTPUT -o eth0 -p tcp -j tcp_out') }
  it { should have_rule('-A OUTPUT -o eth0 -p udp -j udp_out') }
  it { should have_rule('-A OUTPUT -o eth0 -p icmp -j icmp_out') }
  it { should have_rule('-A OUTPUT -j LOG --log-prefix "OUT:DIED:DROP "') }
  it { should have_rule('-A bad -m conntrack --ctstate INVALID -j LOG --log-prefix "ALL:invalid:DROP "') }
  it { should have_rule('-A bad -m conntrack --ctstate INVALID -j DROP') }
  it { should have_rule('-A bad -p tcp -m tcp --tcp-flags RST RST -m limit --limit 2/sec --limit-burst 2 -j bad_tcp') }
  it { should have_rule('-A bad -p tcp -m tcp --tcp-flags RST RST -j LOG --log-prefix "TCP:RST_flood:DROP "') }
  it { should have_rule('-A bad -p tcp -m tcp --tcp-flags RST RST -j DROP') }
  it { should have_rule('-A bad -p tcp -j bad_tcp') }
  it { should have_rule('-A bad -p udp -j bad_udp') }
  it { should have_rule('-A bad -p icmp -j bad_icmp') }
  it { should have_rule('-A bad_icmp -p icmp -f -j LOG --log-prefix "ICMP:Fragment:DROP "') }
  it { should have_rule('-A bad_icmp -p icmp -f -j DROP') }
  it { should have_rule('-A bad_tcp -p tcp -m conntrack --ctstate NEW -m tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -j LOG --log-prefix "TCP:NEW_not_SYN:DROP "') }
  it { should have_rule('-A bad_tcp -p tcp -m conntrack --ctstate NEW -m tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -j DROP') }
  it { should have_rule('-A bad_tcp -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG NONE -j LOG --log-prefix "TCP:stealth_scan:DROP "') }
  it { should have_rule('-A bad_tcp -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG NONE -j DROP') }
  it { should have_rule('-A bad_tcp -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG FIN,SYN,RST,PSH,ACK,URG -j LOG --log-prefix "TCP:stealth_scan:DROP "') }
  it { should have_rule('-A bad_tcp -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG FIN,SYN,RST,PSH,ACK,URG -j DROP') }
  it { should have_rule('-A bad_tcp -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG FIN,PSH,URG -j LOG --log-prefix "TCP:stealth_scan:DROP "') }
  it { should have_rule('-A bad_tcp -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG FIN,PSH,URG -j DROP') }
  it { should have_rule('-A bad_tcp -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG FIN,SYN,RST,ACK,URG -j LOG --log-prefix "TCP:stealth_scan:DROP "') }
  it { should have_rule('-A bad_tcp -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG FIN,SYN,RST,ACK,URG -j DROP') }
  it { should have_rule('-A bad_tcp -p tcp -m tcp --tcp-flags SYN,RST SYN,RST -j LOG --log-prefix "TCP:stealth_scan:DROP "') }
  it { should have_rule('-A bad_tcp -p tcp -m tcp --tcp-flags SYN,RST SYN,RST -j DROP') }
  it { should have_rule('-A bad_tcp -p tcp -m tcp --tcp-flags FIN,SYN FIN,SYN -j LOG --log-prefix "TCP:stealth_scan:DROP "') }
  it { should have_rule('-A bad_tcp -p tcp -m tcp --tcp-flags FIN,SYN FIN,SYN -j DROP') }
  it { should have_rule('-A bad_tcp -p tcp -m conntrack --ctstate NEW -m tcp --sport 1:32767 -j LOG --log-prefix "TCP:SYN_priv_ports:DROP "') }
  it { should have_rule('-A bad_tcp -p tcp -m conntrack --ctstate NEW -m tcp --sport 1:32767 -j DROP') }
  it { should have_rule('-A bad_udp -p udp -m length --length 0:28 -j LOG --log-prefix "UDP:Small_size:DROP "') }
  it { should have_rule('-A bad_udp -p udp -m length --length 0:28 -j DROP') }
  it { should have_rule('-A icmp_in -p icmp -m icmp --icmp-type 0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT') }
  it { should have_rule('-A icmp_in -p icmp -m icmp --icmp-type 8 -m limit --limit 1/sec --limit-burst 1 -j ACCEPT') }
  it { should have_rule('-A icmp_in -p icmp -j DROP') }
  it { should have_rule('-A icmp_out -p icmp -m icmp --icmp-type 0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT') }
  it { should have_rule('-A icmp_out -p icmp -m icmp --icmp-type 8 -m limit --limit 1/sec --limit-burst 1 -j ACCEPT') }
  it { should have_rule('-A tcp_out -p tcp -m tcp -m multiport --dports 80,443 -j ACCEPT') }
  it { should have_rule('-A udp_out -p udp -m udp --dport 53 -j ACCEPT') }
end

This test specification applicable for the "clean" containers and used as shared resource for testing ssh rules:

require_relative '../shared_tests/iptables_spec'

describe iptables do
  it { should have_rule('-A tcp_in -s 123.45.56.0/26 -p tcp -m tcp --dport 22 -j ACCEPT') }
  it { should have_rule('-A tcp_in -s 78.90.123.0/28 -p tcp -m tcp --dport 22 -j ACCEPT') }
  it { should have_rule('-A tcp_in -s 45.67.89.0/28 -p tcp -m tcp --dport 22 -j ACCEPT') }
  it { should have_rule('-A tcp_in -s 172.17.0.0/16 -p tcp -m tcp --dport 22 -j ACCEPT') }
end

So I think same testing model helps you with development.

WBR, Eugene

[fpytloun]

Hello Eugene, it's good that you came with using kitchen-ci & serverspec for testing, because @epcim is already working on this :-) He added support for some of the formulas already (tcpcloud/salt-formula-postfix) and written blueprint for openstack formulas testing: https://blueprints.launchpad.net/openstack-salt/+spec/system-level-testing

Can you look at what we have and give us feedback? Or better - submit tests for formulas that you are using? :-)

Filip

[axmetishe]

Hello Filip,

I can't submit our formulas with testing due legal issues, but I'll take a look one of yours in my spare time and configure tests on it.

WBR, Eugene

[fpytloun]

Thank you, contributions are always welcome :-) You can also contact us at #openstack-salt or #tcpcloud at Freenode IRC.

[epcim]

Eugene, i Will provide you a handy script to easily add kitchen setup to our formulas. On 22 Sep 2016 07:10, Eugene Akhmetkhanov notifications@github.com wrote:

Hello Filip,

I can't submit our formulas with testing due legal issues, but I'll take a look one of yours in my spare time and configure tests on it.

WBR, Eugene

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[epcim]

@axmetishe in repo below is handy script "kitchen-init.sh" that will help you to create initial structure. I am about to add kitchen to all our formulas in future. Definitively feel free to propose PR. Thx for the test sample.

https://github.com/tcpcloud/cookiecutter-salt-formula