search

fititnt / ap-application-load-balancer

AP Application Load Balancer (AP-ALB). Sophisticated monolithic Ansible role to manage standalone and clusters of cross-platform and multicloud load balancers. Abstract HAProxy + OpenResty + On-the-fly auto HTTPS. Dedicated to Public Domain.
https://ap-application-load-balancer.etica.ai/
The Unlicense
1 stars 0 forks source link

ALB on BSD Systems #37

Open fititnt opened 4 years ago

fititnt commented 4 years ago

This issue started with AP-ALB v0.8.x adaptations to support RHEL/CentOS 8 family and design changes to allow flexibility even for non-tested OSs #34. This issue mostly likely to be a place to mention things on ALB that are different from the other systems and/or have a place to mention what is not implemented.

Note: I do not use BSDs systems in production, but this does not means that, when viable, at least made the base ALB have at least some bare minimum compatibility. Initially the ALB is tested on FreeBSD 12.

Aboud BSDs & BSDs & Ansible

Ansible Packages related to BSD

Ansible System modules related to BSD

BSD service management

Update 1:

fititnt commented 4 years ago

v0.8.4-alpha, I'm having errors similar to what had before with ansible facts related for permissions of directories. This one is very specific to FreeBSD

TASK [Gathering Facts] ****************************************************************************************************************************************************************************************************************
ok: [ap_foxtrot_centos8]
ok: [ap_delta_ubuntu18]
ok: [ap_echo_debian10]
ok: [ap_golf_archilinux]
fatal: [rocha_anortosito_freebsd12]: FAILED! => {"ansible_facts": {}, "changed": false, "failed_modules": {"setup": {"ansible_facts": {"discovered_interpreter_python": "/usr/local/bin/python3.6"}, "cmd": "/etc/ansible/facts.d/alb_openresty.fact", "failed": true, "invocation": {"module_args": {"fact_path": "/etc/ansible/facts.d", "filter": "*", "gather_subset": ["all"], "gather_timeout": 10}}, "msg": "[Errno 2] No such file or directory: b'/etc/ansible/facts.d/alb_openresty.fact': b'/etc/ansible/facts.d/alb_openresty.fact'", "rc": 2, "warnings": ["Platform freebsd on host rocha_anortosito_freebsd12 is using the discovered Python interpreter at /usr/local/bin/python3.6, but future installation of another Python interpreter could change this. See https://docs.ansible.com/ansible/2.9/reference_appendices/interpreter_discovery.html for more information."]}}, "msg": "The following modules failed to execute: setup\n"}

dirty fix is deleting the file (like ansible rocha_anortosito_freebsd12 -i hosts.yml -m shell -u root -a "rm -f /etc/ansible/facts.d/alb_openresty.fact") but this must be solved later

fititnt commented 4 years ago

This

-#!/bin/bash
+#!/bin/sh

solves https://github.com/fititnt/ap-application-load-balancer/issues/37#issuecomment-564359038.

fititnt commented 4 years ago

The HAProxy path is also different from the other *nix.

(...)
TASK [ap-application-load-balancer : reconfigure.yml] 
included: /alligo/code/fititnt/ap-application-load-balancer/tasks/haproxy/reconfigure.yml for ap_delta_ubuntu18, ap_echo_debian10, ap_foxtrot_centos8, ap_golf_archilinux, rocha_anortosito_freebsd12, rocha_basalto_opensuse15

TASK [ap-application-load-balancer : haproxy | reconfigure: /etc/haproxy/haproxy.cfg]
changed: [ap_echo_debian10]
changed: [ap_delta_ubuntu18]
fatal: [rocha_anortosito_freebsd12]: FAILED! => {"changed": false, "checksum": "af6b6093e5663f933e09e50c4ce1ae273316ab43", "msg": "Destination directory /etc/haproxy does not exist"}
changed: [ap_golf_archilinux]
[rocha_anortosito_freebsd12] TASK: ap-application-load-balancer : haproxy | reconfigure: /etc/haproxy/haproxy.cfg (debug)>

On the node

[root@rocha-anortosito-freebsd12 /]# find / -iname haproxy
/usr/ports/net/haproxy
/usr/local/sbin/haproxy
/usr/local/etc/rc.d/haproxy
/usr/local/share/examples/haproxy
/usr/local/share/doc/haproxy
/opt/alb/haproxy

This is our actual yaml task

- name: "haproxy | reconfigure: /etc/haproxy/haproxy.cfg"
  template:
    # src: "{{ role_path }}/templates/openresty/nginx/conf/nginx.conf.j2"
    src: "{{ alb_haproxy_haproxy_template }}"
    dest: /etc/haproxy/haproxy.cfg
    owner: "{{ alb_internal_root_user }}"
    group: "{{ alb_internal_root_group }}"
    mode: '0644'
    backup: yes
  notify:
    - reload haproxy
fititnt commented 4 years ago

HUMMM... FreeBSD does not use systemd. It's the old school service. Not that this is bad, just different. Will need more specific strategy. Also not so easy to find tutorials on internet, but maybe will not take that long.

Also, something I really, really liked on all other OSs tested that are not Centos or Ubuntu/Debian: all they already ship with HAProxy 2.0 or 2.1. So actually they do not need a repo for what we need.

[root@rocha-anortosito-freebsd12 /]# cat /usr/local/etc/rc.d/haproxy

Here I' trying to discover where to make things work and where to put everything.


#!/bin/sh
#
# $FreeBSD: branches/2019Q4/net/haproxy/files/haproxy.in 477980 2018-08-24 19:19:26Z demon $
#

# PROVIDE: haproxy
# REQUIRE: DAEMON LOGIN
# KEYWORD: shutdown

#
# Add the following lines to /etc/rc.conf to enable haproxy:
#
# haproxy_enable (bool):    default: "NO"
#               Set to "YES" to enable haproxy
# haproxy_pidfile (str):    default: /var/run/haproxy.pid
#               Set to the full path of the pid file
# haproxy_config (str):     default: /usr/local/etc/haproxy.conf
#               Set to the full path of the config file
# haproxy_flags (str):      default: Autogenerated using pidfile and config options
#               Set to override with your own options
# haproxy_profiles (str):   default: empty
# Set to space-separated list of profiles: for each profile separate haproxy
# process will be spawned, with haproxy-${profile}.conf config file.
# You can override default pidfile and config file for each profile with
# haproxy_${profile}_config and haproxy_${profile}_pidfile.

. /etc/rc.subr

name="haproxy"
rcvar=haproxy_enable
command="/usr/local/sbin/haproxy"
extra_commands="reload configtest hardstop hardreload softreload"
reload_cmd="haproxy_reload"
hardreload_cmd="haproxy_reload"
hardreload_precmd="def_hardreload_option"
softreload_cmd="haproxy_reload"
softreload_precmd="def_softreload_option"
stop_cmd="haproxy_stop"
hardstop_cmd="haproxy_stop"
hardstop_precmd="def_hardstop_signal"

: ${haproxy_enable:="NO"}
: ${haproxy_config:="/usr/local/etc/${name}.conf"}
: ${haproxy_socket:="/var/run/${name}/socket"}
pidfile=${haproxy_pidfile:-"/var/run/haproxy.pid"}

def_hardreload_option()
{
    reload_opt="-st"
}

def_softreload_option()
{
    reload_opt="-x ${haproxy_socket} -sf"
}

def_hardstop_signal()
{
    sig_stop="TERM"
}

load_rc_config $name

is_valid_profile() {
    local profile
    for profile in $haproxy_profiles; do
        if [ "$profile" = "$1" ]; then
            return 0
        fi
    done
    return 1
}

if [ -n "$2" ]; then
    profile=$2
    if ! is_valid_profile $profile; then
        echo "$0: no such profile ($profile) defined in ${name}_profiles."
        exit 1
    fi
    eval haproxy_config="\${haproxy_${profile}_config:-/usr/local/etc/haproxy-${profile}.conf}"
    eval pidfile="\${haproxy_${profile}_pidfile:-/var/run/haproxy-${profile}.pid}"
else
    if [ "x${haproxy_profiles}" != "x" -a "x$1" != "x" ]; then
        for profile in ${haproxy_profiles}; do
            echo "===> ${name} profile: ${profile}"
            /usr/local/etc/rc.d/haproxy $1 ${profile}
            retcode="$?"
            if [ ${retcode} -ne 0 ]; then
                failed="${profile} (${retcode}) ${failed:-}"
            else
                success="${profile} ${success:-}"
            fi
        done
        exit 0
    fi
fi

: ${haproxy_flags:="-q -f ${haproxy_config} -p ${pidfile}"}
configtest_cmd="$command -c -f $haproxy_config"
start_precmd="$command -q -c -f $haproxy_config"
required_files=$haproxy_config
sig_stop=SIGUSR1
reload_opt="-sf"

haproxy_reload()
{
    ${command} -q -c -f ${haproxy_config}
    if [ $? -ne 0 ]; then
        err 1 "Error found in ${haproxy_config} - not reloading current process!"
    fi
    rc_pid=$(check_pidfile ${pidfile} ${command})
    if [ $rc_pid ]; then
        ${command} ${haproxy_flags} $reload_opt $(cat ${pidfile})
    else
        _run_rc_notrunning
        return 1
    fi
}

haproxy_stop()
{
    rc_pid=$(check_pidfile ${pidfile} ${command})
    if [ $rc_pid ]; then
        rc_pid=$(cat ${pidfile})
        kill -$sig_stop $rc_pid
        wait_for_pids $rc_pid
    else
        _run_rc_notrunning
        return 1
    fi
}

run_rc_command "$1"

Edit: added another file

[root@rocha-anortosito-freebsd12 /]# cat /etc/rc.conf

zfs_enable="YES"
hostname="vmi321960.contaboserver.net"
ifconfig_vtnet0="inet 144.91.107.138 netmask 255.255.192.0"
gateway_if="vtnet0"
gateway_ip="144.91.64.1"
static_routes="gateway default"
route_gateway="-host $gateway_ip -interface $gateway_if"
route_default="default $gateway_ip"
#ipv6_default_interface="vtnet0"
#ifconfig_vtnet0_ipv6="2a02:c207:2032:1960:0000:0000:0000:0001/64"
#ipv6_defaultrouter="fe80::1%vtnet0"
static_ndp_pairs="gw"
static_ndp_gw="fe80::1%vtnet0 28:99:3a:4d:30:af"
sshd_enable="YES"
ntpd_enable="YES"
ntpd_sync_on_start="YES"
fititnt commented 4 years ago

Lastest commit we're already have HAProxy installed based on OS Family (the Ansible Generic package fails even for Debian/Ubuntu when I try to enforce one specific version.

Because BSD, this also means that at least the other step, that separate how to reload/restart/start the HAProxy, must be also dependend of the system.

This is our version using systemd (this Ansible module https://docs.ansible.com/ansible/latest/modules/systemd_module.html).

# Enabled/Started must be AFTER moving new /etc/haproxy/haproxy.cfg
# configurations or Ansible will not be able to update one new valid
# configuration if old one was already with error. By moving this step
# after we avoid user being forced to solve manually on the server 
- name: "haproxy | reconfigure: sudo systemctl enable haproxy.service"
  systemd:
    name: haproxy
    state: started
    enabled: yes

The BSD I guess will use the service, https://docs.ansible.com/ansible/latest/modules/service_module.html. Loggin on rocha_anortosito_freebsd12 I see that service command works like the old school method. But seems that the service alone will not make it aware of some default location of configutation, so have to add rules on /etc/rc.conf file. That's a bit scary if not well done.

fititnt commented 4 years ago

[root@rocha-anortosito-freebsd12 /]# cat /usr/local/share/examples/haproxy/transparent_proxy.cfg

#
# This is an example of how to configure HAProxy to be used as a 'full transparent proxy' for a single backend server.
#
# Note that to actually make this work extra firewall/nat rules are required.
# Also HAProxy needs to be compiled with support for this, in HAProxy1.5-dev19 you can check if this is the case with "haproxy -vv".
#

global
defaults
    timeout client      30s
    timeout server      30s
    timeout connect     30s

frontend MyFrontend
    bind    192.168.1.22:80
    default_backend     TransparentBack_http

backend TransparentBack_http
    mode            http
    source 0.0.0.0 usesrc client
    server          MyWebServer 192.168.0.40:80

#
# To create the the nat rules perform the following:
#
# ### (FreeBSD 8) ###
# --- Step 1 ---
# ipfw is needed to get 'reply traffic' back to the HAProxy process, this can be achieved by configuring a rule like this:
#   fwd localhost tcp from 192.168.0.40 80 to any in recv em0
#
# The following would be even better but this did not seam to work on the pfSense2.1 distribution of FreeBSD 8.3:
#   fwd 127.0.0.1:80 tcp from any 80 to any in recv ${outside_iface} uid ${proxy_uid}
#
# If only 'pf' is currently used some aditional steps are needed to load and configure ipfw:
# You need to configure this to always run on startup:
#
# /sbin/kldload ipfw
# /sbin/sysctl net.inet.ip.pfil.inbound="pf" net.inet6.ip6.pfil.inbound="pf" net.inet.ip.pfil.outbound="pf" net.inet6.ip6.pfil.outbound="pf"
# /sbin/sysctl net.link.ether.ipfw=1
# ipfw add 10 fwd localhost tcp from 192.168.0.40 80 to any in recv em0
#
# the above does the following:
# - load the ipfw kernal module
# - set pf as the outer firewall to keep control of routing packets for example to route them to a non-default gateway
# - enable ipfw
# - set a rule to catches reply traffic on em0 coming from the webserver
#
# --- Step 2 ---
# To also make the client connection transparent its possible to redirect incomming requests to HAProxy with a pf rule:
#   rdr on em1 proto tcp from any to 192.168.0.40 port 80 -> 192.168.1.22
# here em1 is the interface that faces the clients, and traffic that is originally send straight to the webserver is redirected to HAProxy
#
# ### (FreeBSD 9) (OpenBSD 4.4) ###
#   pf supports "divert-reply" which is probably better suited for the job above then ipfw..
#

[root@rocha-anortosito-freebsd12 /]# cat /usr/local/share/examples/haproxy/haproxy.init

#!/bin/sh
#
# chkconfig: - 85 15
# description: HA-Proxy is a TCP/HTTP reverse proxy which is particularly suited \
#              for high availability environments.
# processname: haproxy
# config: /etc/haproxy/haproxy.cfg
# pidfile: /var/run/haproxy.pid

# Script Author: Simon Matter <simon.matter@invoca.ch>
# Version: 2004060600

# Source function library.
if [ -f /etc/init.d/functions ]; then
  . /etc/init.d/functions
elif [ -f /etc/rc.d/init.d/functions ] ; then
  . /etc/rc.d/init.d/functions
else
  exit 0
fi

# Source networking configuration.
. /etc/sysconfig/network

# Check that networking is up.
[ ${NETWORKING} = "no" ] && exit 0

# This is our service name
BASENAME=`basename $0`
if [ -L $0 ]; then
  BASENAME=`find $0 -name $BASENAME -printf %l`
  BASENAME=`basename $BASENAME`
fi

BIN=/usr/sbin/$BASENAME

CFG=/etc/$BASENAME/$BASENAME.cfg
[ -f $CFG ] || exit 1

PIDFILE=/var/run/$BASENAME.pid
LOCKFILE=/var/lock/subsys/$BASENAME

RETVAL=0

start() {
  quiet_check
  if [ $? -ne 0 ]; then
    echo "Errors found in configuration file, check it with '$BASENAME check'."
    return 1
  fi

  echo -n "Starting $BASENAME: "
  daemon $BIN -D -f $CFG -p $PIDFILE
  RETVAL=$?
  echo
  [ $RETVAL -eq 0 ] && touch $LOCKFILE
  return $RETVAL
}

stop() {
  echo -n "Shutting down $BASENAME: "
  killproc $BASENAME -USR1
  RETVAL=$?
  echo
  [ $RETVAL -eq 0 ] && rm -f $LOCKFILE
  [ $RETVAL -eq 0 ] && rm -f $PIDFILE
  return $RETVAL
}

restart() {
  quiet_check
  if [ $? -ne 0 ]; then
    echo "Errors found in configuration file, check it with '$BASENAME check'."
    return 1
  fi
  stop
  start
}

reload() {
  if ! [ -s $PIDFILE ]; then
    return 0
  fi

  quiet_check
  if [ $? -ne 0 ]; then
    echo "Errors found in configuration file, check it with '$BASENAME check'."
    return 1
  fi
  $BIN -D -f $CFG -p $PIDFILE -sf $(cat $PIDFILE)
}

check() {
  $BIN -c -q -V -f $CFG
}

quiet_check() {
  $BIN -c -q -f $CFG
}

rhstatus() {
  status $BASENAME
}

condrestart() {
  [ -e $LOCKFILE ] && restart || :
}

# See how we were called.
case "$1" in
  start)
    start
    ;;
  stop)
    stop
    ;;
  restart)
    restart
    ;;
  reload)
    reload
    ;;
  condrestart)
    condrestart
    ;;
  status)
    rhstatus
    ;;
  check)
    check
    ;;
  *)
    echo $"Usage: $BASENAME {start|stop|restart|reload|condrestart|status|check}"
    exit 1
esac

exit $?

Edit: and more this one as reference

[root@rocha-anortosito-freebsd12 /]# cat /usr/local/share/examples/haproxy/content-sw-sample.cfg

#
# This is a sample configuration. It illustrates how to separate static objects
# traffic from dynamic traffic, and how to dynamically regulate the server load.
#
# It listens on 192.168.1.10:80, and directs all requests for Host 'img' or
# URIs starting with /img or /css to a dedicated group of servers. URIs
# starting with /admin/stats deliver the stats page.
#

global
        maxconn         10000
        stats socket    /var/run/haproxy.stat mode 600 level admin
        log             127.0.0.1 local0
        uid             200
        gid             200
        chroot          /var/empty
        daemon

# The public 'www' address in the DMZ
frontend public
        bind            192.168.1.10:80 name clear
        #bind            192.168.1.10:443 ssl crt /etc/haproxy/haproxy.pem
        mode            http
        log             global
        option          httplog
        option          dontlognull
        monitor-uri     /monitoruri
        maxconn         8000
        timeout client  30s

        stats uri       /admin/stats
        use_backend     static if { hdr_beg(host) -i img }
        use_backend     static if { path_beg /img /css   }
        default_backend dynamic

# The static backend backend for 'Host: img', /img and /css.
backend static
        mode            http
        balance         roundrobin
        option prefer-last-server
        retries         2
        option redispatch
        timeout connect 5s
        timeout server  5s
        option httpchk  HEAD /favicon.ico
        server          statsrv1 192.168.1.8:80 check inter 1000
        server          statsrv2 192.168.1.9:80 check inter 1000

# the application servers go here
backend dynamic
        mode            http
        balance         roundrobin
        retries         2
        option redispatch
        timeout connect 5s
        timeout server  30s
        timeout queue   30s
        option httpchk  HEAD /login.php
        cookie          DYNSRV insert indirect nocache
        fullconn        4000 # the servers will be used at full load above this number of connections
        server          dynsrv1 192.168.1.1:80 minconn 50 maxconn 500 cookie s1 check inter 1000
        server          dynsrv2 192.168.1.2:80 minconn 50 maxconn 500 cookie s2 check inter 1000
        server          dynsrv3 192.168.1.3:80 minconn 50 maxconn 500 cookie s3 check inter 1000
        server          dynsrv4 192.168.1.4:80 minconn 50 maxconn 500 cookie s4 check inter 1000
fititnt commented 4 years ago

Ok, FreeBSD does not come with an haproxy user/group at all. Some discussion here https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=199314 and proposed patch here https://bugs.freebsd.org/bugzilla/attachment.cgi?id=155368&action=diff.

fititnt commented 4 years ago

See Implementation of Ansible Molecule on AP-ALB & travis-ci integration #2

Reliable CI testing (like with travis-CI) may be more complicated than other linuxes. Here have some discussion on travis-ci about not supporting https://github.com/travis-ci/travis-ci/issues/1818.

In short, while is not as complicated than running a Windows conteiner inside linux host, run a BSD container requires a BSD host. BSDs are not a major concern on Docker to give full support as a host.

I found some places like these ones

that seems to use some linux OS (in the case, Alpine and Ubuntu) to then download a BSD image and somewhat "work" as if was BSD.

Not sure if this would be able to emulate a full BSD (at at least the bare minimum that could already be sufficient for our tests). But in this case, I guess that means that