IPV6 support for xtrabackup SST

[philip-galera]

Joe Topjian joe@topjian.net has provided the following patch and configuration suggestion for xtrabackup SST over IPv6:

[mysqld]
wsrep_provider_options = "gmcast.listen_addr=tcp://[::]:4567; ist.recv_addr=[IPV6-ADDR]:4568"

wsrep_node_address = [IPV6-ADDR]:4567

wsrep_sst_receive_address = [IPV6-ADDR]:4444

The patch is:

--- wsrep_sst_xtrabackup-v2.orig        2015-07-28 05:45:15.860108107 +0000
+++ wsrep_sst_xtrabackup-v2     2015-07-28 05:46:53.488108107 +0000
@@ -206,10 +206,10 @@
             stagemsg+="-OpenSSL-Encrypted-2"
             if [[ "$WSREP_SST_OPT_ROLE"  == "joiner" ]];then
                 wsrep_log_info "Decrypting with PEM $tpem, CA: $tcert"
-                tcmd="socat -u openssl-listen:${TSST_PORT},reuseaddr,cert=$tpem,cafile=${tcert}${sockopt} stdio"
+                tcmd="socat -6 -u openssl-listen:${TSST_PORT},reuseaddr,cert=$tpem,cafile=${tcert}${sockopt} stdio"
             else
                 wsrep_log_info "Encrypting with PEM $tpem, CA: $tcert"
-                tcmd="socat -u stdio openssl-connect:${REMOTEIP}:${TSST_PORT},cert=$tpem,cafile=${tcert}${sockopt}"
+                tcmd="socat -6 -u stdio openssl-connect:${REMOTEIP}:${TSST_PORT},cert=$tpem,cafile=${tcert}${sockopt}"
             fi
         elif [[ $encrypt -eq 3 ]];then
             wsrep_log_info "Using openssl based encryption with socat: with key and crt"
@@ -220,17 +220,17 @@
             stagemsg+="-OpenSSL-Encrypted-3"
             if [[ "$WSREP_SST_OPT_ROLE"  == "joiner" ]];then
                 wsrep_log_info "Decrypting with certificate $tpem, key $tkey"
-                tcmd="socat -u openssl-listen:${TSST_PORT},reuseaddr,cert=$tpem,key=${tkey},verify=0${sockopt} stdio"
+                tcmd="socat -6 -u openssl-listen:${TSST_PORT},reuseaddr,cert=$tpem,key=${tkey},verify=0${sockopt} stdio"
             else
                 wsrep_log_info "Encrypting with certificate $tpem, key $tkey"
-                tcmd="socat -u stdio openssl-connect:${REMOTEIP}:${TSST_PORT},cert=$tpem,key=${tkey},verify=0${sockopt}"
+                tcmd="socat -6 -u stdio openssl-connect:${REMOTEIP}:${TSST_PORT},cert=$tpem,key=${tkey},verify=0${sockopt}"
             fi

         else
             if [[ "$WSREP_SST_OPT_ROLE"  == "joiner" ]];then
-                tcmd="socat -u TCP-LISTEN:${TSST_PORT},reuseaddr${sockopt} stdio"
+                tcmd="socat -6 -u TCP-LISTEN:${TSST_PORT},reuseaddr${sockopt} stdio"
             else
-                tcmd="socat -u stdio TCP:${REMOTEIP}:${TSST_PORT}${sockopt}"
+                tcmd="socat -6 -u stdio TCP:${REMOTEIP}:${TSST_PORT}${sockopt}"
             fi
         fi
     fi
@@ -480,12 +480,12 @@
 setup_ports()
 {
     if [[ "$WSREP_SST_OPT_ROLE"  == "donor" ]];then
-        SST_PORT=$(echo $WSREP_SST_OPT_ADDR | awk -F '[:/]' '{ print $2 }')
-        REMOTEIP=$(echo $WSREP_SST_OPT_ADDR | awk -F ':' '{ print $1 }')
-        lsn=$(echo $WSREP_SST_OPT_ADDR | awk -F '[:/]' '{ print $4 }')
-        sst_ver=$(echo $WSREP_SST_OPT_ADDR | awk -F '[:/]' '{ print $5 }')
+        SST_PORT=$(echo ${WSREP_SST_OPT_ADDR##*:} | tr -d [])
+        REMOTEIP="[$(echo ${WSREP_SST_OPT_ADDR%:*} | tr -d [])]"
+        lsn=$(echo $WSREP_SST_OPT_ADDR | awk -F '[/]' '{ print $3 }' | tr -d [])
+        sst_ver=$(echo $WSREP_SST_OPT_ADDR | awk -F '[/]' '{ print $4 }' | tr -d [])
     else
-        SST_PORT=$(echo ${WSREP_SST_OPT_ADDR} | awk -F ':' '{ print $2 }')
+        SST_PORT=$(echo ${WSREP_SST_OPT_ADDR##*:} | tr -d [])
     fi
 }

@@ -814,7 +814,7 @@
     if [ -z "${SST_PORT}" ]
     then
         SST_PORT=4444
-        ADDR="$(echo ${WSREP_SST_OPT_ADDR} | awk -F ':' '{ print $1 }'):${SST_PORT}"
+        ADDR="[$(echo ${WSREP_SST_OPT_ADDR%:*} | tr -d [])]:${SST_PORT}"
     fi

     wait_for_listen ${SST_PORT} ${ADDR} ${MODULE} &

Maybe we should ship it as a separate IPv6-only SST script?

[ayurchen]

Philip, what is the effect of -6 option in IPv4 environments? Socat manual page is not really clear on that.

[jtopjian]

I'm only speculating and haven't verified this, but socat -6 will enforce IPv6 connectivity and would probably not work for IPv4 connectivity even if IPv6 is not explicitly disabled and traffic is going over IPv4.

The socat man page has a few different options that can be used to prioritize IPv6 name resolution and connectivity and possibly fall back on IPv4 (search for pf and -6). In that way, it may be possible to have IPv6 and IPv4 coexist in the same scripts.

gmcast.listen_addr=tcp://[::]:4567 should be safe to use in IPv4 environments where IPv6 has not been explicitly disabled, however, the same caveats apply as binding to all IPv4 interfaces.

[philip-galera]

Patch in raw format:

https://gist.githubusercontent.com/jtopjian/b280b4f04942e456a880/raw/a09bcdc6ad3da9f413d0b88c32f1962c6991b416/wsrep_sst_xtrabackup-v2.patch

[philip-galera]

@ayurchen, it turns out that the attached patch indeed does not work on a pure-IPv4 host:

socat -6 -u stdio TCP:[127.0.0.2]:13007/xtrabackup_sst//1
2015/08/02 23:21:55 socat[63722] E getaddrinfo("127.0.0.2", "(null)", {5,10,1,6}, {}): Address family for hostname not supported

The "pf" socat option still requires that the user choose either one IP version or the other. So there does not seem to be a single socat command that can accommodate both.

[philip-galera]

Actually, it seems that it is the format of the address makes a difference. The patch adds explicit angle brackets to all IPs, even IPv4 ones. So socat seems to go for IPv6 regardless of the options specified.

[jtopjian]

@philip-galera Indeed - the patch was strictly to get IPv6 to work so some extra logic will need to be added to account for both IPv4 and IPv6 addresses. Additionally either more logic to use both IPv6 notation with and without [ ], or just standardize one one format.

[Roguelazer]

I would be very interested in getting working IPv6 support in this SST as well.

As per the man page, socat with no -4 or -6 argument will automatically determine whether to connect with IPv4 or IPv6 socket, so all we actually need to do is change how REMOTE_IP is parsed out to ensure that it gets surrounded with square brackets when it's an IPv6 address, and that the -6 flag is used with TCP-LISTEN (which will actually bind to both IPv4 and IPv6 unless the Linux sysctl net.ipv6.bindv6only is set to 1).

FWIW, this is what we're running here (which seems to work) is https://gist.github.com/Roguelazer/436ddad699acea115ec2fbf297824e03