iptables forward doesn't work for mysql replicated stateful application

RKE version: v0.1.10-rc2

Docker version: (docker version,docker info preferred) Containers: 70 Running: 26 Paused: 0 Stopped: 44 Images: 16 Server Version: 17.03.2-ce Storage Driver: aufs Root Dir: /var/lib/docker/aufs Backing Filesystem: extfs Dirs: 227 Dirperm1 Supported: true Logging Driver: json-file Cgroup Driver: cgroupfs Plugins: Volume: local Network: bridge host macvlan null overlay Swarm: inactive Runtimes: runc Default Runtime: runc Init Binary: docker-init containerd version: 4ab9917febca54791c5f071a9d1f404867857fcc runc version: 54296cf40ad8143b62dbcaa1d90e520a2136ddfe init version: 949e6fa Security Options: apparmor seccomp Profile: default Kernel Version: 4.15.0-35-generic Operating System: Ubuntu 18.04.1 LTS OSType: linux Architecture: x86_64 CPUs: 1 Total Memory: 3.914 GiB Name: ubuntu-rke-1 ID: VLIA:XYZ4:6QYJ:AQUF:ESH4:OHIE:ICRU:2CPI:5QAI:6633:5QHD:NPF4 Docker Root Dir: /var/lib/docker Debug Mode (client): false Debug Mode (server): false Registry: https://index.docker.io/v1/ Experimental: false Insecure Registries: 192.168.101.181:5000 127.0.0.0/8 Live Restore Enabled: false

WARNING: No swap limit support

Operating system and kernel: (cat /etc/os-release, uname -r preferred) Ubuntu Server 18.04 (Bionic)

Type/provider of hosts: (VirtualBox/Bare-metal/AWS/GCE/DO) VirtualBox

cluster.yml file:

nodes:
  - address: "192.168.200.2" # hostname or IP to access nodes
    user: "devops" # root user (usually 'root')
    role: [controlplane,etcd,worker] # K8s roles for node
    ssh_key_path: "/home/devops/.ssh/id_rsa" # path to PEM file
  - address: "192.168.200.3" # hostname or IP to access nodes
    user: "devops" # root user (usually 'root')
    role: [worker] # K8s roles for node
    ssh_key_path: "/home/devops/.ssh/id_rsa" # path to PEM file

services:
  etcd:
    backup: true
    creation: 6h
    retention: 24h

addons: |-
  ---
  kind: Namespace
  apiVersion: v1
  metadata:
    name: cattle-system
  ---
  kind: ServiceAccount
  apiVersion: v1
  metadata:
    name: cattle-admin
    namespace: cattle-system
  ---
  kind: ClusterRoleBinding
  apiVersion: rbac.authorization.k8s.io/v1
  metadata:
    name: cattle-crb
    namespace: cattle-system
  subjects:
  - kind: ServiceAccount
    name: cattle-admin
    namespace: cattle-system
  roleRef:
    kind: ClusterRole
    name: cluster-admin
    apiGroup: rbac.authorization.k8s.io
  ---
  apiVersion: v1
  kind: Secret
  metadata:
    name: cattle-keys-ingress
    namespace: cattle-system
  type: Opaque
  data:
    tls.crt: TFMwdExTMUNSVWRKVGlCRFJWSlVTVVpKUTBGVVJTMHRMUzB0Q2sxSlNVZE9ha05EUWtJMlowRjNTVUpCWjBsS1FVcHVka014YVhkSEwwWTVUVUV3UjBOVGNVZFRTV0l6UkZGRlFrTjNWVUZOU1VkMlRWRnpkME5SV1VRS1ZsRlJSMFYzU2tOVmFrVldUVUpOUjBFeFZVVkRRWGROVkZkc2RWbFlUWFJTTWxaNVdWZHNlazFTVVhkRloxbEVWbEZSU0VSQmRHaGlTRnB3WW0wNWR3cGlNbmh3WTNwRmNrMURhMGRCTVZWRlEyZDNhVkZ0YkhaSlJWWTBaRWhLYUdSSVZucEpSVTUyWXpJeGJHUkhiR3BKUlRWb1pFaFdlVmxYZDJkVVNGSnJDbGxVUlZaTlFrMUhRVEZWUlVGM2QwMVJiV3gyU1VWV05HUklTbWhrU0ZaNlRWTTRkMHhSV1VwTGIxcEphSFpqVGtGUmEwSkdhVUp3WW0xYWRtTnRNV2dLWkVkc2FsbFVSWHBSUjBwd1lqSldOR1JJU21oa1NGWjZURzFPZG1KVE5XbGpha0ZsUm5jd2VFOUVRVEpOYW10NFRXcFZkMDVVYUdGR2R6QjRUMVJCTWdwTmFtdDRUV3BWZDA1VWFHRk5TVWQyVFZGemQwTlJXVVJXVVZGSFJYZEtRMVZxUlZaTlFrMUhRVEZWUlVOQmQwMVVWMngxV1ZoTmRGSXlWbmxaVjJ4NkNrMVNVWGRGWjFsRVZsRlJTRVJCZEdoaVNGcHdZbTA1ZDJJeWVIQmpla1Z5VFVOclIwRXhWVVZEWjNkcFVXMXNka2xGVmpSa1NFcG9aRWhXZWtsRlRuWUtZekl4YkdSSGJHcEpSVFZvWkVoV2VWbFhkMmRVU0ZKcldWUkZWazFDVFVkQk1WVkZRWGQzVFZGdGJIWkpSVlkwWkVoS2FHUklWbnBOVXpoM1RGRlpTZ3BMYjFwSmFIWmpUa0ZSYTBKR2FVSndZbTFhZG1OdE1XaGtSMnhxV1ZSRmVsRkhTbkJpTWxZMFpFaEthR1JJVm5wTWJVNTJZbE0xYVdOcVEwTkJhVWwzQ2tSUldVcExiMXBKYUhaalRrRlJSVUpDVVVGRVoyZEpVRUZFUTBOQloyOURaMmRKUWtGT0wxWmxNVEJDV25sM1lVaDNVMGsyUkZkSGFXMUNMM05DV1dFS0swaDFiM1ZvWkZOTGNXMDBZV1ZhSzIxc1NsVmxSV3hWWld0Q2QydDJaM2xvTTJwc1kwdHlZbmxVWmsxS2VYa3JWbGN4VW5aTlUyUmhlbU4xZHk5M1NBcFdVemtyYW5ocksyZHFXblphTlVoak9UTmlWbmx5Y1VVemRYcGFiMU52ZUVvM1QwZDZiM2t5YkdaUmRGcGtaV2RCTkRoMllVcHNjRUYzYzJvMmJqRTVDamQxVkVSVVFpczFUV2xsTkZZeVdtVmpTMDAzZUd4MGJEbG5ZVXg2WmtOYVRraDFVRFpET1ZkUFdYTkhaVmhWWWxONGVtOXZRWGM1VWxKSFIyNU5kVEVLSzJKa2NrRk5jVGxxWkVST1lUbEhja3R0Ukc1VE0yWlVRbVpVVUd0WlN6VndhWGRPZDFrcmVFWkZXVGQ0YkUxUk1GUjNOM2huV0dVM2NYVXpNbVp6WndwNVoweEdVVGRFUVdSNVFsTk1ORUZaUTBWRlFub3JMMkp3T0hoUWFWUk1Vbk51U1VKRmJtUXhhR1V3Y1RWUU9GWmFRMUZEZFZsNmJtcFRVMWszTlRGV0NsazFka0Z5Vm5OemJVcGhlV3BpV25sV01qWnlVV0pqVVhSS1ZtTnVZVXMyYm1GbmNrOU5hbkJZUm1zeVJqRkdaRmRuZEhweWVrNVFPRkppWjA0ck9GY0tWRWx5U0c4NVRFMUdRa05xU2pKclJVcEdiMWhDZFhsR1VVUTFRamxRVlhCVmVsSktWVlJ5TjJrd1NYUjBSMFJNYlM5dFRFaFFSRlJzTkdNdmVWVlVjZ3BoYWtGSlZtWjNaRzVzZEdwcVlYWllaa3hTWm10VGFHcEdWbWRoZUc5Vk9VWXZSVlJ3VGpKc2NFSmxURlJSSzFaMmFWVTFZMUZPWkUxNVNsRjZNM1pwQ25WaFVtcEtORU41Y1dvMmEzVnBXR3hYVjJsaVRuVXhOVUZOWTFGcVJHWXpkRzl2VnpkWE0wcHVVRUp3SzFCcFZUa3pWVkJsZEZKQ1ZFTm1kMlJ2TlRVS09HUndOWEpRVm0wM1J5OHlWVVp2V0hOcE1uWk1SMUZDZEhwdU5sVmxhMlpIUVROdGNVRlBhMVUwYkdKTlRUbFlhazgzYlU5VU9VRm5iRUp6UWsxd1Z3cExWa05JWjFWUEsyOU9XVU5wUTBkVVFXZE5Ra0ZCUjJwVmVrSlNUVUl3UjBFeFZXUkVaMUZYUWtKU2NsVm5VbGR6YWpodk0yVlhTak5JVDJoV01HOTZDbFJpTUZabGVrRm1RbWRPVmtoVFRVVkhSRUZYWjBKU2NsVm5VbGR6YWpodk0yVlhTak5JVDJoV01HOTZWR0l3Vm1WNlFWQkNaMDVXU0ZKTlFrRm1PRVVLUWxSQlJFRlJTQzlOUVRCSFExTnhSMU5KWWpORVVVVkNRM2RWUVVFMFNVTkJVVUZaWWtSR01qYzVkazgzUzNvM1IwNHlPRTFDTmtwSWVWaFVNSEpJT1FwdVpDdDFNQzlvVGsxV1QzUXJUVVExUXpsR05EUTNkamswT0RoWE9XSm5VU3RMYzJ3eVNUWXlORnBFZFROb1Mwc3hhRXQxYVV4WFdGUkVNRE5PTVdkM0NtMTJValV6V2pSbmJYQjBheTlzVEZab09VRmlPVEZYUldrMVpFb3lOakJUY0dwV01VRlZaVGxGTmxOV09VcEtRbkZZZVZOT1oyOVlSMHAzTW1KWGVFNEtjM2h6ZVRCUmMyZEpXRWhVUlV4YVJFUlFXbU5qWjJGclNYZEZlWFpZUm0weU9FUnVaRmhSYkZGTVJVWlhSalV2VFhvME5FMW5jQzgzVFdJeWRHaEVVd292VUZKS1VqaDVURVZPYmtkbWNESkVZbUpaUTBkdVZYaDBRVzFCVGxwV1kzTXZWMGRsYW5sdmVWSm1kMWgxTm1SVmVuSjBTRVZuZUVkaWVETk9VV3QyQ21aTllYbDRWMDUxV1RsNkwzZDVObkZqVm5Zd1ZHTlRjbXBHV21ONll6RlhTMmRYWmtoUlpqYzNlREZqWVdWTlZHcDVkblJsTjA5elZ6Vmtka0ZDTmxrS0x6RTRlSFZMVWxoemMweEJhM05RVml0eFZtMU1UbTVxZW10WWVFZEpMMFF6SzFsbldqZ3dTMjlYVHpZMlRXcDVNWFkwU0ZoSGJqUjNORXQxUWxKNFVRbzRhM1p3UkV4R1lYbE5OMkZ5YnpGeVJEQnFPUzhyWVdzeFdtNXBla2xWWW5oNk9FRnNSWHBzUTBsTlpXSmtkV2swY2poWWVGUjZRVUpyTWxkeFkzWnhDbGcyZW5sc2NrVkpiV2RTU0VnMVFXWlhVRkp3VDNOWVRXWXpTRkZZY0V0VmFFTkNVQ3QyT1ZWbFJrdFJkRGszSzJ4d1pqZEhiRUpPYWtoNFRFeEpja0VLUkhSWmMxbzNlRk42VWtaaVpUVTJMMU5MVnpoTlFVOHpWMjVNY2pnd1dtZ3lXa05IVTBSdGNHSklOWGNyVVM5blRHRXhNelZ2TjFZd1QzTlJObXhvWkFvMVpWYzRjVFIwZVhGVWRrSTVSbGh6YnpNMGRuTjZaVXhxYkZCQlFuVjZPVVJpUzFGMlVVVk5OWFZ3VVc5RVlsUnViVkpGVDBkd1VFSktiSFprU1c5Q0NuZDVNMjl2TVdwRFNTOHlZVE5CUFQwS0xTMHRMUzFGVGtRZ1EwVlNWRWxHU1VOQlZFVXRMUzB0TFE9PQ==  # ssl cert for ingress. If selfsigned, must be signed by same CA as cattle server
    tls.key: TFMwdExTMUNSVWRKVGlCRlRrTlNXVkJVUlVRZ1VGSkpWa0ZVUlNCTFJWa3RMUzB0TFFwTlNVbEtjRVJDVDBKbmEzRm9hMmxIT1hjd1FrSlJNSGRSVkVGd1FtZHJjV2hyYVVjNWR6QkNRbEYzZDBoQlVVa3JjWGh1VmpabWFHaEpORU5CWjJkQkNrMUJkMGREUTNGSFUwbGlNMFJSU1VwQ1VVRjNSa0ZaU1V0dldrbG9kbU5PUVhkalJVTktTbFpZTkhVclZEZGtSMEpKU1VwVlExTmpOa1JOWjJGVVJEY0tWRXczYUdkNFFtazNkVk5EUkRBNFFtSkpSWEZvWm1KUFIwUXZOemN4VDB4NllrbFFRbVZVVGxRM1YyUktOako1VUVwYVR6VlZUakJ1YldoVmMzZHZWUXB2Vm1SMmRIZE5NSHByU1hSd1FYZHBSM0pZV0ZGVlFraG9TMGMwTkZOWlpYZEVPWEU0VUZwM1UwMVVNbEJrSzBObGJEZGtXbmx6Y2xkS1pTdEZibWhhQ20xMk5rMTVRbXhyUldwME56SjVWRTQ0Vlc5SVIzTnZUVUlyY3l0NmFVOUZSbTF0UVZSRFEwOHpSbEZOVDJGRmNFOUpPRkpxT0ZWTlYzaFBRVGx4ZVNzS1IyRllXamR4YkhBcmVFVnNPVUpLVjBoTVZqRnhUM3BFYzJSaFVFTlFNVWRqVVU1NFVEQjZXVzQxUmtjeFdGcGtUR2hDZEVoSWIwOVdSR3cyV1ZCeU1ncFRjVWxKVmpBeFFWZFNiRzEzYldoME1UUlRjVXRoUlVSV09YVkxTSFJ6V0U1d1VFZE1WUzlSU0RGRWN6TnZOWEp4ZVcxU1pUVnJZbVpaVmsxcmRFb3pDa2N6VVRWRVVFaFJSVGhuY0hNMU5HeElWSGc0YWxsTVEydFhiM0p2Wlc0NWFIQkhNM3BUU0dKM1puVkpNMWh4VHl0TlR6TnlUV2MxUVc5ck9FcDBPRWdLWVhkdlExUTBkRzVRZDFSTVFqQk1SVmdyYmtGNFRISlJTSGs0WXpkYVVEVm1ZMGxoZFUxblVHRXZhRzlXY1hGT2ExUlVUa1I1YW1sMmNFUkVkM001WWdwSlFuaEpialY0WTBkNU1rOVNjRlprUTBvMWVqVktVVEJ6UjNsak5YbFlURGxpUVc5NFkzVjJTMVozY1ZGSmFrOURNVzV2YjFoaE0wUjFTV0l5Tm5GcENtVTNUMXBZYVdsSWN6SkJPSEI2ZWpnMlNVNXRXbEp2ZG0xR2NWRkpWbE5xUkZsM2FXWk1TMEZETVZweVdreEZaMnBIWldjeVIxcDZXVFJqUlhSSlpuY0tOVzF3YzA1UlVuTTFRV1ZUTjNwVlJEZHhOWGhwVFVWbU4wbHVObEZrUlRoR0wxWkhOVEIxWjFOWVMyVnVRMUZCVlhVMVNFdEROekJxYUdWcWNXTnVVZ3BUUTFoSVNFOUhTbGx3UkdsV1Mwb3dPWFpoYUN0a1NXTnNVM3BxTWpsNFltTXhlazFWYkd4VWRsazBaMmhDVkRSaVFsQk1SREZtWjJ4NWJrazRiQ3MxQ2l0WE9HUk1UbXQxUm5KRE5VZFBjRmRxT1ZCeWEzVnhhMHd2V2xGbFJuTjVRMUZ4TkZwUFNrUnphalJqU21KMVlVOWFkalU1UWxsclp6bFJXVkpsZERVS2VrbDBaalJMTlVsak56VlpWMkpVYm5jdlVVOVBOMjEyZDNoalZWZzNjbTkxZVRkMmQzUnlTVVF6Um1JMVZqaEJVM0ZGVldka1EyMVZRWGQ0ZVd4M2VBcFdTV3BRVEVGVlUzSlBNVmMzVW5Gc1FtTklja1ZWV214TlNGWkpRek5XV1ZVd1dIbHRVRUZGYlZJemRXUlNSMGh2UVdwYU5FSk1WbFlyUTNWb2NubFVDbGRXU21OaE0yRmpUMVV3THpoR1VYaHRiRGR6ZVhwQmNHMXFObTk2ZVU1WFVHRXpNRVZRY1hoV1IyRmhZbEpUZWxoVlZsUTVibE13T0dGaWRFdzBha01LVVVWblNtUlVUWGxSWlZVNFIxaHdjemR6ZGtOTlZESTNjRkkxVVVkU0sxZ3dNbHB1U2taeU5YbElNSFpHZVZCTFIwRjZUV1ZYWlRKWllUaERRbGd3YXdwbVREZG9lVnBXZEVwWE1HTnVVVFJYVjBOclJXdHlRMmc0Um05TWQySjVhWFJ3WVZCcGVEQlVXRzlVUVhwU2NrOUNaSFJXWnl0U2IyUjNiMDl6Y0dSdUNtTTFkamRyUm5OclRra3pjR1pRZEZJM05tdFdUbFZFUVZwM1RYTm5Sa3g2VEdGRmFIWTNWR0Z0ZERGMlpDdEROamN4T0dWM1UzSlhUMFYyVlV0Tk5HRUtiWFZCWkRoUlRYVXJZbUUyVVVNclQwcERTakEyZG1RMGMxVnNOMkkxYkNzdkwzVllhV2htTUdWcFNYUnVNbFJYVkVVMVdsbzBSWGxKZEhobVF6SndlUXBXZW5sc05YWkVaMHRVWm5KaE5sSkJOUzh4VjNweU1GUlZSM2RJYnpOcFRtZ3dObTVNWTB0VWFrVndTMGQxVDBzM055OVVUR0pCY21GU05tTnBhU3MyQ2xwak1WcERaVGh0ZW05TGNqZFRaVmRxV2t0UGNEWlJaelJyUTNvMWQxTTRiMnRYUzFkUUwzRnVUazFpUkZwRE5GQjRhVTR3TUVaRWNGa3llRTlvY1RnS2NrRllSMnd2TTI5UWExSTFlRk01VGxWQlRIWjFTa3R3ZEVWMlJsQXJiamgxYUhvd04xcFJVR2N3V2xZNFJWa3ZVMjE0YVd0NWNUTnZhVTEyVW1ScVl3bzJVRGRFVVdGTGRtOUlkazVXTTBZeU5EZHFWQ3RYZEZCRmEzZDViVmt4WW1KSFZWSkJXbEJaVTJGRlkyZEhlSFJaWWl0cFVYa3pUMVpKY25CMVFVUnlDbTFqTHlzeVMwSjJibGtyUVRsTGEyaE9NbEZCY2t0cVNVVTJXbGRNZVZkNlYwNUllR0YxVkROTVJGWkdhVU12ZG1aRFFWQlNVM2QxVEdWbVRtSXZNRm9LY0VwTFNWSnlOMWN4Tm5KWVREWndVMGRXVmtwUVlUWnllRUZpU25KTGJYZDRjVVZvVWtkcFFuUk5SemsxTmtWWE5VODBUVll3YW1NellXcEZha3cxZHdwVmJGTlpjMjEzZEhCeWVIVTFXQ3RCWkhsRmJHUTVNV3hXY0RoNlUzUndUVkZVUjBvNFFTdGxhVFJHVjJGcmRIcGpaVTE1UlVkRmNrdHNVa0ZEY1VaRUNrcE1OMmhSVVM5bVRuZFBORFJPWjJKNFpVODFiRTQyT0RodWVUbGtWa2hJVEdZMWVtNWhRWE5sTDA5eWJHczNiRmt6ZWtvM1NIQnRTRE5UV1dac2NGSUtOSFZ0SzBRekwwMXJORWxRUVdoQ2FIQndOa3hxY1ZjNU5XcG1UekIwVnpjMVZuUkxUWGhCZGt0NVVEZERWRVp6SzB0alFtNTVVbkpXVEUwNWNWbHBPUXBMVUdjNU9WRmhVbUUwYm1Gb1pEWkNhbkJOTmt0blpXeHRNbGxFVlRCa1JWbFFUek5NTW5aS1MzRnBWRUpXZDJSYWRHbHhTaTlyYzA0MVV6aHRUVmx1Q25CYVpEazJjbXhNVjBKRWVFVkJVREV6WkVGNVNEQkhSVGx4VWtsbmFWazJXbmRuT0d4aWJWSnZNbWhPUjJGSlNtVmlVbUZCSzAxcmFUUnJkR2hSWjAwS1lrSkxkRTEzWjBkclkzbHhkMFJGYVc1clVFaDNiekZpVEZScE5rVjZZM0pJWldwRlJEVTRSMUIyVFRkSlZGRk9RMUpQUjNWMkwzbHpiRVpPYmxGU01RcEVUMUpWT0N0U1ZEZDVia2xRUWpaVmRGcEtPWGg1YmxsdGRXRnZhSEZ3TUdGbFJqVjZTbmxOWlVkaVZDOTVabWQyZVVvelFtdEliVXhWYnpSRkwweGpDalJrVFhOaGVVeGpPRlI1T1ROT2QxaFljMko2WldJMGFuSjRaMVJNZWtka2VYRm9SVnBsVlZGM1RWRlFOVzVVUTJkemMzaE5MM3BaWjNSMGNtWnhTbmtLVWxob1ZFaE9SM3BJZUdoc1ZTczNSaXROVW5KcVkzRmpOV1kxUVRaak1IUlNibXg0V0dWSmRrUnpVaXRYYWt0WE1XcHZMMWhTYkZob2RuWlRZMHhDY1FwcGFIYzRRek0xV21WVWRGUTBibE5VY0ZkcGRHVm1TbU5hUXpCb1RrazBWamhHUmpGTFZVbFRUakJGYTB0b1VFeGpTRkpOTVZwSFRVSTNTemR3YzBaVkNuRlZlVVpTUWtOa09XZHJTM0JoYjBwbVpFd3dOMHBWTlhvcmJucEtabFZKZUZJd2NsRlViVFZ1ZDNKSlFXOUJaRlpqTkdabGVXdHNiRzEwZERoVEsxa0tXa1ZpWmtsM1kyVmhiR0Z1Vm5kMFJYQlFjV1JUWW05dFQxZFdXa0pMU2tGWFZUQmtjbkZDY1habVZYVkZSbTk1ZEZwdlZVbG5XR3BYY1RRelNHYzRRd3BrYVZkb1UwOUNOM2s0UjBKVmNWTTBNVVZpUTFOU2QydFZXa2xuYTJveFNFVXJTbk5vT1U0eGNqVTJTa1ZOVkhkTU1Wa3pZbEV2UzFJdlVqUTBUbGgwQ2xsSk0xZHlZU3RTVGtGeVMzZE1TSGRMU3pVeGNYcG1WR1V2U1ZoaGVVNU1hV05FVURkdVlVMVNORmMyWjFObFJVRkdNVklyY0dsdlZqZGxTM05EVlVzS1QyODRRakJYYXpWYVNHdGxhMGN6VWxvMFNIaFVVR1JpYld0dGRrTk5hMDQzUzJWaGQyTjBjMmhtY1hFeE9TOWlla0Z1WVRGeEsxQlNTRE52VlhOR2N3cDFZM0p2VDBscVNWQjJhRFZ0WjFwa1lWUTJka3R3YnpKUFV6bEtXV0ZWYkdKU1dWZHRTbTFTVTFWWFlVeDJXV1ZpYkU4NGVITTROMjFYS3pNMlFrRjZDbGMwWTBWUGJYQTBOVmh2YUVOdU5tczVjbGN3T1RCUlRrOVlRbmsxZUZsaFJHdFlRMWhZZUdkcmJrcFhZMlJCVERsaE16QXdjRVJJZUZobllqZG1Zak1LZWtaUGIyNURUV3Q2YVdrMmFuSXpkVFpHZUZVM2VtTnVRM0JHTnpkS1EzaHdVMUZ6VlhGaWNXSkJPVmxCWjBRd1dtMVZjV1pZY21seVEwSkdSek5GVmdwdVJuazFVemR0VW5kQlpIcDJhMkY2WTFGcWRIQkRNMGhTTUVoVU5FZEJjMFIwTkdoTWJFbHVXRmRQTnlzdmJXZE1SRkV2Y2xsdlRVbExkMmRsUkVGcUNrTm1WSGh3U2tocGVEbHhaekpTWjB4alpGbDJVWGs0YzBSVVIyRldWRGx1VW05aFN6aGFTRlpRYmpnd2JWWnZjbTFtZVhwd2VXbHBRazVKUkdSSVdHSUthMHR2TXl0U2FuTkNRbU14YTBScGIyMHJRWGgwWjIxUWN6QkRMemNyWXpkallXNWtWREo0U1VGblJGQnlZM2hGUlV3M1lWRldkbXhMVG5CRVFuTkNVQW93VEVkME0zUllkVk4wYjJsTlFVcDRhazFuWjBSWmRqUndPSGQyUmxSbGJUQTJXV1J5Wm1SMVQyOVFjM05DVFdvelJXaHhaREpKUkRCaFNtVnBZbTFhQ2tWV1VsZ3piRVJZV0RONWMxQjBaazFRUVdkek5WWlhhbFpJUzJJemEyMUtPRVpuYjFrd05tMXJkSGhyVkVoVVkwaE9PVlZ2WTNrM1kyZzFNbUpsVEdnS1VHcHZSR3d4YVc5UmVrZ3lXRk54WXlzNFJEZE5WbGN2YjBaUE9UQldVMjE2YTA5MFRHVlVRamxtZGxSV1NDOUxZV3N6WW0xbk1rTXJXRXhFZUhSdWRncG9UazlOVVRFNUszRlVSbmxEWTFWUFJta3Jhalp1ZEZwUFZEQkNaVGgwT0FvdExTMHRMVVZPUkNCRlRrTlNXVkJVUlVRZ1VGSkpWa0ZVUlNCTFJWa3RMUzB0TFE9PQ==  # ssl key for ingress. If selfsigned, must be signed by same CA as cattle server
  ---
  apiVersion: v1
  kind: Secret
  metadata:
    name: cattle-keys-server
    namespace: cattle-system
  type: Opaque
  data:
    cacerts.pem: TFMwdExTMUNSVWRKVGlCRFJWSlVTVVpKUTBGVVJTMHRMUzB0Q2sxSlNVZE9ha05EUWtJMlowRjNTVUpCWjBsS1FWQllZV1p2VFVGdVFWSjVUVUV3UjBOVGNVZFRTV0l6UkZGRlFrTjNWVUZOU1VkMlRWRnpkME5SV1VRS1ZsRlJSMFYzU2tOVmFrVldUVUpOUjBFeFZVVkRRWGROVkZkc2RWbFlUWFJTTWxaNVdWZHNlazFTVVhkRloxbEVWbEZSU0VSQmRHaGlTRnB3WW0wNWR3cGlNbmh3WTNwRmNrMURhMGRCTVZWRlEyZDNhVkZ0YkhaSlJWWTBaRWhLYUdSSVZucEpSVTUyWXpJeGJHUkhiR3BKUlRWb1pFaFdlVmxYZDJkVVNGSnJDbGxVUlZaTlFrMUhRVEZWUlVGM2QwMVJiV3gyU1VWV05HUklTbWhrU0ZaNlRWTTRkMHhSV1VwTGIxcEphSFpqVGtGUmEwSkdhVUp3WW0xYWRtTnRNV2dLWkVkc2FsbFVSWHBSUjBwd1lqSldOR1JJU21oa1NGWjZURzFPZG1KVE5XbGpha0ZsUm5jd2VFOUVRVEpOYW10NFRYcEJNVTFFU21GR2R6QjVUVlJCTUFwTlZHZDRUWHBCTVUxRVNtRk5TVWQyVFZGemQwTlJXVVJXVVZGSFJYZEtRMVZxUlZaTlFrMUhRVEZWUlVOQmQwMVVWMngxV1ZoTmRGSXlWbmxaVjJ4NkNrMVNVWGRGWjFsRVZsRlJTRVJCZEdoaVNGcHdZbTA1ZDJJeWVIQmpla1Z5VFVOclIwRXhWVVZEWjNkcFVXMXNka2xGVmpSa1NFcG9aRWhXZWtsRlRuWUtZekl4YkdSSGJHcEpSVFZvWkVoV2VWbFhkMmRVU0ZKcldWUkZWazFDVFVkQk1WVkZRWGQzVFZGdGJIWkpSVlkwWkVoS2FHUklWbnBOVXpoM1RGRlpTZ3BMYjFwSmFIWmpUa0ZSYTBKR2FVSndZbTFhZG1OdE1XaGtSMnhxV1ZSRmVsRkhTbkJpTWxZMFpFaEthR1JJVm5wTWJVNTJZbE0xYVdOcVEwTkJhVWwzQ2tSUldVcExiMXBKYUhaalRrRlJSVUpDVVVGRVoyZEpVRUZFUTBOQloyOURaMmRKUWtGTU5DdE1LMEZ3WlZwb0wyNWxiRVYzWjJVdk0zaDFPSGRuVG04S2VHTnRhRGRSVTBkcU0yVkhOVEZhZFd0UE56VndZMnMzUkRCU1FUWkpTbE5vVVVaVFFraE5SWGsxWjJKNVJubzNTbTV0WmpJMU5saGxUMGxTYzA5a1ZBbzVjMmxqVWpGTVkyOVJkM1ZTU1RoaE16bE1SR1UxVnl0RlJXOU9jRmhHYURneVMxcEpiSGxFWWtseVVWVTFXVzFZWm1sUVdITXZPWE5ZTkRCWU9YVXhDbFp3YUhSQmEzQkpRVlJFYzI1NmVWazRkSFZZYTAwelpVeFlURGxMTlZVd1NrOVlla0pQYkdSWmVYRmhTRmM0YUdzMWVETlhTVVJoU3l0clFuUk5UU3NLWXpoV05WaGxOVTlDWVVaSlZrNVBTbWxhWm1SVWIzVm9URWxUZHpSM2FFcGxlRFJzVUM5UEx6TmhVbE5xTm1SbFQzVldPV1pzU0VWMFNFNWxhVEZtVkFwaldGUldXRTE1T0U1b2VIazFUV2w0UjAweU9GTjRaM3B2U1V0elN5dEdRVU5qZHpGSGNVMWlVa05YYW5FMGN6QjRXVVpNVEdKblNrcDZLM2xxV25wV0NtUnZlRmhwTWtsdE4zRlNSSGRpY0VKWVkyMW9ORlV3Y1RVeVYydGtRVVk0WjBVeVZ6aFJVM2RUVm1GSWJuWmxjQzl6VjFWV1VtSlZUR2N3ZG5wTk5Ga0tMMGdyWm5WalUzWXZjVlowVVdWblN6ZFNNVzVLYTNFemRHWnJObWRHTjNOMVUzWkZZVU0zYUU5SlZYRndRbkkzWkhoRmNtWTVZbWs1UlVwV2FFbGxTUW8wWVdwalVsVklLeXRKVUdZMVRWa3lVR3cwTVZKa2FVRXpSVE5tZHpKWk5teGpjMDFEUVZwUlozVkZkSFl6T0haSVRrSm9Rak5pV0doMldGcHBTbkV2Q2psRE5HMU9NRkpRWXpsWVEyUlhWVWhhTmtoSGNEVm1aV1I0UWpGVlkydHZNRVFyT0c1WE0yMUlTVzk2ZDBGb1kwaHRSV1pUVURsTmJFcHZZV0o1WjFrS2NrdEhkelZ6YnpkVFRGaHlZbkJuVldseGNETm9VMElyYVVKTmREUXpPRVpWVkhKSVpraHZNVWRLTjJGQ1UxZHpkRGhEUVRWdmJVUllOa1JFWTI1S1pncEtjM0V3WlZwclVFNXJjMGh0TjNWYVFXZE5Ra0ZCUjJwVmVrSlNUVUl3UjBFeFZXUkVaMUZYUWtKU1RVZDVORkowVTFkdmFESTRPRE14T0RGNVMyRjJDbTVEUW5FeGVrRm1RbWRPVmtoVFRVVkhSRUZYWjBKU1RVZDVORkowVTFkdmFESTRPRE14T0RGNVMyRjJia05DY1RGNlFWQkNaMDVXU0ZKTlFrRm1PRVVLUWxSQlJFRlJTQzlOUVRCSFExTnhSMU5KWWpORVVVVkNRM2RWUVVFMFNVTkJVVUU1U2tsVFoxRjVXVVpzVDBScVJuVnZhRUZ3U0U5Mk1HMTZSVzlFUkFwek5HNWlOVXg1ZEVOQ1NGUmlOV295VTBaTU9XaHlOMUU0YlVNelRUTjBlRVZRWVVGbGJFTXlRbGQzY0RSdWJtZHRka2c0UVVWTFUwMXpVSGRPTjNsTUNtRjZlV05aZUU5eGNUUm9haTlQYVVSNmJFZGtkVEp2YXpCNmJuSk9VRzF0YjFwUFVVMUtPR2xMTDB0aVJsSnpVMWhXZVRRdlNrRjNWVzA1TUd4d1YwRUtOMWR4TVdGRUsxZHZiMWxtVjNJNVFqVlpSRUpNU201b1QxQlNla2cwYTBocmMwbDBNRzVzY210Q1IwOUtSVE5pUm1NMGFYZGplRWQxTlhkTlJGQXhad3BaYUVjM2FXNUNlbWhrYVc1b2NtUnBhR3RHT0hKVFJubG1TV3h2Wm1oTGRVRmlZVmh5UVZkVWNIQkxaWFVyUzA1WmFrTnRUWGhFY0haSWRYaFFVemszQ2pWeFVUSnZNVTh3WWtsMGEwbGlhSGxxYW00MWJERnZUMDQwUlRKUFozWndkbmwzVVVGMGJDOTZVRkprUkN0RlpYUjBVa05HVG5SdWIyeEJVMFEwTWxNS2R6ZGtNV0UyVDBNeFJ6bEdWa0ppYkU1T01WQlNZbHBwVVhncmFYcHJjM1ZOY2xod01IY3pSbU4wY0VKNGFFMHdaekp6Y1c4ek5EaENORVpvYnpVck9RcFdkMWRQTHpkQ1ZFRTNhakJ1SzA5UlFWbGhObGxOWlc5aWEyOWFUV1JhY21SdFVrRTNhamhKUjJGMGVtRnJkR0ZWV2tWak5YSXJLMmxHUTI5cVpITnJDamRqZGpWSU1VeFNlWEl4UkcwM1VrWkVaVVJHTUd4eVJXNHpVWEkxTUhZd2JqUmlOelI2SzBSTk0xRnRiV2RSZVZrck55OUNVV053ZUZOblJYbzRjVGdLU1U1TlZHUkRlREJYYVM5NFJVcHFlVlpLWW1FMk1FOUlSRWxYUkVaaVZrWkNXSFYyYlZGWFVYQmphVTB5UTBWYVZuVkllbU4yZVVSSFZuVnFja1EwTmdvelNqSlRUeXRwUXpncloyd3lhVXBQWlU1V00wcHZWSHBaTTJsTVVXMTRXVmhYYUZjNVVtNVlWRzFYVFZaa1lYbG5kRkF6UjJWRk1qQXZaRll5WXpGVkNuRmlRM01yUlVOYVUyOWtTMmRCUFQwS0xTMHRMUzFGVGtRZ1EwVlNWRWxHU1VOQlZFVXRMUzB0TFE9PQ==  # CA cert used to sign cattle server cert and key
  ---
  apiVersion: v1
  kind: Service
  metadata:
    namespace: cattle-system
    name: cattle-service
    labels:
      app: cattle
  spec:
    ports:
    - port: 80
      targetPort: 80
      protocol: TCP
      name: http
    - port: 443
      targetPort: 443
      protocol: TCP
      name: https
    selector:
      app: cattle
  ---
  apiVersion: extensions/v1beta1
  kind: Ingress
  metadata:
    namespace: cattle-system
    name: cattle-ingress-http
    annotations:
      nginx.ingress.kubernetes.io/proxy-connect-timeout: "30"
      nginx.ingress.kubernetes.io/proxy-read-timeout: "1800"   # Max time in seconds for ws to remain shell window open
      nginx.ingress.kubernetes.io/proxy-send-timeout: "1800"   # Max time in seconds for ws to remain shell window open
  spec:
    rules:
    - host: servidor_cluster  # FQDN to access cattle server
      http:
        paths:
        - backend:
            serviceName: cattle-service
            servicePort: 80
    tls:
    - secretName: cattle-keys-ingress
      hosts:
      - servidor_cluster      # FQDN to access cattle server
  ---
  kind: Deployment
  apiVersion: extensions/v1beta1
  metadata:
    namespace: cattle-system
    name: cattle
  spec:
    replicas: 1
    template:
      metadata:
        labels:
          app: cattle
      spec:
        serviceAccountName: cattle-admin
        containers:
        - image: rancher/rancher:stable
          imagePullPolicy: Always
          name: cattle-server
  #       env:
  #       - name: HTTP_PROXY
  #         value: "http://your_proxy_address:port"
  #       - name: HTTPS_PROXY
  #         value: "http://your_proxy_address:port"
  #       - name: NO_PROXY
  #         value: "localhost,127.0.0.1,0.0.0.0,10.43.0.0/16,your_network_ranges_that_dont_need_proxy_to_access"
          livenessProbe:
            httpGet:
              path: /ping
              port: 80
            initialDelaySeconds: 60
            periodSeconds: 60
          readinessProbe:
            httpGet:
              path: /ping
              port: 80
            initialDelaySeconds: 20
            periodSeconds: 10
          ports:
          - containerPort: 80
            protocol: TCP
          - containerPort: 443
            protocol: TCP
          volumeMounts:
          - mountPath: /etc/rancher/ssl
            name: cattle-keys-volume
            readOnly: true
        volumes:
        - name: cattle-keys-volume
          secret:
            defaultMode: 420
            secretName: cattle-keys-server
services:
  etcd:
    snapshot: true # enables recurring etcd snapshots
    creation: 6h0s # time increment between snapshots
    retention: 24h # time increment before snapshot purge

Steps to Reproduce: Make the tutorial Run Replicated Stateful Application of k8s docs.

Link to docs: https://kubernetes.io/docs/tasks/run-application/run-replicated-stateful-application/

Results: After to create the statefulset (with implicit PersistentVolume and PersistentVolumeClaim that aren't in the tutorial). the replica of the pod mysql-0 won't to be created and will report CrashLoopBackOff status with the log below:

$ kbk logs pod/mysql-0 
Error from server (BadRequest): a container name must be specified for pod mysql-0, choose one of: [mysql xtrabackup] or one of the init containers: [init-mysql clone-mysql]

I've discovered that this is a bug of docker with IPTABLES in the node host and are specified in Upgrading docker 1.13 on nodes causes outbound container traffic to stop working issue of kubernetes.

But when I tried to change default iptables forward (running iptables -P FORWARD ACCEPT on the node - virtualbox VM) nothing happed and still persists. I try to disable ufw firewall to, but again, persists.

rancher / rke

iptables forward doesn't work for mysql replicated stateful application #912