OLCNE: Errors in /var/log/messages after system initlization

[hussam-qasem]

Describe the issue

1. Numerous error logs are printed in /var/log/messages after system initialization (details below)
2. Run the provisioning script as non-root (user Vagrant) to match the instructions laid our in the Getting Started guide.
3. Change the ssh LogLevel to FATAL (instead of QUIET) which can mask genuine connectivity errors.

Environment (please complete the following information):

• Host OS: macOS Monterey 12.3.1
• Kernel version: Darwin Kernel Version 21.4.0
• Vagrant version: 2.2.19
• Vagrant provider: VirtualBox version: 6.1.3r149290
• Vagrant project: OLCNE

Additional information

• On Kubernetes Worker Nodes. Missing Kubernetes manifests directory. Error:

  kubelet: "Unable to read config path" err="path does not exist, ignoring" path="/etc/kubernetes/manifests"

  To fix:

  mkdir /etc/kubernetes/manifests

  ---

• On Kubernetes Master & Worker Nodes. Error:

  kubelet: summary_sys_containers.go: "Failed to get system container stats"
       err='failed to get cgroup stats for "/system.slice/kubelet.service":
            failed to get container info for "/system.slice/kubelet.service":
            unknown container "/system.slice/kubelet.service"'
       containerName="/system.slice/kubelet.service"

  To fix:

  
  cat <<EOF | sudo tee /etc/systemd/system/kubelet.service.d/11-cgroups.conf
  [Service]
  CPUAccounting=true
  MemoryAccounting=true
  EOF

sudo systemctl daemon-reload sudo systemctl restart kubelet

---
* On Kubernetes **Master** & **Worker** Nodes. AVC denial on iptables. Error:

audit: type=1400 avc: denied { ioctl } for comm="iptables" path="/sys/fs/cgroup" dev="tmpfs"

To fix:
```Shell
echo '(allow iptables_t cgroup_t (dir (ioctl)))' > /tmp/local_iptables.cil
sudo semodule -i /tmp/local_iptables.cil
rm -f /tmp/local_iptables.cil

---

• On Kubernetes Master Nodes, where MULTI_MASTER=true and in-built load balancer / virtual-ip. Error:

  Keepalived_vrrp: (VI_1) WARNING - equal priority advert received from remote host with our IP address.

  This is due to the unicast_src_ip being in the unicast_peers. To fix:

  1. Edit /etc/keepalived/keepalived.conf; find the unicast_src_ip value and remove it from the unicast_peers section.
  2. Reload the service:

     sudo systemctl restart keepalived.service

     ---

• On Kubernetes Master Nodes. crio common group Error:

  kubelet: manager.go: Failed to create existing container: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod*.slice/crio-*.scope:
                   Error finding container *: 
                   Status 404 returned error &{%!s(*http.body=&{0xc000f5c0a8 <nil> <nil> false false {0 0} false false false <nil>}) {%!s(int32=0) %!s(uint32=0)} %!s(bool=false) <nil> %!s(func(error) error=0x77c300) %!s(func() error=0x77c280)}

  To fix:

• Reboot your Master node(s) one at a time.

---

I'll submit a PR shortly to add the fixes above in the fixups() function of the provisioning script.

[hussam-qasem]

Thank you @rafabene @scoter-oracle