smart-edge-open / converged-edge-experience-kits

Source code for experience kits with Ansible-based deployment.
Apache License 2.0
37 stars 40 forks source link

RMD bootstraping fails. #45

Closed rsys-asakalab closed 3 years ago

rsys-asakalab commented 4 years ago

default intel-rmd-operator-5656d64798-nv9ql 1/1 Running 0 172m 10.16.0.23 node01 default rmd-node-agent-node01 1/1 Running 0 3h15m 10.16.0.22 node01 default rmd-node01 0/1 CrashLoopBackOff 37 172m 10.16.0.18 node01

Error logs of rmd-node01 [root@controller01 ~]# kubectl logs rmd-node01 Resctrl mount or Mount path failed: false

[root@node01 ~]# ls /sys/fs/ bpf btrfs cgroup ext4 fuse pstore resctrl selinux xfs [root@node01 ~]#

[root@node01 ~]# cat /proc/filesystems | grep resctrl nodev resctrl [root@node01 ~]#

  1. Additionally change in golang version upgrade to go1.13.14 in openness configuration required.
  2. I also see mismatch in code vs rmd-pod.yaml. /etc/rmd/rmd.toml: sysresctrl = "/sys/fs/resctrl"

rmd/src/utils/proc/proc.go … 63 // IsResctrlMounted Checks if ResCtrl is mounted and if the path is valid 64 func IsResctrlMounted(flag string) bool { 65 f, err := os.Open(Mounts) …
263 //check if resctrl is mounted 264 isresctrlenabled := IsResctrlMounted(ResctrlPath) 265 if isresctrlenabled != true { 266 fmt.Println("Resctrl mount or Mount path failed:", isresctrlenabled)

rmd-pod.yaml … volumeMounts:

Mounting resctrl didn't help. mount -t resctrl resctrl /sys/fs/resctrl

https://github.com/open-ness/openness-experience-kits/blob/fa5039031d1e639c1695d500d26995aed07d650f/network_edge.yml#L107

archie951 commented 4 years ago

Hi @discoveravis ,

This problem because of the mount issue can be resolved by doing below workarounds.

  1. Run the cleanup script first.

  2. Append rdt=l3cat to the correct grub config on the node. (openness-experience-kits/group_vars/edgenode_group/10-default.yml ). _default_grub_params: "default_hugepagesz={{ hugepage_size }} hugepagesz={{ hugepage_size }} hugepages={{ hugepage_amount }} inteliommu=on iommu=pt rdt=l3cat"

  3. Addition of resctrl /sys/fs/resctrl resctrl in /etc/fstab (for fixing the mount issue)

cat /etc/fstab

resctrl /sys/fs/resctrl resctrl

Please let me know if this fixes your issue.

Regards, Archit