search

sipeed / LonganPi-3H-SDK

LonganPi 3H SDK
GNU General Public License v3.0
28 stars 14 forks source link

hard time installing kubernetes distributions in defualt image (k3s / k0s) #26

Closed lonerkim closed 4 months ago

lonerkim commented 6 months ago

When I tried to k0s install and start services, this happened

Error: pre-flight checks failed, check out `k0s sysinfo`: lookup localhost on 8.8.8.8:53: no such host

so I tried to edit my /etc/hosts to include 

127.0.0.1       localhost lpi3h-c6dd
::1             localhost ip6-localhost ip6-loopback

and service starts

but then when I tried to join my worker nodes, below happened.

root@lpi3h-(devicename)~# sudo k0s worker "mytoken"
Error: pre-flight checks failed, check out `k0s sysinfo`: failed to load/attach eBPF device filter program: load ext_infos: detect support for Program BTF (func/line_info): detect support for BTF: function not implemented
root@lpi3h-(devicename):~# k0s sysinfo

Machine ID: "(machine ID)" (from machine) (pass)
Total memory: 3.9 GiB (pass)
Disk space available for /var/lib/k0s: 24.6 GiB (pass)
Name resolution: localhost: [127.0.0.1 ::1] (pass)
Operating system: Linux (pass)
  Linux kernel release: 6.7.0-rc3+ (pass)
  Max. file descriptors per process: current: 1048576 / max: 1048576 (pass)
  AppArmor: unavailable (pass)
  Executable in PATH: modprobe: /usr/sbin/modprobe (pass)
  Executable in PATH: mount: /usr/bin/mount (pass)
  Executable in PATH: umount: /usr/bin/umount (pass)
  /proc file system: mounted (0x9fa0) (pass)
  Control Groups: version 2 (pass)
    cgroup controller "cpu": available (is a listed root controller) (pass)
    cgroup controller "cpuacct": available (via cpu in version 2) (pass)
    cgroup controller "cpuset": available (is a listed root controller) (pass)
    cgroup controller "memory": available (is a listed root controller) (pass)
    cgroup controller "devices": error: failed to load/attach eBPF device filter program: load ext_infos: detect support for Program BTF (func/line_info): detect support for BTF: function not implemented
    cgroup controller "freezer": available (cgroup.freeze exists) (pass)
    cgroup controller "pids": available (is a listed root controller) (pass)
    cgroup controller "hugetlb": unavailable (warning)
    cgroup controller "blkio": available (via io in version 2) (pass)
  CONFIG_CGROUPS: Control Group support: no kernel config found (warning)
  CONFIG_NAMESPACES: Namespaces support: no kernel config found (warning)
  CONFIG_NET: Networking support: no kernel config found (warning)
  CONFIG_EXT4_FS: The Extended 4 (ext4) filesystem: no kernel config found (warning)
  CONFIG_PROC_FS: /proc file system support: no kernel config found (warning)
Error: sysinfo failed

so I tried other distribution (k3s)

and this happened

root@lpi3h-(devicename):~# curl -sfL https://get.k3s.io | sh -
[INFO]  Finding release for channel stable
[INFO]  Using v1.28.6+k3s2 as release
[INFO]  Downloading hash https://github.com/k3s-io/k3s/releases/download/v1.28.6+k3s2/sha256sum-arm64.txt
[INFO]  Downloading binary https://github.com/k3s-io/k3s/releases/download/v1.28.6+k3s2/k3s-arm64
[INFO]  Verifying binary download
[INFO]  Installing k3s to /usr/local/bin/k3s
sh: 581: [: Illegal number:
[INFO]  Skipping installation of SELinux RPM
[INFO]  Creating /usr/local/bin/kubectl symlink to k3s
[INFO]  Creating /usr/local/bin/crictl symlink to k3s
[INFO]  Creating /usr/local/bin/ctr symlink to k3s
[INFO]  Creating killall script /usr/local/bin/k3s-killall.sh
[INFO]  Creating uninstall script /usr/local/bin/k3s-uninstall.sh
[INFO]  env: Creating environment file /etc/systemd/system/k3s.service.env
[INFO]  systemd: Creating service file /etc/systemd/system/k3s.service
[INFO]  systemd: Enabling k3s unit
Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service.
iptables-restore/1.8.10 (nf_tables) Failed to initialize nft: Protocol not supported

so I tried

update-alternatives --set iptables /usr/sbin/iptables-legacy
update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy
update-alternatives --set arptables /usr/sbin/arptables-legacy
update-alternatives --set ebtables /usr/sbin/ebtables-legacy
=========================================
update-alternatives: using /usr/sbin/iptables-legacy to provide /usr/sbin/iptables (iptables) in manual mode
update-alternatives: using /usr/sbin/ip6tables-legacy to provide /usr/sbin/ip6tables (ip6tables) in manual mode
update-alternatives: error: alternative /usr/sbin/arptables-legacy for arptables not registered; not setting
update-alternatives: error: alternative /usr/sbin/ebtables-legacy for ebtables not registered; not setting

it seems can start a service at first, but when I checked status

root@lpi3h-(devicename):~# systemctl status k3s
● k3s.service - Lightweight Kubernetes
     Loaded: loaded (/etc/systemd/system/k3s.service; enabled; preset: enabled)
     Active: active (running) since Wed 2024-02-28 22:36:11 CST; 20s ago
       Docs: https://k3s.io
    Process: 776 ExecStartPre=/bin/sh -xc ! /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service 2>/dev/null (code=exited, status=0/SUCCESS)
    Process: 779 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=1/FAILURE)
    Process: 782 ExecStartPre=/sbin/modprobe overlay (code=exited, status=1/FAILURE)
   Main PID: 784 (k3s-server)
      Tasks: 26
     Memory: 549.0M (peak: 554.3M)
        CPU: 1min 31.407s
     CGroup: /system.slice/k3s.service
             └─784 "/usr/local/bin/k3s server"

Feb 28 22:36:31 lpi3h--(devicename)k3s[784]: time="2024-02-28T22:36:31+08:00" level=info msg="Waiting for control-plane node lpi3h-(devicename)startup: nodes \"lpi3h-(devicename)\" not found"
Feb 28 22:36:31 lpi3h-(devicename)k3s[784]: I0228 22:36:31.928469     784 shared_informer.go:318] Caches are synced for garbage collector
Feb 28 22:36:31 lpi3h-(devicename)k3s[784]: I0228 22:36:31.936853     784 shared_informer.go:318] Caches are synced for garbage collector
Feb 28 22:36:31 lpi3h-(devicename)k3s[784]: I0228 22:36:31.936943     784 garbagecollector.go:166] "All resource monitors have synced. Proceeding to collect garbage"
Feb 28 22:36:32 lpi3h-(devicename)k3s[784]: I0228 22:36:32.142245     784 event.go:307] "Event occurred" object="kube-system/local-path-provisioner-84db5d44d9" fieldPath="" kind="ReplicaSet" apiVersion="apps/v1" type="Normal" reason="SuccessfulCreate" message="Created pod: local-path-p>
Feb 28 22:36:32 lpi3h-(devicename)k3s[784]: I0228 22:36:32.172260     784 event.go:307] "Event occurred" object="kube-system/coredns-6799fbcd5" fieldPath="" kind="ReplicaSet" apiVersion="apps/v1" type="Normal" reason="SuccessfulCreate" message="Created pod: coredns-6799fbcd5-v7hrv"
Feb 28 22:36:32 lpi3h-(devicename)k3s[784]: W0228 22:36:32.188201     784 handler_proxy.go:93] no RequestInfo found in the context
Feb 28 22:36:32 lpi3h-(devicename)k3s[784]: E0228 22:36:32.188476     784 controller.go:146] Error updating APIService "v1beta1.metrics.k8s.io" with err: failed to download v1beta1.metrics.k8s.io: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
Feb 28 22:36:32 lpi3h-(devicename)k3s[784]: , Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
Feb 28 22:36:32 lpi3h-(devicename)k3s[784]: E0228 22:36:32.188747     784 handler_proxy.go:137] error resolving kube-system/metrics-server: no endpoints available for service "metrics-server"
Feb 28 22:36:32 lpi3h-(devicename)k3s[784]: I0228 22:36:32.199985     784 replica_set.go:676] "Finished syncing" kind="ReplicaSet" key="kube-system/local-path-provisioner-84db5d44d9" duration="689.235263ms"

also when I checked br_netfilter / overlay myself

root@lpi3h-(devicename):~# sudo modprobe br_netfilter sudo modprobe overlay
modprobe: FATAL: Module br_netfilter not found in directory /lib/modules/6.7.0-rc3+
modprobe: FATAL: Module overlay not found in directory /lib/modules/6.7.0-rc3+

this came out. br_netfilter and overlay / eBPF, BTF is not supported I guess?

wu-yue-yu commented 6 months ago

You can check this pr #27 to enable some kernel configs.

lonerkim commented 6 months ago

keep failed to build an image from sdk so cannot try further at the moment. will try later. thanks for support anyway.

2024년 3월 1일 (금) 오후 3:31, ztd @.***>님이 작성:

You can check this pr #27 https://github.com/sipeed/LonganPi-3H-SDK/pull/27 to enable some kernel configs.

— Reply to this email directly, view it on GitHub https://github.com/sipeed/LonganPi-3H-SDK/issues/26#issuecomment-1972592464, or unsubscribe https://github.com/notifications/unsubscribe-auth/AISRXLGGMUERMJMPEDL2SPLYWAODVAVCNFSM6AAAAABD7XFXR2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNZSGU4TENBWGQ . You are receiving this because you authored the thread.Message ID: @.***>

lonerkim commented 4 months ago

seems some moudle activaton with recent patch for 'main' sucessfully activate k3s.services still not to test anything due to #30 on my side issue, but there was major progress to share. (thus issue is still open till more detailed test) ;) thanks for helping me 

sipeed@lpi3h-c6dd:~$ systemctl status k3s
● k3s.service - Lightweight Kubernetes
     Loaded: loaded (/etc/systemd/system/k3s.service; enabled; preset: enabled)
     Active: active (running) since Mon 2024-04-08 22:50:08 KST; 4min 1s ago
       Docs: https://k3s.io
   Main PID: 1824 (k3s-server)
      Tasks: 29
     Memory: 586.4M (peak: 613.5M)
        CPU: 7min 12.762s
     CGroup: /system.slice/k3s.service
             └─1824 "/usr/local/bin/k3s server"
sipeed@lpi3h-c6dd:~$ k3s check-config
cat: /sys/kernel/security/apparmor/profiles: No such file or directory

Verifying binaries in /var/lib/rancher/k3s/data/15bd42671378bf2abf3d83fb74cd4854c083c3c857c6bead9802329b4f1fc5d3/bin:
- sha256sum: good
- aux/ip6tables: symlink to xtables-legacy-multi
- aux/ip6tables-restore: symlink to xtables-legacy-multi
- aux/ip6tables-save: symlink to xtables-legacy-multi
- aux/iptables: symlink to xtables-legacy-multi
- aux/iptables-restore: symlink to xtables-legacy-multi
- aux/iptables-save: symlink to xtables-legacy-multi
- links: good

System:
- /var/lib/rancher/k3s/data/15bd42671378bf2abf3d83fb74cd4854c083c3c857c6bead9802329b4f1fc5d3/bin/aux iptables v1.8.8 (legacy): ok
- swap: disabled
- routes: ok

Limits:
- /proc/sys/kernel/keys/root_maxkeys: 1000000

modprobe: FATAL: Module configs not found in directory /lib/modules/6.7.0-rc3+
info: reading kernel config from /boot/config-6.7.0-rc3+ ...

Generally Necessary:
- cgroup hierarchy: cgroups V2 mounted, cpu|cpuset|memory controllers status: good
- CONFIG_NAMESPACES: enabled
- CONFIG_NET_NS: enabled
- CONFIG_PID_NS: enabled
- CONFIG_IPC_NS: enabled
- CONFIG_UTS_NS: enabled
- CONFIG_CGROUPS: enabled
- CONFIG_CGROUP_PIDS: enabled
- CONFIG_CGROUP_CPUACCT: enabled
- CONFIG_CGROUP_DEVICE: enabled
- CONFIG_CGROUP_FREEZER: enabled
- CONFIG_CGROUP_SCHED: enabled
- CONFIG_CPUSETS: enabled
- CONFIG_MEMCG: enabled
- CONFIG_KEYS: enabled
- CONFIG_VETH: enabled (as module)
- CONFIG_BRIDGE: enabled
- CONFIG_BRIDGE_NETFILTER: enabled
- CONFIG_IP_NF_FILTER: enabled
- CONFIG_IP_NF_TARGET_MASQUERADE: enabled
- CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: enabled
- CONFIG_NETFILTER_XT_MATCH_CONNTRACK: enabled
- CONFIG_NETFILTER_XT_MATCH_IPVS: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_COMMENT: enabled
- CONFIG_NETFILTER_XT_MATCH_MULTIPORT: enabled
- CONFIG_IP_NF_NAT: enabled
- CONFIG_NF_NAT: enabled
- CONFIG_POSIX_MQUEUE: enabled

Optional Features:
- CONFIG_USER_NS: enabled
- CONFIG_SECCOMP: enabled
- CONFIG_BLK_CGROUP: enabled
- CONFIG_BLK_DEV_THROTTLING: enabled
- CONFIG_CGROUP_PERF: enabled
- CONFIG_CGROUP_HUGETLB: enabled
- CONFIG_NET_CLS_CGROUP: missing
- CONFIG_CGROUP_NET_PRIO: enabled
- CONFIG_CFS_BANDWIDTH: enabled
- CONFIG_FAIR_GROUP_SCHED: enabled
- CONFIG_RT_GROUP_SCHED: enabled
- CONFIG_IP_NF_TARGET_REDIRECT: enabled (as module)
- CONFIG_IP_SET: enabled (as module)
- CONFIG_IP_VS: enabled (as module)
- CONFIG_IP_VS_NFCT: enabled
- CONFIG_IP_VS_PROTO_TCP: missing
- CONFIG_IP_VS_PROTO_UDP: missing
- CONFIG_IP_VS_RR: enabled (as module)
- CONFIG_EXT4_FS: enabled
- CONFIG_EXT4_FS_POSIX_ACL: enabled
- CONFIG_EXT4_FS_SECURITY: enabled
- Network Drivers:
  - "overlay":
    - CONFIG_VXLAN: enabled
      Optional (for encrypted networks):
      - CONFIG_CRYPTO: enabled
      - CONFIG_CRYPTO_AEAD: enabled
      - CONFIG_CRYPTO_GCM: enabled
      - CONFIG_CRYPTO_SEQIV: enabled
      - CONFIG_CRYPTO_GHASH: enabled
      - CONFIG_XFRM: enabled
      - CONFIG_XFRM_USER: enabled
      - CONFIG_XFRM_ALGO: enabled
      - CONFIG_INET_ESP: enabled
      - CONFIG_INET_XFRM_MODE_TRANSPORT: missing
- Storage Drivers:
  - "overlay":
    - CONFIG_OVERLAY_FS: enabled

STATUS: pass
lonerkim commented 4 months ago

image

k3s is running on LonganPi 3H Ubuntu-CLI addtional configs I tried on

seems there isn't signifiacnt issue for now, I'll mark this issues as cloesed. thanks for help.