Closed ruhengChen closed 4 months ago
你好,可以附件上传一下日志信息,以便更好排查~
你好,请检查一下磁盘空间和内存空间~
[root@ecs-46f7 ~]# df -h Filesystem Size Used Avail Use% Mounted on devtmpfs 7.5G 0 7.5G 0% /dev tmpfs 7.7G 0 7.7G 0% /dev/shm tmpfs 7.7G 20M 7.7G 1% /run tmpfs 7.7G 0 7.7G 0% /sys/fs/cgroup /dev/vda2 39G 25G 12G 69% / tmpfs 7.7G 64K 7.7G 1% /tmp /dev/vda1 1022M 5.8M 1017M 1% /boot/efi tmpfs 1.6G 0 1.6G 0% /run/user/0 overlay 39G 25G 12G 69% /var/lib/docker/overlay2/00626d91713bc9ff720b57033ce00003090fabe691ed1c1ac3c5012cb7f8ba90/merged overlay 39G 25G 12G 69% /var/lib/docker/overlay2/bcc69f619a98555dce1835b216eae462edfb8086a23fb6984262d4814c9152cb/merged overlay 39G 25G 12G 69% /var/lib/docker/overlay2/939b60b3c97410e6bc150475b870cc360303938dc1a68e72aa233505bfc01173/merged overlay 39G 25G 12G 69% /var/lib/docker/overlay2/0c16f58d56413cdd16e5de8a70df48b9b5297eec6bba71f6f28020a63069917c/merged overlay 39G 25G 12G 69% /var/lib/docker/overlay2/71ea8c68e821b82958497bbf9fcd789cf0b6833a8a273e074957551938f579a9/merged overlay 39G 25G 12G 69% /var/lib/docker/overlay2/ac61fe3b656c4bbfae107c21fe109ebe03d66224db3b8d75f31dc0e184c38290/merged [root@ecs-46f7 ~]# free -m total used free shared buff/cache available Mem: 15760 4425 1487 9319 9848 523 Swap: 0 0 0
容器里执行下: kubectl describe node 看下
[root@root-kuscia-autonomy-bob-ecs-46f7 kuscia]# kubectl describe node
Name: root-kuscia-autonomy-bob-ecs-46f7
Roles: agent
Labels: beta.kubernetes.io/arch=aarch64
beta.kubernetes.io/os=linux
domain=bob
kubernetes.io/apiVersion=0.26.6
kubernetes.io/arch=aarch64
kubernetes.io/hostname=root-kuscia-autonomy-bob-ecs-46f7
kubernetes.io/os=linux
kubernetes.io/role=agent
kuscia.secretflow/namespace=bob
kuscia.secretflow/runtime=runc
Annotations: node.alpha.kubernetes.io/ttl: 0
CreationTimestamp: Mon, 15 Jul 2024 18:17:58 +0800
Taints: kuscia.secretflow/agent=v1:NoSchedule
node.kubernetes.io/memory-pressure:NoSchedule
Unschedulable: false
Lease:
HolderIdentity: root-kuscia-autonomy-bob-ecs-46f7
AcquireTime:
NetworkUnavailable False Tue, 16 Jul 2024 13:24:47 +0800 Tue, 16 Jul 2024 13:24:47 +0800 RouteCreated RouteController created a route PIDPressure False Tue, 16 Jul 2024 13:24:47 +0800 Tue, 16 Jul 2024 13:24:47 +0800 AgentHasSufficientPID Agent has sufficient PID available MemoryPressure True Tue, 16 Jul 2024 15:38:18 +0800 Tue, 16 Jul 2024 13:24:47 +0800 AgentHasMemoryPressure Memory is about to run out, total=15.4GB, available=576.5MB DiskPressure False Tue, 16 Jul 2024 15:38:18 +0800 Tue, 16 Jul 2024 13:24:47 +0800 AgentHasNoDiskPressure Agent has no disk pressure. @agent_volume: space=24.6GB/36.2GB(68.1%) inode=278.3k/2.6M(10.9%) OutOfDisk False Tue, 16 Jul 2024 15:38:18 +0800 Tue, 16 Jul 2024 13:24:47 +0800 AgentHasSufficientDisk Agent has sufficient disk space available. @agent_volume: free_space=11.5GB, free_inode=2.3M Ready True Tue, 16 Jul 2024 15:38:18 +0800 Tue, 16 Jul 2024 13:24:47 +0800 AgentReady Agent is ready Addresses: InternalIP: 172.18.0.6 Capacity: cpu: 8 memory: 16139200Ki pods: 500 storage: 39988608Ki Allocatable: cpu: 8 memory: 459520Ki pods: 500 storage: 11986828Ki System Info: Machine ID: 94118d27-3e36-4e55-bf94-945fb0e7ead3 System UUID: Boot ID: 1721107223-1721107487570958605 Kernel Version: 4.19.90-17.5.ky10.aarch64 OS Image: docker://linux/anolis:23 (guest) Operating System: linux Architecture: aarch64 Container Runtime Version: Kubelet Version: v0.9.0b0-2-g7663c20 Kube-Proxy Version: PodCIDR: 10.42.0.0/24 PodCIDRs: 10.42.0.0/24 Non-terminated Pods: (0 in total) Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits Age
Allocated resources: (Total limits may be over 100 percent, i.e., overcommitted.) Resource Requests Limits
cpu 0 (0%) 0 (0%)
memory 0 (0%) 0 (0%)
ephemeral-storage 0 (0%) 0 (0%)
storage 0 0
Events:
你好,节点信息中显示:node.kubernetes.io/memory-pressure:NoSchedule 内存不足所导致的,kuscia最少需要6g内存,如果使用docker环境运行 请使用docker update --memory 调整内存资源
好的 感谢~
你好,我这边已经把内存调整到6g了,但是还是显示 node.kubernetes.io/memory-pressure:NoSchedule 请问该怎么解决
你好,调整资源分配后重新启动一下镜像~
已经重新启动过了
可以了 应该还是内存不够的原因
Issue Type
Install/Deploy
Search for existing issues similar to yours
Yes
OS Platform and Distribution
Linux ecs-46f7 4.19.90-17.5.ky10.aarch64 #1 SMP Fri Aug 7 13:35:33 CST 2020 aarch64 aarch64 aarch64 GNU/Linux
Kuscia Version
0.9.0b0
Deployment
docker
deployment Version
24.0.8
App Running type
secretflow
App Running version
1.7.0b0
Configuration file used to run kuscia.
What happend and What you expected to happen.
Kuscia log output.