Open stat0s2p opened 2 years ago
- 请确认prod-job-job-master是否成功启动
- prod-job-job-master异常通常是es未成功启动,请确认sreworks-dataops这个ns下的es是否成功启动
prod-job-job-master启动是成功的.下面是其中一个pod的log:
[2022-03-24 18:11:26 679] INFO [scheduling-1][monitor:38]- >>>monitorJob|threadPool|inst=0, node=0, job=0
[2022-03-24 18:11:28 678] INFO [scheduling-1][monitor:32]- >>>monitorJob|cost|inst=[0:0.0], nodeStart=[0:0.0], job=[0:0.0], task=[0:0.0], stdout=[0:0.0], status=[0:0.0]
[2022-03-24 18:11:28 678] INFO [scheduling-1][monitor:38]- >>>monitorJob|threadPool|inst=0, node=0, job=0
[2022-03-24 18:11:30 678] INFO [scheduling-1][monitor:32]- >>>monitorJob|cost|inst=[0:0.0], nodeStart=[0:0.0], job=[0:0.0], task=[0:0.0], stdout=[0:0.0], status=[0:0.0]
[2022-03-24 18:11:30 678] INFO [scheduling-1][monitor:38]- >>>monitorJob|threadPool|inst=0, node=0, job=0
[2022-03-24 18:11:31 677] INFO [scheduling-1][c.a.s.j.m.j.d.DagInstFixedRateSchedule:38]- action=taskFlowDispatchJob|execute|exit|timeoutCount=0
[2022-03-24 18:11:32 678] INFO [scheduling-1][monitor:32]- >>>monitorJob|cost|inst=[0:0.0], nodeStart=[0:0.0], job=[0:0.0], task=[0:0.0], stdout=[0:0.0], status=[0:0.0]
[2022-03-24 18:11:32 678] INFO [scheduling-1][monitor:38]- >>>monitorJob|threadPool|inst=0, node=0, job=0
[2022-03-24 18:11:34 677] INFO [scheduling-1][monitor:32]- >>>monitorJob|cost|inst=[0:0.0], nodeStart=[0:0.0], job=[0:0.0], task=[0:0.0], stdout=[0:0.0], status=[0:0.0]
[2022-03-24 18:11:34 678] INFO [scheduling-1][monitor:38]- >>>monitorJob|threadPool|inst=0, node=0, job=0
[2022-03-24 18:11:36 677] INFO [scheduling-1][c.a.s.j.m.j.d.DagInstFixedRateSchedule:38]- action=taskFlowDispatchJob|execute|exit|timeoutCount=0
[2022-03-24 18:11:36 679] INFO [scheduling-1][monitor:32]- >>>monitorJob|cost|inst=[0:0.0], nodeStart=[0:0.0], job=[0:0.0], task=[0:0.0], stdout=[0:0.0], status=[0:0.0]
[2022-03-24 18:11:36 679] INFO [scheduling-1][monitor:38]- >>>monitorJob|threadPool|inst=0, node=0, job=0
[2022-03-24 18:11:38 678] INFO [scheduling-1][monitor:32]- >>>monitorJob|cost|inst=[0:0.0], nodeStart=[0:0.0], job=[0:0.0], task=[0:0.0], stdout=[0:0.0], status=[0:0.0]
[2022-03-24 18:11:38 678] INFO [scheduling-1][monitor:38]- >>>monitorJob|threadPool|inst=0, node=0, job=0
[2022-03-24 18:11:40 678] INFO [scheduling-1][monitor:32]- >>>monitorJob|cost|inst=[0:0.0], nodeStart=[0:0.0], job=[0:0.0], task=[0:0.0], stdout=[0:0.0], status=[0:0.0]
[2022-03-24 18:11:40 679] INFO [scheduling-1][monitor:38]- >>>monitorJob|threadPool|inst=0, node=0, job=0
[2022-03-24 18:11:41 681] INFO [scheduling-1][c.a.s.j.m.j.d.DagInstFixedRateSchedule:38]- action=taskFlowDispatchJob|execute|exit|timeoutCount=0
[2022-03-24 18:11:42 678] INFO [scheduling-1][monitor:32]- >>>monitorJob|cost|inst=[0:0.0], nodeStart=[0:0.0], job=[0:0.0], task=[0:0.0], stdout=[0:0.0], status=[0:0.0]
[2022-03-24 18:11:42 678] INFO [scheduling-1][monitor:38]- >>>monitorJob|threadPool|inst=0, node=0, job=0
[2022-03-24 18:11:44 678] INFO [scheduling-1][monitor:32]- >>>monitorJob|cost|inst=[0:0.0], nodeStart=[0:0.0], job=[0:0.0], task=[0:0.0], stdout=[0:0.0], status=[0:0.0]
[2022-03-24 18:11:44 678] INFO [scheduling-1][monitor:38]- >>>monitorJob|threadPool|inst=0, node=0, job=0
[2022-03-24 18:11:46 677] INFO [scheduling-1][c.a.s.j.m.j.d.DagInstFixedRateSchedule:38]- action=taskFlowDispatchJob|execute|exit|timeoutCount=0
[2022-03-24 18:11:46 678] INFO [scheduling-1][monitor:32]- >>>monitorJob|cost|inst=[0:0.0], nodeStart=[0:0.0], job=[0:0.0], task=[0:0.0], stdout=[0:0.0], status=[0:0.0]
[2022-03-24 18:11:46 678] INFO [scheduling-1][monitor:38]- >>>monitorJob|threadPool|inst=0, node=0, job=0
[2022-03-24 18:11:48 678] INFO [scheduling-1][monitor:32]- >>>monitorJob|cost|inst=[0:0.0], nodeStart=[0:0.0], job=[0:0.0], task=[0:0.0], stdout=[0:0.0], status=[0:0.0]
[2022-03-24 18:11:48 678] INFO [scheduling-1][monitor:38]- >>>monitorJob|threadPool|inst=0, node=0, job=0
[2022-03-24 18:11:50 678] INFO [scheduling-1][monitor:32]- >>>monitorJob|cost|inst=[0:0.0], nodeStart=[0:0.0], job=[0:0.0], task=[0:0.0], stdout=[0:0.0], status=[0:0.0]
[2022-03-24 18:11:50 678] INFO [scheduling-1][monitor:38]- >>>monitorJob|threadPool|inst=0, node=0, job=0
[2022-03-24 18:11:51 677] INFO [scheduling-1][c.a.s.j.m.j.d.DagInstFixedRateSchedule:38]- action=taskFlowDispatchJob|execute|exit|timeoutCount=0
[2022-03-24 18:11:52 680] INFO [scheduling-1][monitor:32]- >>>monitorJob|cost|inst=[0:0.0], nodeStart=[0:0.0], job=[0:0.0], task=[0:0.0], stdout=[0:0.0], status=[0:0.0]
[2022-03-24 18:11:52 680] INFO [scheduling-1][monitor:38]- >>>monitorJob|threadPool|inst=0, node=0, job=0
[2022-03-24 18:11:54 678] INFO [scheduling-1][monitor:32]- >>>monitorJob|cost|inst=[0:0.0], nodeStart=[0:0.0], job=[0:0.0], task=[0:0.0], stdout=[0:0.0], status=[0:0.0]
[2022-03-24 18:11:54 678] INFO [scheduling-1][monitor:38]- >>>monitorJob|threadPool|inst=0, node=0, job=0
[2022-03-24 18:11:56 677] INFO [scheduling-1][c.a.t.d.services.DagInstClearService:41]- >>>dagInstClearService|clearDataBefore|exit|costTime=4, delete4DagInst=0, delete4DagInstNode=0, delete4DagInstEdge=0, delete4DagInstNodeStd=0
[2022-03-24 18:11:56 678] INFO [scheduling-1][c.a.s.j.m.j.d.DagInstFixedRateSchedule:38]- action=taskFlowDispatchJob|execute|exit|timeoutCount=0
[2022-03-24 18:11:56 680] INFO [scheduling-1][monitor:32]- >>>monitorJob|cost|inst=[0:0.0], nodeStart=[0:0.0], job=[0:0.0], task=[0:0.0], stdout=[0:0.0], status=[0:0.0]
[2022-03-24 18:11:56 680] INFO [scheduling-1][monitor:38]- >>>monitorJob|threadPool|inst=0, node=0, job=0
[2022-03-24 18:11:58 677] INFO [scheduling-1][monitor:32]- >>>monitorJob|cost|inst=[0:0.0], nodeStart=[0:0.0], job=[0:0.0], task=[0:0.0], stdout=[0:0.0], status=[0:0.0]
[2022-03-24 18:11:58 677] INFO [scheduling-1][monitor:38]- >>>monitorJob|threadPool|inst=0, node=0, job=0
[2022-03-24 18:12:00 679] INFO [scheduling-1][monitor:32]- >>>monitorJob|cost|inst=[0:0.0], nodeStart=[0:0.0], job=[0:0.0], task=[0:0.0], stdout=[0:0.0], status=[0:0.0]
[2022-03-24 18:12:00 680] INFO [scheduling-1][monitor:38]- >>>monitorJob|threadPool|inst=0, node=0, job=0
[2022-03-24 18:12:01 677] INFO [scheduling-1][c.a.s.j.m.j.d.DagInstFixedRateSchedule:38]- action=taskFlowDispatchJob|execute|exit|timeoutCount=0
[2022-03-24 18:12:02 678] INFO [scheduling-1][monitor:32]- >>>monitorJob|cost|inst=[0:0.0], nodeStart=[0:0.0], job=[0:0.0], task=[0:0.0], stdout=[0:0.0], status=[0:0.0]
[2022-03-24 18:12:02 678] INFO [scheduling-1][monitor:38]- >>>monitorJob|threadPool|inst=0, node=0, job=0
[2022-03-24 18:12:04 678] INFO [scheduling-1][monitor:32]- >>>monitorJob|cost|inst=[0:0.0], nodeStart=[0:0.0], job=[0:0.0], task=[0:0.0], stdout=[0:0.0], status=[0:0.0]
[2022-03-24 18:12:04 678] INFO [scheduling-1][monitor:38]- >>>monitorJob|threadPool|inst=0, node=0, job=0
[2022-03-24 18:12:06 677] INFO [scheduling-1][c.a.s.j.m.j.d.DagInstFixedRateSchedule:38]- action=taskFlowDispatchJob|execute|exit|timeoutCount=0
[2022-03-24 18:12:06 679] INFO [scheduling-1][monitor:32]- >>>monitorJob|cost|inst=[0:0.0], nodeStart=[0:0.0], job=[0:0.0], task=[0:0.0], stdout=[0:0.0], status=[0:0.0]
[2022-03-24 18:12:06 679] INFO [scheduling-1][monitor:38]- >>>monitorJob|threadPool|inst=0, node=0, job=0
[2022-03-24 18:12:08 678] INFO [scheduling-1][monitor:32]- >>>monitorJob|cost|inst=[0:0.0], nodeStart=[0:0.0], job=[0:0.0], task=[0:0.0], stdout=[0:0.0], status=[0:0.0]
[2022-03-24 18:12:08 679] INFO [scheduling-1][monitor:38]- >>>monitorJob|threadPool|inst=0, node=0, job=0
[2022-03-24 18:12:10 678] INFO [scheduling-1][monitor:32]- >>>monitorJob|cost|inst=[0:0.0], nodeStart=[0:0.0], job=[0:0.0], task=[0:0.0], stdout=[0:0.0], status=[0:0.0]
[2022-03-24 18:12:10 678] INFO [scheduling-1][monitor:38]- >>>monitorJob|threadPool|inst=0, node=0, job=0
[2022-03-24 18:12:11 677] INFO [scheduling-1][c.a.s.j.m.j.d.DagInstFixedRateSchedule:38]- action=taskFlowDispatchJob|execute|exit|timeoutCount=0
[2022-03-24 18:12:12 678] INFO [scheduling-1][monitor:32]- >>>monitorJob|cost|inst=[0:0.0], nodeStart=[0:0.0], job=[0:0.0], task=[0:0.0], stdout=[0:0.0], status=[0:0.0]
[2022-03-24 18:12:12 678] INFO [scheduling-1][monitor:38]- >>>monitorJob|threadPool|inst=0, node=0, job=0
[2022-03-24 18:12:14 678] INFO [scheduling-1][monitor:32]- >>>monitorJob|cost|inst=[0:0.0], nodeStart=[0:0.0], job=[0:0.0], task=[0:0.0], stdout=[0:0.0], status=[0:0.0]
[2022-03-24 18:12:14 678] INFO [scheduling-1][monitor:38]- >>>monitorJob|threadPool|inst=0, node=0, job=0
[2022-03-24 18:12:16 677] INFO [scheduling-1][c.a.s.j.m.j.d.DagInstFixedRateSchedule:38]- action=taskFlowDispatchJob|execute|exit|timeoutCount=0
[2022-03-24 18:12:16 679] INFO [scheduling-1][monitor:32]- >>>monitorJob|cost|inst=[0:0.0], nodeStart=[0:0.0], job=[0:0.0], task=[0:0.0], stdout=[0:0.0], status=[0:0.0]
[2022-03-24 18:12:16 679] INFO [scheduling-1][monitor:38]- >>>monitorJob|threadPool|inst=0, node=0, job=0
[2022-03-24 18:12:18 680] INFO [scheduling-1][monitor:32]- >>>monitorJob|cost|inst=[0:0.0], nodeStart=[0:0.0], job=[0:0.0], task=[0:0.0], stdout=[0:0.0], status=[0:0.0]
[2022-03-24 18:12:18 680] INFO [scheduling-1][monitor:38]- >>>monitorJob|threadPool|inst=0, node=0, job=0
[2022-03-24 18:12:20 678] INFO [scheduling-1][monitor:32]- >>>monitorJob|cost|inst=[0:0.0], nodeStart=[0:0.0], job=[0:0.0], task=[0:0.0], stdout=[0:0.0], status=[0:0.0]
[2022-03-24 18:12:20 678] INFO [scheduling-1][monitor:38]- >>>monitorJob|threadPool|inst=0, node=0, job=0
[2022-03-24 18:12:21 677] INFO [scheduling-1][c.a.s.j.m.j.d.DagInstFixedRateSchedule:38]- action=taskFlowDispatchJob|execute|exit|timeoutCount=0
[2022-03-24 18:12:22 678] INFO [scheduling-1][monitor:32]- >>>monitorJob|cost|inst=[0:0.0], nodeStart=[0:0.0], job=[0:0.0], task=[0:0.0], stdout=[0:0.0], status=[0:0.0]
[2022-03-24 18:12:22 678] INFO [scheduling-1][monitor:38]- >>>monitorJob|threadPool|inst=0, node=0, job=0
[2022-03-24 18:12:24 677] INFO [scheduling-1][monitor:32]- >>>monitorJob|cost|inst=[0:0.0], nodeStart=[0:0.0], job=[0:0.0], task=[0:0.0], stdout=[0:0.0], status=[0:0.0]
[2022-03-24 18:12:24 678] INFO [scheduling-1][monitor:38]- >>>monitorJob|threadPool|inst=0, node=0, job=0
[2022-03-24 18:12:26 677] INFO [scheduling-1][c.a.s.j.m.j.d.DagInstFixedRateSchedule:38]- action=taskFlowDispatchJob|execute|exit|timeoutCount=0
[2022-03-24 18:12:26 679] INFO [scheduling-1][monitor:32]- >>>monitorJob|cost|inst=[0:0.0], nodeStart=[0:0.0], job=[0:0.0], task=[0:0.0], stdout=[0:0.0], status=[0:0.0]
[2022-03-24 18:12:26 679] INFO [scheduling-1][monitor:38]- >>>monitorJob|threadPool|inst=0, node=0, job=0
[2022-03-24 18:12:28 677] INFO [scheduling-1][monitor:32]- >>>monitorJob|cost|inst=[0:0.0], nodeStart=[0:0.0], job=[0:0.0], task=[0:0.0], stdout=[0:0.0], status=[0:0.0]
[2022-03-24 18:12:28 678] INFO [scheduling-1][monitor:38]- >>>monitorJob|threadPool|inst=0, node=0, job=0
[2022-03-24 18:12:30 678] INFO [scheduling-1][monitor:32]- >>>monitorJob|cost|inst=[0:0.0], nodeStart=[0:0.0], job=[0:0.0], task=[0:0.0], stdout=[0:0.0], status=[0:0.0]
[2022-03-24 18:12:30 678] INFO [scheduling-1][monitor:38]- >>>monitorJob|threadPool|inst=0, node=0, job=0
[2022-03-24 18:12:31 677] INFO [scheduling-1][c.a.s.j.m.j.d.DagInstFixedRateSchedule:38]- action=taskFlowDispatchJob|execute|exit|timeoutCount=0
[2022-03-24 18:12:32 678] INFO [scheduling-1][monitor:32]- >>>monitorJob|cost|inst=[0:0.0], nodeStart=[0:0.0], job=[0:0.0], task=[0:0.0], stdout=[0:0.0], status=[0:0.0]
[2022-03-24 18:12:32 678] INFO [scheduling-1][monitor:38]- >>>monitorJob|threadPool|inst=0, node=0, job=0
[2022-03-24 18:12:34 678] INFO [scheduling-1][monitor:32]- >>>monitorJob|cost|inst=[0:0.0], nodeStart=[0:0.0], job=[0:0.0], task=[0:0.0], stdout=[0:0.0], status=[0:0.0]
[2022-03-24 18:12:34 678] INFO [scheduling-1][monitor:38]- >>>monitorJob|threadPool|inst=0, node=0, job=0
[2022-03-24 18:12:36 677] INFO [scheduling-1][c.a.s.j.m.j.d.DagInstFixedRateSchedule:38]- action=taskFlowDispatchJob|execute|exit|timeoutCount=0
[2022-03-24 18:12:36 679] INFO [scheduling-1][monitor:32]- >>>monitorJob|cost|inst=[0:0.0], nodeStart=[0:0.0], job=[0:0.0], task=[0:0.0], stdout=[0:0.0], status=[0:0.0]
[2022-03-24 18:12:36 679] INFO [scheduling-1][monitor:38]- >>>monitorJob|threadPool|inst=0, node=0, job=0
[2022-03-24 18:12:38 677] INFO [scheduling-1][monitor:32]- >>>monitorJob|cost|inst=[0:0.0], nodeStart=[0:0.0], job=[0:0.0], task=[0:0.0], stdout=[0:0.0], status=[0:0.0]
[2022-03-24 18:12:38 677] INFO [scheduling-1][monitor:38]- >>>monitorJob|threadPool|inst=0, node=0, job=0
[2022-03-24 18:12:40 685] INFO [scheduling-1][monitor:32]- >>>monitorJob|cost|inst=[0:0.0], nodeStart=[0:0.0], job=[0:0.0], task=[0:0.0], stdout=[0:0.0], status=[0:0.0]
[2022-03-24 18:12:40 685] INFO [scheduling-1][monitor:38]- >>>monitorJob|threadPool|inst=0, node=0, job=0
[2022-03-24 18:12:41 677] INFO [scheduling-1][c.a.s.j.m.j.d.DagInstFixedRateSchedule:38]- action=taskFlowDispatchJob|execute|exit|timeoutCount=0
[2022-03-24 18:12:42 677] INFO [scheduling-1][monitor:32]- >>>monitorJob|cost|inst=[0:0.0], nodeStart=[0:0.0], job=[0:0.0], task=[0:0.0], stdout=[0:0.0], status=[0:0.0]
[2022-03-24 18:12:42 678] INFO [scheduling-1][monitor:38]- >>>monitorJob|threadPool|inst=0, node=0, job=0
[2022-03-24 18:12:44 678] INFO [scheduling-1][monitor:32]- >>>monitorJob|cost|inst=[0:0.0], nodeStart=[0:0.0], job=[0:0.0], task=[0:0.0], stdout=[0:0.0], status=[0:0.0]
[2022-03-24 18:12:44 678] INFO [scheduling-1][monitor:38]- >>>monitorJob|threadPool|inst=0, node=0, job=0
[2022-03-24 18:12:46 677] INFO [scheduling-1][c.a.s.j.m.j.d.DagInstFixedRateSchedule:38]- action=taskFlowDispatchJob|execute|exit|timeoutCount=0
[2022-03-24 18:12:46 679] INFO [scheduling-1][monitor:32]- >>>monitorJob|cost|inst=[0:0.0], nodeStart=[0:0.0], job=[0:0.0], task=[0:0.0], stdout=[0:0.0], status=[0:0.0]
[2022-03-24 18:12:46 679] INFO [scheduling-1][monitor:38]- >>>monitorJob|threadPool|inst=0, node=0, job=0
[2022-03-24 18:12:48 678] INFO [scheduling-1][monitor:32]- >>>monitorJob|cost|inst=[0:0.0], nodeStart=[0:0.0], job=[0:0.0], task=[0:0.0], stdout=[0:0.0], status=[0:0.0]
[2022-03-24 18:12:48 678] INFO [scheduling-1][monitor:38]- >>>monitorJob|threadPool|inst=0, node=0, job=0
SREWorks用helm部署完成,网页能进去,不过和运维相关的功能都提示404错误. 看k8s相关资源,有个sreworks命名空间下的prod-job-job-worker deployment失败了,不知道是不是这个原因.
相关信息: git commit id:e85c723 k8s版本:v1.23.4,v1.23.5 ingress控制器:nginx ingress storageClass:nfs-client 失败的pod log: