安全加固方案 案例执行失败

[byeyang]

Issue Type

Install/Deploy

Search for existing issues similar to yours

No

OS Platform and Distribution

linux kylin

Kuscia Version

kuscia v0.10.0b0

Deployment

docker

deployment Version

docker

App Running type

secretflow

App Running version

secretflow 1.8

Configuration file used to run kuscia.

# 登入 master 容器。
docker exec -it ${USER}-kuscia-master bash
# 创建并启动作业（两方 PSI 任务），作业会在安全沙箱中执行。
scripts/user/create_example_job.sh NSJAIL_PSI
kubectl get kj -n cross-domain

What happend and What you expected to happen.

[root@root-kuscia-master-localhost-localdomain kuscia]# kubectl get kt secretflow-task-20240926152313-single-psi -n cross-domain -o yaml
apiVersion: kuscia.secretflow/v1alpha1
kind: KusciaTask
metadata:
  annotations:
    kuscia.secretflow/job-id: secretflow-task-20240926152313
    kuscia.secretflow/self-cluster-as-participant: "true"
    kuscia.secretflow/task-alias: single-psi
  creationTimestamp: "2024-09-26T07:23:14Z"
  generation: 1
  labels:
    kuscia.secretflow/controller: kuscia-job
    kuscia.secretflow/job-uid: 516d636e-e33a-4a29-b326-30d579da2446
  name: secretflow-task-20240926152313-single-psi
  namespace: cross-domain
  ownerReferences:
  - apiVersion: kuscia.secretflow/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: KusciaJob
    name: secretflow-task-20240926152313
    uid: 516d636e-e33a-4a29-b326-30d579da2446
  resourceVersion: "2220"
  uid: 694e8b60-d57e-4b9a-b05a-72d783fbe416
spec:
  initiator: alice
  parties:
  - appImageRef: secretflow-nsjail-image
    domainID: alice
    template:
      spec: {}
  - appImageRef: secretflow-nsjail-image
    domainID: bob
    template:
      spec: {}
  scheduleConfig: {}
  taskInputConfig: '{"sf_datasource_config":{"alice":{"id":"default-data-source"},"bob":{"id":"default-data-source"}},"sf_cluster_desc":{"parties":["alice","bob"],"devices":[{"name":"spu","type":"spu","parties":["alice","bob"],"config":"{\"runtime_config\":{\"protocol\":\"REF2K\",\"field\":\"FM64\"},\"link_desc\":{\"connect_retry_times\":60,\"connect_retry_interval_ms\":1000,\"brpc_channel_protocol\":\"http\",\"brpc_channel_connection_type\":\"pooled\",\"recv_timeout_ms\":1200000,\"http_timeout_ms\":1200000}}"},{"name":"heu","type":"heu","parties":["alice","bob"],"config":"{\"mode\":
    \"PHEU\", \"schema\": \"paillier\", \"key_size\": 2048}"}],"ray_fed_config":{"cross_silo_comm_backend":"brpc_link"}},"sf_node_eval_param":{"domain":"data_prep","name":"psi","version":"0.0.5","attr_paths":["protocol","sort_result","allow_duplicate_keys

Kuscia log output.

[root@root-kuscia-master-localhost-localdomain kuscia]# kubectl describe pod secretflow-task-20240926152313-single-psi-0 -n alice
Name:             secretflow-task-20240926152313-single-psi-0
Namespace:        alice
Priority:         0
Service Account:  default
Node:             root-kuscia-lite-alice-localhost-localdomain/172.20.0.3
Start Time:       Thu, 26 Sep 2024 15:23:16 +0800
Labels:           kuscia.secretflow/communication-role-client=true
                  kuscia.secretflow/communication-role-server=true
                  kuscia.secretflow/controller=kusciatask
                  kuscia.secretflow/pod-identity=694e8b60-d57e-4b9a-b05a-72d783fbe416-0
                  kuscia.secretflow/pod-role=
                  kuscia.secretflow/task-resource-uid=e6cc46ed-1d41-4570-96f3-7f76bdb75e6c
                  kuscia.secretflow/task-uid=694e8b60-d57e-4b9a-b05a-72d783fbe416
Annotations:      kuscia.secretflow/config-template-volumes: config-template
                  kuscia.secretflow/initiator: alice
                  kuscia.secretflow/task-id: secretflow-task-20240926152313-single-psi
                  kuscia.secretflow/task-resource: secretflow-task-20240926152313-single-psi-aaaf2d31391a
                  kuscia.secretflow/task-resource-group: secretflow-task-20240926152313-single-psi
Status:           Failed
IP:               
IPs:              <none>
Containers:
  secretflow:
    Container ID:  containerd://a719868cfe076e25a5e57a80a3b70b358804e3c67507277b2148c931ff78b0af
    Image:         secretflow/sf-dev-anolis8:0.09
    Image ID:      sha256:f3db2fa77d70ce1a4a1383b743132cb38a8a13454eb97ebea13f6d1e815acbf4
    Ports:         25075/TCP, 25076/TCP, 25077/TCP, 25078/TCP, 25079/TCP, 25080/TCP
    Host Ports:    0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP
    Command:
      sh
    Args:
      -c
      sh .nsjail/run.sh
    State:          Terminated
      Reason:       Error
      Exit Code:    255
      Started:      Thu, 26 Sep 2024 15:23:19 +0800
      Finish

cd var/stdout  查看日志  Alice和bob里面都是空的0.log文件均为空白

[wangzul]

kubectl get kj -n cross-domain 和kubectl get pod -A 任务状态是什么样子的。

[wangzul]

kubectl get kt secretflow-task-20240926152313-single-psi -n cross-domain -o yaml 看到上方你有通过命令查看任务信息，但是看起来不完整需要给一个完整的。

[byeyang]

kubectl get kt secretflow-task-20240926152313-single-psi -n cross-domain -o yaml 看到上方你有通过命令查看任务信息，但是看起来不完整需要给一个完整的。

[root@root-kuscia-master kuscia]# kubectl get kj -n cross-domain NAME STARTTIME COMPLETIONTIME LASTRECONCILETIME PHASE secretflow-task-20241008170216 3m41s 3m32s 3m32s Failed [root@root-kuscia-master kuscia]# kubectl get pod -A NAMESPACE NAME READY STATUS RESTARTS AGE bob secretflow-task-20241008170216-single-psi-0 0/1 Error 0 3m53s [root@root-kuscia-master kuscia]# kubectl get kt secretflow-task-20241008170216-single-psi -n cross-domain -o yaml apiVersion: kuscia.secretflow/v1alpha1 kind: KusciaTask metadata: annotations: kuscia.secretflow/job-id: secretflow-task-20241008170216 kuscia.secretflow/self-cluster-as-participant: "true" kuscia.secretflow/task-alias: single-psi creationTimestamp: "2024-10-08T09:02:18Z" generation: 1 labels: kuscia.secretflow/controller: kuscia-job kuscia.secretflow/job-uid: 3ead13d8-876e-4097-b657-11ed4f7609cb name: secretflow-task-20241008170216-single-psi namespace: cross-domain ownerReferences:

• apiVersion: kuscia.secretflow/v1alpha1 blockOwnerDeletion: true controller: true kind: KusciaJob name: secretflow-task-20241008170216 uid: 3ead13d8-876e-4097-b657-11ed4f7609cb resourceVersion: "2654" uid: 37dd97d9-0aa2-4c74-8f54-a16b0d8205e1 spec: initiator: alice parties:
• appImageRef: secretflow-nsjail-image domainID: alice template: spec: {}
• appImageRef: secretflow-nsjail-image domainID: bob template: spec: {} scheduleConfig: {} taskInputConfig: '{"sf_datasource_config":{"alice":{"id":"default-data-source"},"bob":{"id":"default-data-source"}},"sf_cluster_desc":{"parties":["alice","bob"],"devices":[{"name":"spu","type":"spu","parties":["alice","bob"],"config":"{\"runtime_config\":{\"protocol\":\"REF2K\",\"field\":\"FM64\"},\"link_desc\":{\"connect_retry_times\":60,\"connect_retry_interval_ms\":1000,\"brpc_channel_protocol\":\"http\",\"brpc_channel_connection_type\":\"pooled\",\"recv_timeout_ms\":1200000,\"http_timeout_ms\":1200000}}"},{"name":"heu","type":"heu","parties":["alice","bob"],"config":"{\"mode\": \"PHEU\", \"schema\": \"paillier\", \"key_size\": 2048}"}],"ray_fed_config":{"cross_silo_comm_backend":"brpc_link"}},"sf_node_eval_param":{"domain":"data_prep","name":"psi","version":"0.0.5","attr_paths":["protocol","sort_result","allow_duplicate_keys","allow_duplicate_keys/yes/join_type","allow_duplicate_keys/yes/join_type/left_join/left_side","input/receiver_input/key","input/sender_input/key"],"attrs":[{"s":"PROTOCOL_ECDH"},{"b":true},{"s":"yes"},{"s":"left_join"},{"ss":["alice"]},{"ss":["id1"]},{"ss":["id2"]}]},"sf_input_ids":["alice-table","bob-table"],"sf_output_ids":["psi-output"],"sf_output_uris":["psi-output.csv"]}' status: allocatedPorts:
• domainID: bob namedPort: secretflow-task-20241008170216-single-psi-0/client-server: 23713 secretflow-task-20241008170216-single-psi-0/fed: 23715 secretflow-task-20241008170216-single-psi-0/global: 23716 secretflow-task-20241008170216-single-psi-0/node-manager: 23711 secretflow-task-20241008170216-single-psi-0/object-manager: 23712 secretflow-task-20241008170216-single-psi-0/spu: 23714
• domainID: alice namedPort: secretflow-task-20241008170216-single-psi-0/client-server: 31286 secretflow-task-20241008170216-single-psi-0/fed: 31288 secretflow-task-20241008170216-single-psi-0/global: 31289 secretflow-task-20241008170216-single-psi-0/node-manager: 31290 secretflow-task-20241008170216-single-psi-0/object-manager: 31285 secretflow-task-20241008170216-single-psi-0/spu: 31287 completionTime: "2024-10-08T09:02:27Z" conditions:
• lastTransitionTime: "2024-10-08T09:02:18Z" status: "True" type: ResourceCreated
• lastTransitionTime: "2024-10-08T09:02:25Z" status: "True" type: Running
• lastTransitionTime: "2024-10-08T09:02:27Z" status: "False" type: Success lastReconcileTime: "2024-10-08T09:02:27Z" message: The remaining no-failed party task counts 1 are less than the threshold 2 that meets the conditions for task success. pending party[], running party[alice], successful party[], failed party[bob] partyTaskStatus:
• domainID: alice phase: Failed
• domainID: bob phase: Failed phase: Failed podStatuses: alice/secretflow-task-20241008170216-single-psi-0: createTime: "2024-10-08T09:02:18Z" namespace: alice nodeName: root-kuscia-lite-alice podName: secretflow-task-20241008170216-single-psi-0 podPhase: Failed readyTime: "2024-10-08T09:02:25Z" startTime: "2024-10-08T09:02:20Z" bob/secretflow-task-20241008170216-single-psi-0: createTime: "2024-10-08T09:02:18Z" namespace: bob nodeName: root-kuscia-lite-bob podName: secretflow-task-20241008170216-single-psi-0 podPhase: Failed readyTime: "2024-10-08T09:02:25Z" reason: Error startTime: "2024-10-08T09:02:20Z" serviceStatuses: alice/secretflow-task-20241008170216-single-psi-0-fed: createTime: "2024-10-08T09:02:18Z" namespace: alice portName: fed portNumber: 31288 readyTime: "2024-10-08T09:02:25Z" scope: Cluster serviceName: secretflow-task-20241008170216-single-psi-0-fed alice/secretflow-task-20241008170216-single-psi-0-global: createTime: "2024-10-08T09:02:18Z" namespace: alice portName: global portNumber: 31289 readyTime: "2024-10-08T09:02:25Z" scope: Domain serviceName: secretflow-task-20241008170216-single-psi-0-global alice/secretflow-task-20241008170216-single-psi-0-spu: createTime: "2024-10-08T09:02:18Z" namespace: alice portName: spu portNumber: 31287 readyTime: "2024-10-08T09:02:25Z" scope: Cluster serviceName: secretflow-task-20241008170216-single-psi-0-spu bob/secretflow-task-20241008170216-single-psi-0-fed: createTime: "2024-10-08T09:02:18Z" namespace: bob portName: fed portNumber: 23715 readyTime: "2024-10-08T09:02:25Z" scope: Cluster serviceName: secretflow-task-20241008170216-single-psi-0-fed bob/secretflow-task-20241008170216-single-psi-0-global: createTime: "2024-10-08T09:02:19Z" namespace: bob portName: global portNumber: 23716 readyTime: "2024-10-08T09:02:25Z" scope: Domain serviceName: secretflow-task-20241008170216-single-psi-0-global bob/secretflow-task-20241008170216-single-psi-0-spu: createTime: "2024-10-08T09:02:18Z" namespace: bob portName: spu portNumber: 23714 readyTime: "2024-10-08T09:02:25Z" scope: Cluster serviceName: secretflow-task-20241008170216-single-psi-0-spu startTime: "2024-10-08T09:02:18Z" [root@root-kuscia-master kuscia]#

[wangzul]

看一下bob方的kuscia日志 /home/kuscia/var/logs/kuscia.log

[byeyang]

看一下bob方的kuscia日志 /home/kuscia/var/logs/kuscia.log

(base) [root@localhost ~]# docker exec -it ${USER}-kuscia-lite-bob bash [root@root-kuscia-lite-bob kuscia]# cat /home/kuscia/var/logs/kuscia.log 2024-10-08 16:49:02.579 INFO modules/modules.go:337 Start to init all secret backends ... 2024-10-08 16:49:02.580 WARN modules/modules.go:344 Init all secret backend but no provider found, creating default mem type 2024-10-08 16:49:02.580 INFO modules/modules.go:349 Finish Initializing all secret backends 2024-10-08 16:49:02.590 INFO tls/crypt.go:377 Generate cert with key, subject[bob] 2024-10-08 16:49:02.601 INFO tls/crypt.go:377 Generate cert with key, subject[bob] 2024-10-08 16:49:02.619 INFO modules/modules.go:115 Add destroyFunc [coredns] 2024-10-08 16:49:02.619 INFO modules/coredns.go:195 Start preparing coredns resolv.conf, root dir /home/kuscia/ 2024-10-08 16:49:02.623 INFO modules/coredns.go:218 Finish preparing coredns resolv.conf 2024-10-08 16:49:02.626 INFO modules/coredns.go:186 CoreDNS is ready 2024-10-08 16:49:02.627 INFO modules/modules.go:115 Add destroyFunc [containerd] 2024-10-08 16:49:02.705 INFO supervisor/supervisor.go:75 [containerd] start and watch subprocess 2024-10-08 16:49:02.705 INFO supervisor/supervisor.go:101 [containerd] try to start new process 2024-10-08 16:49:02.713 INFO process/process.go:76 Set pid[20] oom score adj to -700 2024-10-08 16:49:04.789 INFO modules/containerd.go:180 Containerd is ready 2024-10-08 16:49:04.789 INFO modules/modules.go:115 Add destroyFunc [transport] 2024-10-08 16:49:04.790 INFO modules/modules.go:115 Add destroyFunc [envoy] 2024-10-08 16:49:04.791 INFO supervisor/supervisor.go:75 [envoy] start and watch subprocess 2024-10-08 16:49:04.791 INFO supervisor/supervisor.go:101 [envoy] try to start new process 2024-10-08 16:49:04.792 INFO modules/modules.go:115 Add destroyFunc [domainroute] 2024-10-08 16:49:04.797 INFO xds/xds.go:160 Management server listening on 10001 2024-10-08 16:49:04.797 INFO process/process.go:76 Set pid[66] oom score adj to -700 2024-10-08 16:49:04.819 INFO commands/root.go:62 Start xds success 2024-10-08 16:49:04.839 INFO commands/root.go:86 [PROBE] success to probe master endpoint https://root-kuscia-master:1080 2024-10-08 16:49:04.846 INFO xds/cluster_config.go:131 Generate tls config for service-masterproxy 2024-10-08 16:49:04.846 INFO xds/cluster_config.go:275 disable keep-alive for cluster:service-masterproxy 2024-10-08 16:49:04.846 INFO xds/xds.go:439 Add cluster:service-masterproxy 2024-10-08 16:49:04.846 INFO clusters/master.go:61 add Master cluster:masterproxy 2024-10-08 16:49:04.893 ERROR modules/envoy.go:59 Get ready err:Get "http://127.0.0.1:10000/ready": dial tcp 127.0.0.1:10000: connect: connection refused 2024-10-08 16:49:04.993 ERROR modules/envoy.go:59 Get ready err:Get "http://127.0.0.1:10000/ready": dial tcp 127.0.0.1:10000: connect: connection refused 2024-10-08 16:49:05.294 INFO modules/envoy.go:226 Envoy is ready 2024-10-08 16:49:05.793 INFO modules/transport.go:144 transport is ready 2024-10-08 16:49:06.879 INFO clusters/master.go:278 Get master gateway namespace: kuscia-system 2024-10-08 16:49:07.063 WARN controller/handshake.go:754 invalid source domain [bob] publickey in domainroute [bob-kuscia-system], error: public key is empty 2024-10-08 16:49:09.129 INFO xds/cluster_config.go:275 enable keep-alive for cluster:service-masterproxy 2024-10-08 16:49:09.129 INFO xds/xds.go:439 Add cluster:service-masterproxy 2024-10-08 16:49:09.207 INFO commands/root.go:218 Check MasterProxy ready 2024-10-08 16:49:09.278 INFO commands/root.go:91 Add master proxy clusters successfully 2024-10-08 16:49:09.279 INFO xds/xds.go:439 Add cluster:service-transport 2024-10-08 16:49:09.279 INFO clusters/interconn.go:70 Add Transport Cluster success 2024-10-08 16:49:09.279 INFO commands/root.go:104 Add interconn clusters success 2024-10-08 16:49:09.280 INFO commands/root.go:170 Gateway running 2024-10-08 16:49:09.280 INFO modules/domainroute.go:159 DomainRoute is ready 2024-10-08 16:49:09.280 INFO modules/modules.go:115 Add destroyFunc [confmanager] 2024-10-08 16:49:09.281 INFO controller/domain_route.go:183 Starting DomainRoute controller 2024-10-08 16:49:09.281 INFO controller/domain_route.go:186 Waiting for informer caches to sync 2024-10-08 16:49:09.281 INFO controller/gateway.go:105 Starting Gateway controller 2024-10-08 16:49:09.281 INFO controller/endpoints.go:122 Waiting for informer caches to sync 2024-10-08 16:49:09.282 INFO nlog/nlog.go:77 I1008 16:49:09.282064 7 shared_informer.go:270] Waiting for caches to sync for endpoints 2024-10-08 16:49:09.282 INFO poller/poll_manager.go:127 Waiting for informer caches to sync 2024-10-08 16:49:09.282 INFO nlog/nlog.go:77 I1008 16:49:09.282977 7 shared_informer.go:270] Waiting for caches to sync for poll manager 2024-10-08 16:49:09.287 INFO nlog/nlog.go:77 I1008 16:49:09.287308 7 shared_informer.go:270] Waiting for caches to sync for endpoints 2024-10-08 16:49:09.290 INFO nlog/nlog.go:77 I1008 16:49:09.290423 7 shared_informer.go:270] Waiting for caches to sync for pod 2024-10-08 16:49:09.393 INFO nlog/nlog.go:77 I1008 16:49:09.393605 7 shared_informer.go:277] Caches are synced for poll manager 2024-10-08 16:49:09.393 INFO poller/poll_manager.go:142 Starting poll manager 2024-10-08 16:49:09.393 INFO controller/domain_route.go:193 Starting workers 2024-10-08 16:49:09.393 INFO controller/domain_route.go:198 Started workers 2024-10-08 16:49:09.394 INFO controller/domain_route.go:357 Update DomainRoute bob/bob-kuscia-system revision:512 2024-10-08 16:49:09.394 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/bob-kuscia-system] (89.723µs) 2024-10-08 16:49:09.395 INFO nlog/nlog.go:77 I1008 16:49:09.395167 7 shared_informer.go:277] Caches are synced for endpoints 2024-10-08 16:49:09.395 INFO controller/endpoints.go:127 Starting endpoints Controller 2024-10-08 16:49:09.397 INFO nlog/nlog.go:77 I1008 16:49:09.397477 7 shared_informer.go:277] Caches are synced for pod 2024-10-08 16:49:09.397 INFO coredns/controller.go:52 Starting pod controller, namespace: bob 2024-10-08 16:49:09.397 INFO nlog/nlog.go:77 I1008 16:49:09.397527 7 shared_informer.go:277] Caches are synced for endpoints 2024-10-08 16:49:09.397 INFO coredns/controller.go:42 Starting endpoint controller, namespace: bob 2024-10-08 16:49:09.447 INFO controller/gateway.go:146 create gateway(name:root-kuscia-lite-bob namespace:bob) success 2024-10-08 16:49:09.476 INFO controller/gateway.go:194 Envoy cluster changed, old: [] new: [172.20.0.4] 2024-10-08 16:49:09.476 INFO xds/xds.go:439 Add cluster:envoy-cluster 2024-10-08 16:49:09.510 INFO controller/domain_route.go:357 Update DomainRoute bob/bob-kuscia-system revision:516 2024-10-08 16:49:09.510 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/bob-kuscia-system] (202.565µs) 2024-10-08 16:49:09.558 INFO controller/domain_route.go:357 Update DomainRoute bob/bob-kuscia-system revision:517 2024-10-08 16:49:09.566 INFO xds/xds.go:439 Add cluster:service-masterproxy 2024-10-08 16:49:09.566 INFO controller/domain_route.go:497 Update rule to master success 2024-10-08 16:49:09.607 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/bob-kuscia-system] (49.238459ms) 2024-10-08 16:49:09.607 INFO controller/domain_route.go:357 Update DomainRoute bob/bob-kuscia-system revision:519 2024-10-08 16:49:09.614 INFO xds/xds.go:439 Add cluster:service-masterproxy 2024-10-08 16:49:09.614 INFO controller/domain_route.go:497 Update rule to master success 2024-10-08 16:49:09.614 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/bob-kuscia-system] (6.856056ms) 2024-10-08 16:49:10.403 INFO engine/engine.go:192 default server bean not exist, skip router regist 2024-10-08 16:49:10.403 INFO engine/engine.go:215 Starting bean.httpServerBean Bean: ConfManagerHttpServer 2024-10-08 16:49:10.403 INFO engine/engine.go:215 Starting bean.grpcServerBean Bean: ConfManagerGrpcServer 2024-10-08 16:49:10.403 INFO bean/grpc_server_bean.go:89 Grpc server listening on :8061 2024-10-08 16:49:10.403 INFO beans/gin_bean.go:133 https server started on :8060 2024-10-08 16:49:11.428 INFO decorator/proto_decorator.go:165 REQUEST Path: /healthZ Content-Type: application/json Body: {} 2024-10-08 16:49:11.442 INFO decorator/proto_decorator.go:216 RESPONSE{"status":{"message":"success"},"data":{"ready":true}} 2024-10-08 16:49:11.442 INFO modules/confmanager.go:184 http/https server is ready 2024-10-08 16:49:11.443 INFO modules/confmanager.go:221 ConfManager is ready 2024-10-08 16:49:11.443 INFO modules/modules.go:115 Add destroyFunc [kusciaapi] 2024-10-08 16:49:13.071 INFO engine/engine.go:192 default server bean not exist, skip router regist 2024-10-08 16:49:13.071 INFO engine/engine.go:215 Starting bean.httpServerBean Bean: kusciaAPIHttpServer 2024-10-08 16:49:13.071 INFO engine/engine.go:215 Starting bean.grpcServerBean Bean: kusciaAPIGrpcServer 2024-10-08 16:49:13.071 INFO bean/grpc_server_bean.go:113 grpc server listening on :8083 2024-10-08 16:49:13.072 INFO beans/gin_bean.go:133 https server started on :8082 2024-10-08 16:49:13.072 INFO beans/gin_bean.go:137 http server started 127.0.0.1:8092 2024-10-08 16:49:13.190 INFO decorator/proto_decorator.go:165 REQUEST Path: /healthZ Content-Type: application/json Body: {} 2024-10-08 16:49:13.191 INFO decorator/proto_decorator.go:216 RESPONSE{"status":{"message":"success"},"data":{"ready":true}} 2024-10-08 16:49:13.193 INFO modules/kusciaapi.go:229 http server is ready 2024-10-08 16:49:13.227 INFO modules/kusciaapi.go:293 KusciaApi is ready 2024-10-08 16:49:13.227 INFO modules/modules.go:115 Add destroyFunc [agent] 2024-10-08 16:49:13.227 INFO commands/root.go:46 Run root command, Namespace=bob 2024-10-08 16:49:13.227 INFO plugin/plugin.go:71 Init plugin hook:image-security succeed 2024-10-08 16:49:13.227 INFO plugin/plugin.go:71 Init plugin hook:env-import succeed 2024-10-08 16:49:13.227 INFO plugin/plugin.go:71 Init plugin hook:cert-issuance succeed 2024-10-08 16:49:13.227 INFO plugin/plugin.go:71 Init plugin hook:config-render succeed 2024-10-08 16:49:13.247 INFO node/capacity_manager.go:56 Capacity Manager, runtime: runc, capacityCfg:&{ }, reservedResCfg: &{0.5 500Mi}, rootDir: /home/kuscia/, localCapacity:true 2024-10-08 16:49:13.249 INFO node/capacity_manager.go:195 Total cpu: 8, available cpu: 8, cpu quota: 750000, cpu period: 100000 2024-10-08 16:49:13.249 INFO node/capacity_manager.go:214 Total memory: 4294967296, available memory: 3770679296 2024-10-08 16:49:13.251 INFO framework/node_controller.go:166 Configure node root-kuscia-lite-bob 2024-10-08 16:49:13.254 INFO node/generic_node.go:70 Configure generic node "root-kuscia-lite-bob" successfully 2024-10-08 16:49:13.389 INFO framework/node_controller.go:567 Created new lease, name=root-kuscia-lite-bob 2024-10-08 16:49:13.389 INFO framework/node_controller.go:203 Node controller started 2024-10-08 16:49:13.411 INFO kuberuntime/kuberuntime_manager.go:190 Container runtime initialized, containerRuntime=containerd, version=v1.7.7-k3s1.26, apiVersion=v1 2024-10-08 16:49:13.412 INFO source/apiserver.go:82 Start running apiserver source 2024-10-08 16:49:13.512 INFO source/apiserver.go:91 Pod Informer: cache sync finished 2024-10-08 16:49:13.512 INFO framework/pods_controller.go:204 Starting Pods controller ... 2024-10-08 16:49:13.512 INFO status/status_manager.go:164 Starting to sync pod status with apiserver 2024-10-08 16:49:13.512 INFO framework/pods_controller.go:311 Starting agent main sync loop 2024-10-08 16:49:13.512 INFO commands/root.go:190 Agent started 2024-10-08 16:49:13.513 INFO modules/agent.go:175 Agent is ready 2024-10-08 16:49:13.513 INFO modules/modules.go:115 Add destroyFunc [datamesh] 2024-10-08 16:49:13.517 INFO pod/cri_provider.go:310 Starting CRI provider ... 2024-10-08 16:49:13.626 INFO framework/node_controller.go:206 Node is ready 2024-10-08 16:49:14.236 INFO modules/datamesh.go:60 Datamesh namespace:bob. 2024-10-08 16:49:14.236 INFO bean/operator_bean.go:53 OperatorBean init 2024-10-08 16:49:14.236 INFO engine/engine.go:192 default server bean not exist, skip router regist 2024-10-08 16:49:14.236 INFO engine/engine.go:215 Starting bean.httpServerBean Bean: DataMeshHttpServer 2024-10-08 16:49:14.236 INFO engine/engine.go:215 Starting bean.grpcServerBean Bean: DataMeshGrpcServer 2024-10-08 16:49:14.236 INFO engine/engine.go:215 Starting *bean.operatorBean Bean: DataMeshOperator 2024-10-08 16:49:14.236 INFO service/operator.go:47 DataMesh operator service start 2024-10-08 16:49:14.237 INFO beans/gin_bean.go:133 https server started on :8070 2024-10-08 16:49:14.237 INFO bean/grpc_server_bean.go:99 Grpc server listening on :8071 2024-10-08 16:49:14.272 INFO service/domaindatasource.go:62 Create default datasource default-data-source. 2024-10-08 16:49:14.306 INFO service/operator.go:91 Datasource default-data-source has been created successful. 2024-10-08 16:49:14.345 INFO service/domaindatasource.go:79 Create default datasource: default-dp-data-source. 2024-10-08 16:49:14.362 INFO decorator/proto_decorator.go:165 REQUEST Path: /healthZ Content-Type: application/json Body: {} 2024-10-08 16:49:14.362 INFO decorator/proto_decorator.go:216 RESPONSE{"status":{"message":"success"},"data":{"ready":true}} 2024-10-08 16:49:14.363 INFO modules/datamesh.go:140 http server is ready 2024-10-08 16:49:14.363 INFO modules/datamesh.go:177 DataMesh is ready 2024-10-08 16:49:14.363 INFO modules/modules.go:115 Add destroyFunc [nodeexporter] 2024-10-08 16:49:14.363 INFO supervisor/supervisor.go:75 [node_exporter] start and watch subprocess 2024-10-08 16:49:14.363 INFO supervisor/supervisor.go:101 [node_exporter] try to start new process 2024-10-08 16:49:14.372 INFO service/operator.go:106 Datasource default-dp-data-source has been created successful. 2024-10-08 16:49:14.377 INFO process/process.go:76 Set pid[97] oom score adj to -600 2024-10-08 16:49:15.371 INFO modules/nodeexporter.go:142 Node_exporter is ready 2024-10-08 16:49:15.371 INFO modules/modules.go:115 Add destroyFunc [ssexporter] 2024-10-08 16:49:16.380 INFO modules/ssexporter.go:128 Ss exporter is ready 2024-10-08 16:49:16.380 INFO modules/modules.go:115 Add destroyFunc [metricexporter] 2024-10-08 16:49:16.380 INFO metricexporter/metricexporter.go:87 Start to export metrics... 2024-10-08 16:49:17.382 INFO modules/metricexporter.go:132 Metric exporter is ready 2024-10-08 16:49:17.383 INFO process/process.go:76 Set pid[1] oom score adj to -999 2024-10-08 16:49:17.383 INFO process/process.go:76 Set pid[7] oom score adj to -900 2024-10-08 16:49:18.882 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (27.695µs) 2024-10-08 16:49:18.882 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:554 2024-10-08 16:49:18.882 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (106.201µs) 2024-10-08 16:49:19.962 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (35.873µs) 2024-10-08 16:49:19.962 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:557 2024-10-08 16:49:19.962 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (104.295µs) 2024-10-08 16:49:19.979 INFO controller/handshake.go:598 Update domainRoute [alice-bob] status successfully 2024-10-08 16:49:20.054 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (31.758µs) 2024-10-08 16:49:20.054 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:559 2024-10-08 16:49:20.150 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (33.153µs) 2024-10-08 16:49:20.151 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (96.547511ms) 2024-10-08 16:49:20.151 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:561 2024-10-08 16:49:20.158 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (6.79125ms) 2024-10-08 16:49:20.193 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (27.844µs) 2024-10-08 16:49:20.193 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:563 2024-10-08 16:49:20.201 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (7.355173ms) 2024-10-08 16:49:20.911 INFO controller/domain_route.go:432 add cluster bob-to-alice name:http protocol:HTTP port:1080 2024-10-08 16:49:20.911 INFO controller/domain_route.go:916 disable keep-alive for cluster:bob-to-alice-http 2024-10-08 16:49:20.911 INFO xds/xds.go:439 Add cluster:bob-to-alice-http 2024-10-08 16:49:20.911 INFO controller/domain_route.go:357 Update DomainRoute bob/bob-alice revision:572 2024-10-08 16:49:20.911 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/bob-alice] (591.837µs) 2024-10-08 16:49:20.974 INFO controller/domain_route.go:432 add cluster bob-to-alice name:http protocol:HTTP port:1080 2024-10-08 16:49:20.974 INFO xds/xds.go:439 Add cluster:bob-to-alice-http 2024-10-08 16:49:20.975 INFO xds/xds.go:439 Add cluster:bob-to-alice-http 2024-10-08 16:49:20.975 INFO controller/domain_route.go:293 DomainRoute bob/bob-alice starts handshake, the last revision is 0 2024-10-08 16:49:20.981 INFO controller/handshake.go:424 DomainRoute alice-bob handle successfully 2024-10-08 16:49:22.217 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/bob-alice] (1.243566704s) 2024-10-08 16:49:22.218 INFO controller/domain_route.go:432 add cluster bob-to-alice name:http protocol:HTTP port:1080 2024-10-08 16:49:22.218 INFO xds/xds.go:439 Add cluster:bob-to-alice-http 2024-10-08 16:49:22.218 INFO controller/domain_route.go:357 Update DomainRoute bob/bob-alice revision:592 2024-10-08 16:49:22.218 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/bob-alice] (417.072µs) 2024-10-08 16:49:22.246 INFO controller/domain_route.go:432 add cluster bob-to-alice name:http protocol:HTTP port:1080 2024-10-08 16:49:22.247 INFO xds/xds.go:439 Add cluster:bob-to-alice-http 2024-10-08 16:49:22.247 INFO controller/domain_route.go:357 Update DomainRoute bob/bob-alice revision:593 2024-10-08 16:49:22.254 INFO xds/cluster_config.go:275 enable keep-alive for cluster:bob-to-alice-http 2024-10-08 16:49:22.254 INFO xds/xds.go:439 Add cluster:bob-to-alice-http 2024-10-08 16:49:22.318 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/bob-alice] (71.665264ms) 2024-10-08 16:49:22.319 INFO controller/domain_route.go:432 add cluster bob-to-alice name:http protocol:HTTP port:1080 2024-10-08 16:49:22.319 INFO xds/xds.go:439 Add cluster:bob-to-alice-http 2024-10-08 16:49:22.319 INFO controller/domain_route.go:357 Update DomainRoute bob/bob-alice revision:595 2024-10-08 16:49:22.327 INFO xds/xds.go:439 Add cluster:bob-to-alice-http 2024-10-08 16:49:22.327 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/bob-alice] (8.380853ms) 2024-10-08 16:49:24.398 INFO poller/poll_manager.go:194 Current gateway member count is 0, while alive gateway count is 1 2024-10-08 16:49:24.399 INFO poller/poll_manager.go:214 Gateway Members changed, old: [] new: [root-kuscia-lite-bob] 2024-10-08 16:49:24.399 INFO queue/queue.go:124 Finish processing item: queue id[service-poll-queue], key[kuscia-handshake] (27.764µs) 2024-10-08 16:49:29.342 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:616 2024-10-08 16:49:29.345 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (42.848µs) 2024-10-08 16:49:29.350 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (7.546865ms) 2024-10-08 16:49:44.360 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (23.061µs) 2024-10-08 16:49:44.360 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:652 2024-10-08 16:49:44.368 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (8.425145ms) 2024-10-08 16:49:59.361 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (21.274µs) 2024-10-08 16:49:59.361 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:691 2024-10-08 16:49:59.368 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (6.816932ms) 2024-10-08 16:50:14.333 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (35.358µs) 2024-10-08 16:50:14.334 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:725 2024-10-08 16:50:14.341 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (6.842106ms) 2024-10-08 16:50:29.263 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (141.263µs) 2024-10-08 16:50:29.263 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:762 2024-10-08 16:50:29.270 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (7.114926ms) 2024-10-08 16:50:44.290 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (20.55µs) 2024-10-08 16:50:44.290 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:798 2024-10-08 16:50:44.296 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (6.39688ms) 2024-10-08 16:50:59.261 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (23.27µs) 2024-10-08 16:50:59.262 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:835 2024-10-08 16:50:59.277 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (14.669908ms) 2024-10-08 16:51:14.317 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (29.058µs) 2024-10-08 16:51:14.317 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:871 2024-10-08 16:51:14.325 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (7.487311ms) 2024-10-08 16:51:29.330 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (30.413µs) 2024-10-08 16:51:29.330 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:908 2024-10-08 16:51:29.340 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (9.485851ms) 2024-10-08 16:51:44.306 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (32.582µs) 2024-10-08 16:51:44.306 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:943 2024-10-08 16:51:44.315 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (9.05695ms) 2024-10-08 16:51:59.339 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (29.268µs) 2024-10-08 16:51:59.339 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:982 2024-10-08 16:51:59.346 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (7.119781ms) 2024-10-08 16:52:14.450 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:1017 2024-10-08 16:52:14.453 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (38.877µs) 2024-10-08 16:52:14.460 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (9.570194ms) 2024-10-08 16:52:29.330 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (32.109µs) 2024-10-08 16:52:29.330 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:1055 2024-10-08 16:52:29.338 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (7.6754ms) 2024-10-08 16:52:44.327 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (70.57µs) 2024-10-08 16:52:44.327 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:1087 2024-10-08 16:52:44.335 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (7.576311ms) 2024-10-08 16:52:59.334 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (33.202µs) 2024-10-08 16:52:59.334 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:1126 2024-10-08 16:52:59.351 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (17.257494ms) 2024-10-08 16:53:14.369 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (172.29µs) 2024-10-08 16:53:14.369 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:1159 2024-10-08 16:53:14.387 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (18.279588ms) 2024-10-08 16:53:29.354 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (26.213µs) 2024-10-08 16:53:29.354 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:1195 2024-10-08 16:53:29.361 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (6.875119ms) 2024-10-08 16:53:44.390 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (33.874µs) 2024-10-08 16:53:44.390 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:1231 2024-10-08 16:53:44.397 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (6.765256ms) 2024-10-08 16:53:59.354 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (31.359µs) 2024-10-08 16:53:59.355 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:1267 2024-10-08 16:53:59.361 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (6.654614ms) 2024-10-08 16:54:14.350 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (32.083µs) 2024-10-08 16:54:14.350 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:1301 2024-10-08 16:54:14.357 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (6.919522ms) 2024-10-08 16:54:29.273 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (34.776µs) 2024-10-08 16:54:29.273 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:1338 2024-10-08 16:54:29.288 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (15.121756ms) 2024-10-08 16:54:44.335 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:1374 2024-10-08 16:54:44.335 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (31.545µs) 2024-10-08 16:54:44.344 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (8.898661ms) 2024-10-08 16:54:59.342 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (25.146µs) 2024-10-08 16:54:59.342 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:1412 2024-10-08 16:54:59.349 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (6.837898ms) 2024-10-08 16:55:14.346 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (30.294µs) 2024-10-08 16:55:14.346 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:1446 2024-10-08 16:55:14.355 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (8.712514ms) 2024-10-08 16:55:29.292 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (25.628µs) 2024-10-08 16:55:29.293 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:1483 2024-10-08 16:55:29.300 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (7.648031ms) 2024-10-08 16:55:44.314 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (22.131µs) 2024-10-08 16:55:44.314 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:1518 2024-10-08 16:55:44.320 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (6.521031ms) 2024-10-08 16:55:59.342 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (28.535µs) 2024-10-08 16:55:59.343 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:1556 2024-10-08 16:55:59.354 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (10.831286ms) 2024-10-08 16:56:14.397 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (33.401µs) 2024-10-08 16:56:14.397 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:1589 2024-10-08 16:56:14.408 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (10.743457ms) 2024-10-08 16:56:29.362 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:1628 2024-10-08 16:56:29.362 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (33.402µs) 2024-10-08 16:56:29.369 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (7.154883ms) 2024-10-08 16:56:44.342 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:1661 2024-10-08 16:56:44.350 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (33.49µs) 2024-10-08 16:56:44.351 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (8.675206ms) 2024-10-08 16:56:59.394 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (62.905µs) 2024-10-08 16:56:59.394 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:1700 2024-10-08 16:56:59.413 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (18.909267ms) 2024-10-08 16:57:14.406 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (18.085µs) 2024-10-08 16:57:14.406 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:1733 2024-10-08 16:57:14.415 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (9.151895ms) 2024-10-08 16:57:29.302 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (28.255µs) 2024-10-08 16:57:29.302 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:1769 2024-10-08 16:57:29.322 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (20.068623ms) 2024-10-08 16:57:44.467 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (29.574µs) 2024-10-08 16:57:44.468 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:1806 2024-10-08 16:57:44.571 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (102.725631ms) 2024-10-08 16:57:59.386 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (31.538µs) 2024-10-08 16:57:59.386 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:1844 2024-10-08 16:57:59.393 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (6.926001ms) 2024-10-08 16:58:14.309 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (69.651µs) 2024-10-08 16:58:14.309 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:1878 2024-10-08 16:58:14.316 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (7.582926ms) 2024-10-08 16:58:29.319 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (26.456µs) 2024-10-08 16:58:29.320 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:1919 2024-10-08 16:58:29.329 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (8.760617ms) 2024-10-08 16:58:44.398 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (28.232µs) 2024-10-08 16:58:44.398 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:1953 2024-10-08 16:58:44.405 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (6.676437ms) 2024-10-08 16:58:55.642 INFO decorator/proto_decorator.go:165 REQUEST Path: /api/v1/datamesh/domaindatagrant/create Content-Type: application/json Body: {"domaindataId":"bob-table","grantDomain":"alice"} 2024-10-08 16:58:55.700 INFO service/domaindatagrant.go:111 Create DomainDataGrant bob/domaindatagrant-3b099891ab5fe47ecff72440302595f4 2024-10-08 16:58:55.700 INFO decorator/proto_decorator.go:216 RESPONSE{"status":{"message":"success"},"data":{"domaindatagrantId":"domaindatagrant-3b099891ab5fe47ecff72440302595f4"}} 2024-10-08 16:58:56.205 INFO decorator/proto_decorator.go:165 REQUEST Path: /api/v1/datamesh/domaindatagrant/create Content-Type: application/json Body: {"domaindataId":"bob-dp-table","grantDomain":"alice"} 2024-10-08 16:58:56.255 INFO service/domaindatagrant.go:111 Create DomainDataGrant bob/domaindatagrant-7b9b7e796ad31fe3564eba9596b827c3 2024-10-08 16:58:56.256 INFO decorator/proto_decorator.go:216 RESPONSE{"status":{"message":"success"},"data":{"domaindatagrantId":"domaindatagrant-7b9b7e796ad31fe3564eba9596b827c3"}} 2024-10-08 16:58:59.359 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (26.067µs) 2024-10-08 16:58:59.359 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:2021 2024-10-08 16:58:59.366 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (6.907131ms) 2024-10-08 16:59:09.344 INFO controller/domain_route.go:357 Update DomainRoute bob/bob-kuscia-system revision:519 2024-10-08 16:59:09.349 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:2021 2024-10-08 16:59:09.353 INFO controller/domain_route.go:432 add cluster bob-to-alice name:http protocol:HTTP port:1080 2024-10-08 16:59:09.353 INFO xds/xds.go:439 Add cluster:bob-to-alice-http 2024-10-08 16:59:09.353 INFO controller/domain_route.go:357 Update DomainRoute bob/bob-alice revision:595 2024-10-08 16:59:09.357 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (8.625452ms) 2024-10-08 16:59:09.358 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (26.078µs) 2024-10-08 16:59:09.360 INFO xds/xds.go:439 Add cluster:bob-to-alice-http 2024-10-08 16:59:09.360 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/bob-alice] (7.08766ms) 2024-10-08 16:59:09.365 INFO xds/xds.go:439 Add cluster:service-masterproxy 2024-10-08 16:59:09.365 INFO controller/domain_route.go:497 Update rule to master success 2024-10-08 16:59:09.365 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/bob-kuscia-system] (21.184916ms) 2024-10-08 16:59:14.358 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (32.46µs) 2024-10-08 16:59:14.358 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:2056 2024-10-08 16:59:14.365 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (7.027253ms) 2024-10-08 16:59:29.327 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (29.41µs) 2024-10-08 16:59:29.327 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:2099 2024-10-08 16:59:29.333 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (6.531955ms) 2024-10-08 16:59:44.307 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (29.44µs) 2024-10-08 16:59:44.307 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:2132 2024-10-08 16:59:44.314 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (7.065366ms) 2024-10-08 16:59:59.330 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (31.285µs) 2024-10-08 16:59:59.330 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:2171 2024-10-08 16:59:59.339 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (8.56145ms) 2024-10-08 17:00:14.334 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (31.915µs) 2024-10-08 17:00:14.335 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:2204 2024-10-08 17:00:14.343 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (8.179401ms) 2024-10-08 17:00:29.351 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (25.845µs) 2024-10-08 17:00:29.351 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:2243 2024-10-08 17:00:29.358 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (6.607891ms) 2024-10-08 17:00:44.315 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (26.446µs) 2024-10-08 17:00:44.316 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:2278 2024-10-08 17:00:44.323 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (7.089398ms) 2024-10-08 17:00:59.423 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:2316 2024-10-08 17:00:59.425 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (36.008µs) 2024-10-08 17:00:59.430 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (7.464379ms) 2024-10-08 17:01:14.360 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (27.519µs) 2024-10-08 17:01:14.360 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:2350 2024-10-08 17:01:14.369 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (9.094935ms) 2024-10-08 17:01:29.319 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (25.284µs) 2024-10-08 17:01:29.319 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:2387 2024-10-08 17:01:29.335 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (15.888693ms) 2024-10-08 17:01:44.319 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (31.854µs) 2024-10-08 17:01:44.319 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:2422 2024-10-08 17:01:44.326 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (6.82308ms) 2024-10-08 17:01:59.382 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (94.792µs) 2024-10-08 17:01:59.382 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:2459 2024-10-08 17:01:59.391 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (8.119623ms) 2024-10-08 17:02:14.332 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (19.458µs) 2024-10-08 17:02:14.332 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:2495 2024-10-08 17:02:14.337 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (4.512272ms) 2024-10-08 17:02:18.974 INFO queue/queue.go:124 Finish processing item: queue id[service-poll-queue], key[bob/secretflow-task-20241008170216-single-psi-0-spu] (35.647µs) 2024-10-08 17:02:18.976 INFO queue/queue.go:124 Finish processing item: queue id[kuscia-coredns-controller], key[bob/secretflow-task-20241008170216-single-psi-0-spu] (55.581µs) 2024-10-08 17:02:19.038 INFO queue/queue.go:124 Finish processing item: queue id[kuscia-coredns-controller], key[bob/secretflow-task-20241008170216-single-psi-0-fed] (27.787µs) 2024-10-08 17:02:19.050 INFO queue/queue.go:124 Finish processing item: queue id[service-poll-queue], key[bob/secretflow-task-20241008170216-single-psi-0-fed] (24.783µs) 2024-10-08 17:02:19.073 INFO controller/endpoints.go:189 Updating endpoint bob/secretflow-task-20241008170216-single-psi-0-spu/2530 2024-10-08 17:02:19.074 INFO queue/queue.go:124 Finish processing item: queue id[endpoints-queue], key[bob/secretflow-task-20241008170216-single-psi-0-spu] (29.906µs) 2024-10-08 17:02:19.074 INFO queue/queue.go:124 Finish processing item: queue id[kuscia-coredns-controller], key[bob/secretflow-task-20241008170216-single-psi-0-spu] (19.055µs) 2024-10-08 17:02:19.109 INFO queue/queue.go:124 Finish processing item: queue id[kuscia-coredns-controller], key[bob/secretflow-task-20241008170216-single-psi-0-global] (60.127µs) 2024-10-08 17:02:19.118 INFO controller/endpoints.go:189 Updating endpoint bob/secretflow-task-20241008170216-single-psi-0-fed/2534 2024-10-08 17:02:19.118 INFO queue/queue.go:124 Finish processing item: queue id[endpoints-queue], key[bob/secretflow-task-20241008170216-single-psi-0-fed] (76.066µs) 2024-10-08 17:02:19.119 INFO queue/queue.go:124 Finish processing item: queue id[kuscia-coredns-controller], key[bob/secretflow-task-20241008170216-single-psi-0-fed] (27.776µs) 2024-10-08 17:02:19.153 INFO controller/endpoints.go:189 Updating endpoint bob/secretflow-task-20241008170216-single-psi-0-global/2537 2024-10-08 17:02:19.153 INFO queue/queue.go:124 Finish processing item: queue id[endpoints-queue], key[bob/secretflow-task-20241008170216-single-psi-0-global] (58.759µs) 2024-10-08 17:02:19.165 INFO queue/queue.go:124 Finish processing item: queue id[kuscia-coredns-controller], key[bob/secretflow-task-20241008170216-single-psi-0-global] (31.647µs) 2024-10-08 17:02:20.699 INFO source/apiserver.go:58 Receive pod "secretflow-task-20241008170216-single-psi-0_bob(e7655aa2-dda6-4e0b-8e15-0d8bd4d8104b)" add event from apiserver 2024-10-08 17:02:20.700 INFO source/config.go:100 Pod change merged, source=api, adds=[secretflow-task-20241008170216-single-psi-0_bob(e7655aa2-dda6-4e0b-8e15-0d8bd4d8104b)], updates=[], deletes=[], removes=[], reconciles=[] 2024-10-08 17:02:20.700 INFO framework/pods_controller.go:263 SyncLoop ADD, source=api, pods=[secretflow-task-20241008170216-single-psi-0_bob(e7655aa2-dda6-4e0b-8e15-0d8bd4d8104b)] 2024-10-08 17:02:20.732 INFO framework/pods_controller.go:515 Sync pod "secretflow-task-20241008170216-single-psi-0_bob(e7655aa2-dda6-4e0b-8e15-0d8bd4d8104b)" enter 2024-10-08 17:02:20.744 INFO pod/cri_provider.go:760 CRIProvider start syncing pod "secretflow-task-20241008170216-single-psi-0_bob(e7655aa2-dda6-4e0b-8e15-0d8bd4d8104b)" 2024-10-08 17:02:20.767 INFO resource/volume_manager.go:178 Mount (dump) file success, path=/home/kuscia/var/pods/e7655aa2-dda6-4e0b-8e15-0d8bd4d8104b/volumes/kubernetes.io~configmap/secretflow-task-20241008170216-single-psi-configtemplate/task-config.conf, mode=420, size=176 2024-10-08 17:02:20.768 INFO resource/volume_manager.go:103 Mount volumes map[config-template:{HostPath:/home/kuscia/var/pods/e7655aa2-dda6-4e0b-8e15-0d8bd4d8104b/volumes/kubernetes.io~configmap/secretflow-task-20241008170216-single-psi-configtemplate ReadOnly:true Managed:true SELinuxRelabel:true}] for pod "secretflow-task-20241008170216-single-psi-0" succeed 2024-10-08 17:02:20.770 INFO kuberuntime/kuberuntime_manager.go:325 No sandbox for pod "secretflow-task-20241008170216-single-psi-0_bob(e7655aa2-dda6-4e0b-8e15-0d8bd4d8104b)" can be found. Need to start a new one 2024-10-08 17:02:20.771 INFO kuberuntime/kuberuntime_manager.go:547 ComputePodActions got for pod "secretflow-task-20241008170216-single-psi-0_bob(e7655aa2-dda6-4e0b-8e15-0d8bd4d8104b)": {true true 0 nil [0] map[] []} 2024-10-08 17:02:20.789 INFO record/event.go:285 Event(v1.ObjectReference{Kind:"Node", Namespace:"bob", Name:"root-kuscia-lite-bob", UID:"root-kuscia-lite-bob", APIVersion:"", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'MissingClusterDNS' kubelet does not have ClusterDNS IP configured and cannot create Pod using "ClusterFirst" policy. Falling back to "Default" policy. 2024-10-08 17:02:20.817 INFO record/event.go:285 Event(v1.ObjectReference{Kind:"Pod", Namespace:"bob", Name:"secretflow-task-20241008170216-single-psi-0", UID:"e7655aa2-dda6-4e0b-8e15-0d8bd4d8104b", APIVersion:"v1", ResourceVersion:"2562", FieldPath:""}): type: 'Warning' reason: 'MissingClusterDNS' pod: "secretflow-task-20241008170216-single-psi-0_bob(e7655aa2-dda6-4e0b-8e15-0d8bd4d8104b)". kubelet does not have ClusterDNS IP configured and cannot create Pod using "ClusterFirst" policy. Falling back to "Default" policy. 2024-10-08 17:02:21.196 INFO source/apiserver.go:65 Receive pod "secretflow-task-20241008170216-single-psi-0_bob(e7655aa2-dda6-4e0b-8e15-0d8bd4d8104b)" update event from apiserver 2024-10-08 17:02:21.201 INFO source/config.go:100 Pod change merged, source=api, adds=[], updates=[], deletes=[], removes=[], reconciles=[secretflow-task-20241008170216-single-psi-0_bob(e7655aa2-dda6-4e0b-8e15-0d8bd4d8104b)] 2024-10-08 17:02:21.202 INFO framework/pods_controller.go:276 SyncLoop RECONCILE, source=api, pods=[secretflow-task-20241008170216-single-psi-0_bob(e7655aa2-dda6-4e0b-8e15-0d8bd4d8104b)] 2024-10-08 17:02:21.214 INFO status/status_manager.go:625 Patch status for pod "secretflow-task-20241008170216-single-psi-0_bob(e7655aa2-dda6-4e0b-8e15-0d8bd4d8104b)", patch={"metadata":{"uid":"e7655aa2-dda6-4e0b-8e15-0d8bd4d8104b"},"status":{"$setElementOrder/conditions":[{"type":"Initialized"},{"type":"Ready"},{"type":"ContainersReady"},{"type":"PodScheduled"}],"conditions":[{"lastProbeTime":null,"lastTransitionTime":"2024-10-08T09:02:20Z","status":"True","type":"Initialized"},{"lastProbeTime":null,"lastTransitionTime":"2024-10-08T09:02:20Z","message":"containers with unready status: [secretflow]","reason":"ContainersNotReady","status":"False","type":"Ready"},{"lastProbeTime":null,"lastTransitionTime":"2024-10-08T09:02:20Z","message":"containers with unready status: [secretflow]","reason":"ContainersNotReady","status":"False","type":"ContainersReady"}],"containerStatuses":[{"image":"secretflow-registry.cn-hangzhou.cr.aliyuncs.com/secretflow/secretflow-lite-anolis8:1.7.0b0","imageID":"","lastState":{},"name":"secretflow","ready":false,"restartCount":0,"started":false,"state":{"waiting":{"reason":"ContainerCreating"}}}],"hostIP":"172.20.0.4","qosClass":null,"startTime":"2024-10-08T09:02:20Z"}} 2024-10-08 17:02:22.961 INFO record/event.go:285 Event(v1.ObjectReference{Kind:"Node", Namespace:"bob", Name:"root-kuscia-lite-bob", UID:"root-kuscia-lite-bob", APIVersion:"", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'MissingClusterDNS' kubelet does not have ClusterDNS IP configured and cannot create Pod using "ClusterFirst" policy. Falling back to "Default" policy. 2024-10-08 17:02:22.962 INFO record/event.go:285 Event(v1.ObjectReference{Kind:"Pod", Namespace:"bob", Name:"secretflow-task-20241008170216-single-psi-0", UID:"e7655aa2-dda6-4e0b-8e15-0d8bd4d8104b", APIVersion:"v1", ResourceVersion:"2562", FieldPath:""}): type: 'Warning' reason: 'MissingClusterDNS' pod: "secretflow-task-20241008170216-single-psi-0_bob(e7655aa2-dda6-4e0b-8e15-0d8bd4d8104b)". kubelet does not have ClusterDNS IP configured and cannot create Pod using "ClusterFirst" policy. Falling back to "Default" policy. 2024-10-08 17:02:22.987 INFO record/event.go:285 Event(v1.ObjectReference{Kind:"Pod", Namespace:"bob", Name:"secretflow-task-20241008170216-single-psi-0", UID:"e7655aa2-dda6-4e0b-8e15-0d8bd4d8104b", APIVersion:"v1", ResourceVersion:"2562", FieldPath:"spec.containers{secretflow}"}): type: 'Normal' reason: 'Pulled' Container image "secretflow-registry.cn-hangzhou.cr.aliyuncs.com/secretflow/secretflow-lite-anolis8:1.7.0b0" already present on machine 2024-10-08 17:02:23.268 INFO pod/cri_provider.go:348 Receive pleg event: &{ID:e7655aa2-dda6-4e0b-8e15-0d8bd4d8104b Type:ContainerStarted Data:80669c65b6a10e69276b52a889ec7483587e02497938a11848b7ed258ebcd489} 2024-10-08 17:02:23.905 INFO certissuance/cert_issuance.go:302 Successfully issued certificate(server=true,client=true) for container "secretflow" in pod "secretflow-task-20241008170216-single-psi-0_bob(e7655aa2-dda6-4e0b-8e15-0d8bd4d8104b)" 2024-10-08 17:02:23.909 INFO configrender/config_render.go:245 Render config template for container "secretflow" in pod "secretflow-task-20241008170216-single-psi-0_bob(e7655aa2-dda6-4e0b-8e15-0d8bd4d8104b)" succeed, templatePath=/home/kuscia/var/pods/e7655aa2-dda6-4e0b-8e15-0d8bd4d8104b/volumes/kubernetes.io~configmap/secretflow-task-20241008170216-single-psi-configtemplate/task-config.conf, configPath=/home/kuscia/var/pods/e7655aa2-dda6-4e0b-8e15-0d8bd4d8104b/volumes/config-render/secretflow/config-template/task-config.conf 2024-10-08 17:02:23.910 INFO pod/cri_provider.go:577 Successfully generated the run options of container "secretflow" in pod "secretflow-task-20241008170216-single-psi-0_bob(e7655aa2-dda6-4e0b-8e15-0d8bd4d8104b)" 2024-10-08 17:02:24.343 INFO record/event.go:285 Event(v1.ObjectReference{Kind:"Pod", Namespace:"bob", Name:"secretflow-task-20241008170216-single-psi-0", UID:"e7655aa2-dda6-4e0b-8e15-0d8bd4d8104b", APIVersion:"v1", ResourceVersion:"2562", FieldPath:"spec.containers{secretflow}"}): type: 'Normal' reason: 'Created' Created container secretflow 2024-10-08 17:02:24.498 INFO provider/provider_factory.go:126 Finish initializing cgroup 2024-10-08 17:02:25.034 INFO framework/pods_controller.go:517 Sync pod "secretflow-task-20241008170216-single-psi-0_bob(e7655aa2-dda6-4e0b-8e15-0d8bd4d8104b)" exit, isTerminal=false 2024-10-08 17:02:25.035 INFO record/event.go:285 Event(v1.ObjectReference{Kind:"Pod", Namespace:"bob", Name:"secretflow-task-20241008170216-single-psi-0", UID:"e7655aa2-dda6-4e0b-8e15-0d8bd4d8104b", APIVersion:"v1", ResourceVersion:"2562", FieldPath:"spec.containers{secretflow}"}): type: 'Normal' reason: 'Started' Started container secretflow 2024-10-08 17:02:25.048 INFO framework/pods_controller.go:515 Sync pod "secretflow-task-20241008170216-single-psi-0_bob(e7655aa2-dda6-4e0b-8e15-0d8bd4d8104b)" enter 2024-10-08 17:02:25.061 INFO pod/cri_provider.go:760 CRIProvider start syncing pod "secretflow-task-20241008170216-single-psi-0_bob(e7655aa2-dda6-4e0b-8e15-0d8bd4d8104b)" 2024-10-08 17:02:25.063 INFO resource/volume_manager.go:178 Mount (dump) file success, path=/home/kuscia/var/pods/e7655aa2-dda6-4e0b-8e15-0d8bd4d8104b/volumes/kubernetes.io~configmap/secretflow-task-20241008170216-single-psi-configtemplate/task-config.conf, mode=420, size=176 2024-10-08 17:02:25.063 INFO resource/volume_manager.go:103 Mount volumes map[config-template:{HostPath:/home/kuscia/var/pods/e7655aa2-dda6-4e0b-8e15-0d8bd4d8104b/volumes/kubernetes.io~configmap/secretflow-task-20241008170216-single-psi-configtemplate ReadOnly:true Managed:true SELinuxRelabel:true}] for pod "secretflow-task-20241008170216-single-psi-0" succeed 2024-10-08 17:02:25.064 INFO kuberuntime/kuberuntime_manager.go:547 ComputePodActions got for pod "secretflow-task-20241008170216-single-psi-0_bob(e7655aa2-dda6-4e0b-8e15-0d8bd4d8104b)": {false false 80669c65b6a10e69276b52a889ec7483587e02497938a11848b7ed258ebcd489 0 nil [] map[] []} 2024-10-08 17:02:25.064 INFO framework/pods_controller.go:517 Sync pod "secretflow-task-20241008170216-single-psi-0_bob(e7655aa2-dda6-4e0b-8e15-0d8bd4d8104b)" exit, isTerminal=false 2024-10-08 17:02:25.065 INFO record/event.go:285 Event(v1.ObjectReference{Kind:"Node", Namespace:"bob", Name:"root-kuscia-lite-bob", UID:"root-kuscia-lite-bob", APIVersion:"", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'MissingClusterDNS' kubelet does not have ClusterDNS IP configured and cannot create Pod using "ClusterFirst" policy. Falling back to "Default" policy. 2024-10-08 17:02:25.065 INFO record/event.go:285 Event(v1.ObjectReference{Kind:"Pod", Namespace:"bob", Name:"secretflow-task-20241008170216-single-psi-0", UID:"e7655aa2-dda6-4e0b-8e15-0d8bd4d8104b", APIVersion:"v1", ResourceVersion:"2562", FieldPath:""}): type: 'Warning' reason: 'MissingClusterDNS' pod: "secretflow-task-20241008170216-single-psi-0_bob(e7655aa2-dda6-4e0b-8e15-0d8bd4d8104b)". kubelet does not have ClusterDNS IP configured and cannot create Pod using "ClusterFirst" policy. Falling back to "Default" policy. 2024-10-08 17:02:25.178 INFO status/status_manager.go:625 Patch status for pod "secretflow-task-20241008170216-single-psi-0_bob(e7655aa2-dda6-4e0b-8e15-0d8bd4d8104b)", patch={"metadata":{"uid":"e7655aa2-dda6-4e0b-8e15-0d8bd4d8104b"},"status":{"$setElementOrder/conditions":[{"type":"Initialized"},{"type":"Ready"},{"type":"ContainersReady"},{"type":"PodScheduled"}],"conditions":[{"lastTransitionTime":"2024-10-08T09:02:25Z","message":null,"reason":null,"status":"True","type":"Ready"},{"lastTransitionTime":"2024-10-08T09:02:25Z","message":null,"reason":null,"status":"True","type":"ContainersReady"}],"containerStatuses":[{"containerID":"containerd://0e2fd5be280766df6ab3317af500a83a46d124bd254520effaf431e4f69792cd","image":"secretflow-registry.cn-hangzhou.cr.aliyuncs.com/secretflow/secretflow-lite-anolis8:1.7.0b0","imageID":"sha256:c9cb78fa836692b01e7fcb147663d2deb0bdac8118e61087d627d15b7f758729","lastState":{},"name":"secretflow","ready":true,"restartCount":0,"started":true,"state":{"running":{"startedAt":"2024-10-08T09:02:25Z"}}}],"phase":"Running","podIP":"10.88.0.2","podIPs":[{"ip":"10.88.0.2"}]}} 2024-10-08 17:02:25.190 INFO source/apiserver.go:65 Receive pod "secretflow-task-20241008170216-single-psi-0_bob(e7655aa2-dda6-4e0b-8e15-0d8bd4d8104b)" update event from apiserver 2024-10-08 17:02:25.191 INFO source/config.go:100 Pod change merged, source=api, adds=[], updates=[], deletes=[], removes=[], reconciles=[secretflow-task-20241008170216-single-psi-0_bob(e7655aa2-dda6-4e0b-8e15-0d8bd4d8104b)] 2024-10-08 17:02:25.191 INFO framework/pods_controller.go:276 SyncLoop RECONCILE, source=api, pods=[secretflow-task-20241008170216-single-psi-0_bob(e7655aa2-dda6-4e0b-8e15-0d8bd4d8104b)] 2024-10-08 17:02:25.266 INFO queue/queue.go:124 Finish processing item: queue id[kuscia-coredns-controller], key[bob/secretflow-task-20241008170216-single-psi-0-global] (26.939µs) 2024-10-08 17:02:25.273 INFO controller/endpoints.go:189 Updating endpoint bob/secretflow-task-20241008170216-single-psi-0-global/2601 2024-10-08 17:02:25.283 INFO queue/queue.go:124 Finish processing item: queue id[kuscia-coredns-controller], key[bob/secretflow-task-20241008170216-single-psi-0-fed] (31.384µs) 2024-10-08 17:02:25.284 INFO controller/endpoints.go:189 Updating endpoint bob/secretflow-task-20241008170216-single-psi-0-fed/2602 2024-10-08 17:02:25.287 INFO queue/queue.go:124 Finish processing item: queue id[kuscia-coredns-controller], key[bob/secretflow-task-20241008170216-single-psi-0-spu] (24.242µs) 2024-10-08 17:02:25.289 INFO xds/xds.go:439 Add cluster:service-secretflow-task-20241008170216-single-psi-0-global 2024-10-08 17:02:25.290 INFO controller/endpoints.go:189 Updating endpoint bob/secretflow-task-20241008170216-single-psi-0-spu/2604 2024-10-08 17:02:25.292 INFO xds/xds.go:439 Add cluster:service-secretflow-task-20241008170216-single-psi-0-spu 2024-10-08 17:02:25.295 INFO pod/cri_provider.go:348 Receive pleg event: &{ID:e7655aa2-dda6-4e0b-8e15-0d8bd4d8104b Type:ContainerStarted Data:0e2fd5be280766df6ab3317af500a83a46d124bd254520effaf431e4f69792cd} 2024-10-08 17:02:25.299 INFO framework/pods_controller.go:515 Sync pod "secretflow-task-20241008170216-single-psi-0_bob(e7655aa2-dda6-4e0b-8e15-0d8bd4d8104b)" enter 2024-10-08 17:02:25.299 INFO status/status_manager.go:490 Ignoring same status for pod "secretflow-task-20241008170216-single-psi-0_bob(e7655aa2-dda6-4e0b-8e15-0d8bd4d8104b)" 2024-10-08 17:02:25.299 INFO pod/cri_provider.go:760 CRIProvider start syncing pod "secretflow-task-20241008170216-single-psi-0_bob(e7655aa2-dda6-4e0b-8e15-0d8bd4d8104b)" 2024-10-08 17:02:25.300 INFO resource/volume_manager.go:178 Mount (dump) file success, path=/home/kuscia/var/pods/e7655aa2-dda6-4e0b-8e15-0d8bd4d8104b/volumes/kubernetes.io~configmap/secretflow-task-20241008170216-single-psi-configtemplate/task-config.conf, mode=420, size=176 2024-10-08 17:02:25.300 INFO resource/volume_manager.go:103 Mount volumes map[config-template:{HostPath:/home/kuscia/var/pods/e7655aa2-dda6-4e0b-8e15-0d8bd4d8104b/volumes/kubernetes.io~configmap/secretflow-task-20241008170216-single-psi-configtemplate ReadOnly:true Managed:true SELinuxRelabel:true}] for pod "secretflow-task-20241008170216-single-psi-0" succeed 2024-10-08 17:02:25.303 INFO kuberuntime/kuberuntime_manager.go:547 ComputePodActions got for pod "secretflow-task-20241008170216-single-psi-0_bob(e7655aa2-dda6-4e0b-8e15-0d8bd4d8104b)": {false false 80669c65b6a10e69276b52a889ec7483587e02497938a11848b7ed258ebcd489 0 nil [] map[] []} 2024-10-08 17:02:25.303 INFO framework/pods_controller.go:517 Sync pod "secretflow-task-20241008170216-single-psi-0_bob(e7655aa2-dda6-4e0b-8e15-0d8bd4d8104b)" exit, isTerminal=false 2024-10-08 17:02:25.304 INFO record/event.go:285 Event(v1.ObjectReference{Kind:"Node", Namespace:"bob", Name:"root-kuscia-lite-bob", UID:"root-kuscia-lite-bob", APIVersion:"", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'MissingClusterDNS' kubelet does not have ClusterDNS IP configured and cannot create Pod using "ClusterFirst" policy. Falling back to "Default" policy. 2024-10-08 17:02:25.304 INFO record/event.go:285 Event(v1.ObjectReference{Kind:"Pod", Namespace:"bob", Name:"secretflow-task-20241008170216-single-psi-0", UID:"e7655aa2-dda6-4e0b-8e15-0d8bd4d8104b", APIVersion:"v1", ResourceVersion:"2562", FieldPath:""}): type: 'Warning' reason: 'MissingClusterDNS' pod: "secretflow-task-20241008170216-single-psi-0_bob(e7655aa2-dda6-4e0b-8e15-0d8bd4d8104b)". kubelet does not have ClusterDNS IP configured and cannot create Pod using "ClusterFirst" policy. Falling back to "Default" policy. 2024-10-08 17:02:25.314 INFO xds/xds.go:439 Add cluster:service-secretflow-task-20241008170216-single-psi-0-fed 2024-10-08 17:02:25.390 INFO queue/queue.go:124 Finish processing item: queue id[kuscia-coredns-controller], key[bob/secretflow-task-20241008170216-single-psi-0-fed] (26.678µs) 2024-10-08 17:02:25.402 INFO queue/queue.go:124 Finish processing item: queue id[service-poll-queue], key[bob/secretflow-task-20241008170216-single-psi-0-fed] (39.566µs) 2024-10-08 17:02:25.413 INFO queue/queue.go:124 Finish processing item: queue id[endpoints-queue], key[bob/secretflow-task-20241008170216-single-psi-0-spu] (121.375616ms) 2024-10-08 17:02:25.414 INFO queue/queue.go:124 Finish processing item: queue id[endpoints-queue], key[bob/secretflow-task-20241008170216-single-psi-0-fed] (130.011333ms) 2024-10-08 17:02:25.446 INFO queue/queue.go:124 Finish processing item: queue id[kuscia-coredns-controller], key[bob/secretflow-task-20241008170216-single-psi-0-global] (33.241µs) 2024-10-08 17:02:25.447 INFO queue/queue.go:124 Finish processing item: queue id[kuscia-coredns-controller], key[bob/secretflow-task-20241008170216-single-psi-0-spu] (42.183µs) 2024-10-08 17:02:25.447 INFO queue/queue.go:124 Finish processing item: queue id[endpoints-queue], key[bob/secretflow-task-20241008170216-single-psi-0-global] (174.058957ms) 2024-10-08 17:02:25.460 INFO queue/queue.go:124 Finish processing item: queue id[service-poll-queue], key[bob/secretflow-task-20241008170216-single-psi-0-spu] (35.289µs) 2024-10-08 17:02:26.347 INFO pleg/generic.go:303 Generic (PLEG): container finished, podID=e7655aa2-dda6-4e0b-8e15-0d8bd4d8104b, containerID=0e2fd5be280766df6ab3317af500a83a46d124bd254520effaf431e4f69792cd, exitCode=255 2024-10-08 17:02:26.348 INFO pod/cri_provider.go:348 Receive pleg event: &{ID:e7655aa2-dda6-4e0b-8e15-0d8bd4d8104b Type:ContainerDied Data:0e2fd5be280766df6ab3317af500a83a46d124bd254520effaf431e4f69792cd} 2024-10-08 17:02:26.354 INFO framework/pods_controller.go:515 Sync pod "secretflow-task-20241008170216-single-psi-0_bob(e7655aa2-dda6-4e0b-8e15-0d8bd4d8104b)" enter 2024-10-08 17:02:26.354 INFO framework/pods_controller.go:517 Sync pod "secretflow-task-20241008170216-single-psi-0_bob(e7655aa2-dda6-4e0b-8e15-0d8bd4d8104b)" exit, isTerminal=true 2024-10-08 17:02:26.371 INFO framework/pods_controller.go:589 Sync terminating pod "secretflow-task-20241008170216-single-psi-0_bob(e7655aa2-dda6-4e0b-8e15-0d8bd4d8104b)" enter 2024-10-08 17:02:26.372 INFO status/status_manager.go:490 Ignoring same status for pod "secretflow-task-20241008170216-single-psi-0_bob(e7655aa2-dda6-4e0b-8e15-0d8bd4d8104b)" 2024-10-08 17:02:26.372 INFO pod/cri_provider.go:800 CRIProvider start killing pod "secretflow-task-20241008170216-single-psi-0_bob(e7655aa2-dda6-4e0b-8e15-0d8bd4d8104b)" 2024-10-08 17:02:26.479 INFO source/apiserver.go:65 Receive pod "secretflow-task-20241008170216-single-psi-0_bob(e7655aa2-dda6-4e0b-8e15-0d8bd4d8104b)" update event from apiserver 2024-10-08 17:02:26.479 INFO source/config.go:100 Pod change merged, source=api, adds=[], updates=[], deletes=[], removes=[], reconciles=[secretflow-task-20241008170216-single-psi-0_bob(e7655aa2-dda6-4e0b-8e15-0d8bd4d8104b)] 2024-10-08 17:02:26.479 INFO framework/pods_controller.go:276 SyncLoop RECONCILE, source=api, pods=[secretflow-task-20241008170216-single-psi-0_bob(e7655aa2-dda6-4e0b-8e15-0d8bd4d8104b)] 2024-10-08 17:02:26.523 INFO status/status_manager.go:625 Patch status for pod "secretflow-task-20241008170216-single-psi-0_bob(e7655aa2-dda6-4e0b-8e15-0d8bd4d8104b)", patch={"metadata":{"uid":"e7655aa2-dda6-4e0b-8e15-0d8bd4d8104b"},"status":{"$setElementOrder/conditions":[{"type":"Initialized"},{"type":"Ready"},{"type":"ContainersReady"},{"type":"PodScheduled"}],"conditions":[{"lastTransitionTime":"2024-10-08T09:02:26Z","reason":"PodFailed","status":"False","type":"Ready"},{"lastTransitionTime":"2024-10-08T09:02:26Z","reason":"PodFailed","status":"False","type":"ContainersReady"}],"containerStatuses":[{"containerID":"containerd://0e2fd5be280766df6ab3317af500a83a46d124bd254520effaf431e4f69792cd","image":"secretflow-registry.cn-hangzhou.cr.aliyuncs.com/secretflow/secretflow-lite-anolis8:1.7.0b0","imageID":"sha256:c9cb78fa836692b01e7fcb147663d2deb0bdac8118e61087d627d15b7f758729","lastState":{},"name":"secretflow","ready":false,"restartCount":0,"started":false,"state":{"terminated":{"containerID":"containerd://0e2fd5be280766df6ab3317af500a83a46d124bd254520effaf431e4f69792cd","exitCode":255,"finishedAt":"2024-10-08T09:02:25Z","reason":"Error","startedAt":"2024-10-08T09:02:25Z"}}}]}} 2024-10-08 17:02:27.223 INFO framework/pods_controller.go:661 Sync terminating pod "secretflow-task-20241008170216-single-psi-0_bob(e7655aa2-dda6-4e0b-8e15-0d8bd4d8104b)" exit 2024-10-08 17:02:27.227 INFO framework/pods_controller.go:669 Sync terminated pod "secretflow-task-20241008170216-single-psi-0_bob(e7655aa2-dda6-4e0b-8e15-0d8bd4d8104b)" enter 2024-10-08 17:02:27.228 INFO pod/cri_provider.go:815 CRIProvider start deleting pod "secretflow-task-20241008170216-single-psi-0_bob(e7655aa2-dda6-4e0b-8e15-0d8bd4d8104b)" 2024-10-08 17:02:27.228 INFO framework/pods_controller.go:685 Sync terminated pod "secretflow-task-20241008170216-single-psi-0_bob(e7655aa2-dda6-4e0b-8e15-0d8bd4d8104b)" exit 2024-10-08 17:02:27.299 INFO source/apiserver.go:65 Receive pod "secretflow-task-20241008170216-single-psi-0_bob(e7655aa2-dda6-4e0b-8e15-0d8bd4d8104b)" update event from apiserver 2024-10-08 17:02:27.300 INFO source/config.go:100 Pod change merged, source=api, adds=[], updates=[], deletes=[], removes=[], reconciles=[secretflow-task-20241008170216-single-psi-0_bob(e7655aa2-dda6-4e0b-8e15-0d8bd4d8104b)] 2024-10-08 17:02:27.300 INFO framework/pods_controller.go:276 SyncLoop RECONCILE, source=api, pods=[secretflow-task-20241008170216-single-psi-0_bob(e7655aa2-dda6-4e0b-8e15-0d8bd4d8104b)] 2024-10-08 17:02:27.332 INFO status/status_manager.go:625 Patch status for pod "secretflow-task-20241008170216-single-psi-0_bob(e7655aa2-dda6-4e0b-8e15-0d8bd4d8104b)", patch={"metadata":{"uid":"e7655aa2-dda6-4e0b-8e15-0d8bd4d8104b"},"status":{"phase":"Failed","podIP":null,"podIPs":null}} 2024-10-08 17:02:27.350 INFO controller/endpoints.go:189 Updating endpoint bob/secretflow-task-20241008170216-single-psi-0-spu/2643 2024-10-08 17:02:27.350 INFO queue/queue.go:124 Finish processing item: queue id[endpoints-queue], key[bob/secretflow-task-20241008170216-single-psi-0-spu] (40.488µs) 2024-10-08 17:02:27.350 INFO queue/queue.go:124 Finish processing item: queue id[kuscia-coredns-controller], key[bob/secretflow-task-20241008170216-single-psi-0-spu] (19.89µs) 2024-10-08 17:02:27.354 INFO status/status_manager.go:625 Patch status for pod "secretflow-task-20241008170216-single-psi-0_bob(e7655aa2-dda6-4e0b-8e15-0d8bd4d8104b)", patch={"metadata":{"uid":"e7655aa2-dda6-4e0b-8e15-0d8bd4d8104b"}} 2024-10-08 17:02:27.360 INFO pod/cri_provider.go:348 Receive pleg event: &{ID:e7655aa2-dda6-4e0b-8e15-0d8bd4d8104b Type:ContainerDied Data:80669c65b6a10e69276b52a889ec7483587e02497938a11848b7ed258ebcd489} 2024-10-08 17:02:27.382 INFO queue/queue.go:124 Finish processing item: queue id[kuscia-coredns-controller], key[bob/secretflow-task-20241008170216-single-psi-0-fed] (29.03µs) 2024-10-08 17:02:27.382 INFO queue/queue.go:124 Finish processing item: queue id[kuscia-coredns-controller], key[bob/secretflow-task-20241008170216-single-psi-0-global] (18.602µs) 2024-10-08 17:02:27.390 INFO controller/endpoints.go:189 Updating endpoint bob/secretflow-task-20241008170216-single-psi-0-fed/2646 2024-10-08 17:02:27.390 INFO controller/endpoints.go:189 Updating endpoint bob/secretflow-task-20241008170216-single-psi-0-global/2648 2024-10-08 17:02:27.390 INFO queue/queue.go:124 Finish processing item: queue id[endpoints-queue], key[bob/secretflow-task-20241008170216-single-psi-0-fed] (71.131µs) 2024-10-08 17:02:27.390 INFO queue/queue.go:124 Finish processing item: queue id[endpoints-queue], key[bob/secretflow-task-20241008170216-single-psi-0-global] (14.312µs) 2024-10-08 17:02:29.302 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (18.023µs) 2024-10-08 17:02:29.302 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:2680 2024-10-08 17:02:29.309 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (6.894045ms) 2024-10-08 17:02:44.314 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (21.455µs) 2024-10-08 17:02:44.314 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:2716 2024-10-08 17:02:44.323 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (9.468081ms) 2024-10-08 17:02:59.259 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (25.558µs) 2024-10-08 17:02:59.259 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:2750 2024-10-08 17:02:59.289 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (29.725389ms) 2024-10-08 17:03:14.338 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (23.936µs) 2024-10-08 17:03:14.338 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:2788 2024-10-08 17:03:14.347 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (8.635663ms) 2024-10-08 17:03:29.426 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:2823 2024-10-08 17:03:29.433 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (7.185635ms) 2024-10-08 17:03:29.426 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (26.445µs) 2024-10-08 17:03:44.350 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (28.348µs) 2024-10-08 17:03:44.350 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:2858 2024-10-08 17:03:44.359 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (8.84586ms) 2024-10-08 17:03:59.327 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (25.963µs) 2024-10-08 17:03:59.327 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:2895 2024-10-08 17:03:59.334 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (6.934495ms) 2024-10-08 17:04:14.314 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (40.822µs) 2024-10-08 17:04:14.315 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:2930 2024-10-08 17:04:14.326 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (11.477121ms) 2024-10-08 17:04:29.367 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:2968 2024-10-08 17:04:29.383 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (34.09µs) 2024-10-08 17:04:29.395 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (28.404436ms) 2024-10-08 17:04:44.337 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (14.725µs) 2024-10-08 17:04:44.338 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:3004 2024-10-08 17:04:44.342 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (4.014193ms) 2024-10-08 17:04:59.382 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:3038 2024-10-08 17:04:59.385 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (34.694µs) 2024-10-08 17:04:59.390 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (7.976176ms) 2024-10-08 17:05:14.370 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (26.068µs) 2024-10-08 17:05:14.371 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:3073 2024-10-08 17:05:14.379 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (8.53768ms) 2024-10-08 17:05:29.317 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (23.927µs) 2024-10-08 17:05:29.317 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:3108 2024-10-08 17:05:29.325 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (7.845883ms) 2024-10-08 17:05:44.344 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (32.224µs) 2024-10-08 17:05:44.344 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:3142 2024-10-08 17:05:44.351 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (6.300043ms) 2024-10-08 17:05:59.329 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (31.489µs) 2024-10-08 17:05:59.329 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:3180 2024-10-08 17:05:59.338 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (8.744938ms) 2024-10-08 17:06:14.359 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (26.447µs) 2024-10-08 17:06:14.359 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:3215 2024-10-08 17:06:14.366 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (6.777955ms) 2024-10-08 17:06:29.366 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (29.234µs) 2024-10-08 17:06:29.366 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:3253 2024-10-08 17:06:29.372 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (6.359917ms) 2024-10-08 17:06:44.382 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:3288 2024-10-08 17:06:44.382 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (27.911µs) 2024-10-08 17:06:44.389 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (7.025715ms) 2024-10-08 17:06:59.354 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (33.433µs) 2024-10-08 17:06:59.354 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:3325 2024-10-08 17:06:59.363 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (9.222116ms) 2024-10-08 17:07:14.267 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (29.286µs) 2024-10-08 17:07:14.267 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:3358 2024-10-08 17:07:14.274 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (6.933039ms) 2024-10-08 17:07:29.339 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (30.629µs) 2024-10-08 17:07:29.339 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:3398 2024-10-08 17:07:29.346 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (6.796485ms) 2024-10-08 17:07:44.324 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:3433 2024-10-08 17:07:44.330 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (33.45µs) 2024-10-08 17:07:44.332 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (8.299351ms) 2024-10-08 17:07:59.336 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:3470 2024-10-08 17:07:59.341 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (35.892µs) 2024-10-08 17:07:59.347 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (11.089785ms) 2024-10-08 17:08:14.333 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (23.135µs) 2024-10-08 17:08:14.334 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:3506 2024-10-08 17:08:14.340 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (6.622315ms) 2024-10-08 17:08:29.266 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (31.051µs) 2024-10-08 17:08:29.266 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:3541 2024-10-08 17:08:29.274 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (7.632605ms) 2024-10-08 17:08:44.351 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (19.197µs) 2024-10-08 17:08:44.351 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:3578 2024-10-08 17:08:44.358 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (7.10602ms) 2024-10-08 17:08:59.370 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (48.129µs) 2024-10-08 17:08:59.370 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:3614 2024-10-08 17:08:59.379 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (8.612677ms) 2024-10-08 17:09:09.317 INFO queue/queue.go:124 Finish processing item: queue id[kuscia-coredns-controller], key[bob/secretflow-task-20241008170216-single-psi-0-spu] (37.305µs) 2024-10-08 17:09:09.317 INFO queue/queue.go:124 Finish processing item: queue id[kuscia-coredns-controller], key[bob/secretflow-task-20241008170216-single-psi-0-fed] (17.337µs) 2024-10-08 17:09:09.317 INFO queue/queue.go:124 Finish processing item: queue id[kuscia-coredns-controller], key[bob/secretflow-task-20241008170216-single-psi-0-global] (15.286µs) 2024-10-08 17:09:09.337 INFO controller/endpoints.go:189 Updating endpoint bob/secretflow-task-20241008170216-single-psi-0-spu/2643 2024-10-08 17:09:09.337 INFO controller/endpoints.go:189 Updating endpoint bob/secretflow-task-20241008170216-single-psi-0-fed/2646 2024-10-08 17:09:09.337 INFO queue/queue.go:124 Finish processing item: queue id[endpoints-queue], key[bob/secretflow-task-20241008170216-single-psi-0-spu] (76.564µs) 2024-10-08 17:09:09.338 INFO queue/queue.go:124 Finish processing item: queue id[endpoints-queue], key[bob/secretflow-task-20241008170216-single-psi-0-fed] (15.229µs) 2024-10-08 17:09:09.338 INFO controller/endpoints.go:189 Updating endpoint bob/secretflow-task-20241008170216-single-psi-0-global/2648 2024-10-08 17:09:09.338 INFO queue/queue.go:124 Finish processing item: queue id[endpoints-queue], key[bob/secretflow-task-20241008170216-single-psi-0-global] (12.19µs) 2024-10-08 17:09:09.353 INFO controller/domain_route.go:432 add cluster bob-to-alice name:http protocol:HTTP port:1080 2024-10-08 17:09:09.354 INFO xds/xds.go:439 Add cluster:bob-to-alice-http 2024-10-08 17:09:09.354 INFO controller/domain_route.go:357 Update DomainRoute bob/bob-alice revision:595 2024-10-08 17:09:09.357 INFO controller/domain_route.go:357 Update DomainRoute bob/bob-kuscia-system revision:519 2024-10-08 17:09:09.361 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:3614 2024-10-08 17:09:09.365 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (29.425µs) 2024-10-08 17:09:09.369 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (7.975188ms) 2024-10-08 17:09:09.373 INFO xds/xds.go:439 Add cluster:service-masterproxy 2024-10-08 17:09:09.374 INFO controller/domain_route.go:497 Update rule to master success 2024-10-08 17:09:09.374 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/bob-kuscia-system] (16.297738ms) 2024-10-08 17:09:09.377 INFO xds/xds.go:439 Add cluster:bob-to-alice-http 2024-10-08 17:09:09.377 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/bob-alice] (24.537201ms) 2024-10-08 17:09:09.389 INFO queue/queue.go:124 Finish processing item: queue id[service-poll-queue], key[bob/secretflow-task-20241008170216-single-psi-0-spu] (55.342µs) 2024-10-08 17:09:09.390 INFO queue/queue.go:124 Finish processing item: queue id[kuscia-coredns-controller], key[bob/secretflow-task-20241008170216-single-psi-0-fed] (16.921µs) 2024-10-08 17:09:09.390 INFO queue/queue.go:124 Finish processing item: queue id[service-poll-queue], key[bob/secretflow-task-20241008170216-single-psi-0-fed] (26.563µs) 2024-10-08 17:09:09.390 INFO queue/queue.go:124 Finish processing item: queue id[kuscia-coredns-controller], key[bob/secretflow-task-20241008170216-single-psi-0-global] (13.772µs) 2024-10-08 17:09:09.390 INFO queue/queue.go:124 Finish processing item: queue id[kuscia-coredns-controller], key[bob/secretflow-task-20241008170216-single-psi-0-spu] (15.094µs) 2024-10-08 17:09:14.349 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:3650 2024-10-08 17:09:14.349 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (25.181µs) 2024-10-08 17:09:14.355 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (6.234169ms) 2024-10-08 17:09:29.313 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (34.45µs) 2024-10-08 17:09:29.313 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:3685 2024-10-08 17:09:29.321 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (8.387993ms) 2024-10-08 17:09:44.306 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (19.141µs) 2024-10-08 17:09:44.307 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:3722 2024-10-08 17:09:44.313 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (6.902598ms) 2024-10-08 17:09:59.334 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (26.302µs) 2024-10-08 17:09:59.334 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:3758 2024-10-08 17:09:59.342 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (7.817724ms) 2024-10-08 17:10:14.317 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:3794 2024-10-08 17:10:14.322 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (4.256677ms) 2024-10-08 17:10:14.322 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (11.884µs) 2024-10-08 17:10:29.346 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (14.809µs) 2024-10-08 17:10:29.346 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:3831 2024-10-08 17:10:29.353 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (6.786614ms) 2024-10-08 17:10:44.347 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (30.736µs) 2024-10-08 17:10:44.347 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:3867 2024-10-08 17:10:44.354 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (6.719863ms) 2024-10-08 17:10:59.310 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (31.605µs) 2024-10-08 17:10:59.311 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:3903 2024-10-08 17:10:59.319 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (7.944736ms) 2024-10-08 17:11:14.361 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (12.158µs) 2024-10-08 17:11:14.361 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:3940 2024-10-08 17:11:14.368 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (6.62541ms) 2024-10-08 17:11:29.359 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (26.198µs) 2024-10-08 17:11:29.359 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:3976 2024-10-08 17:11:29.364 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (4.854876ms) 2024-10-08 17:11:44.378 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (33.137µs) 2024-10-08 17:11:44.379 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:4012 2024-10-08 17:11:44.384 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (5.474334ms) 2024-10-08 17:11:59.307 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (27.319µs) 2024-10-08 17:11:59.307 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:4049 2024-10-08 17:11:59.312 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (4.590926ms) 2024-10-08 17:12:14.310 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (31.184µs) 2024-10-08 17:12:14.311 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:4083 2024-10-08 17:12:14.316 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (5.477963ms) 2024-10-08 17:12:29.335 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:4118 2024-10-08 17:12:29.342 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (6.799606ms) 2024-10-08 17:12:29.342 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (24.313µs) 2024-10-08 17:12:44.327 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (27.888µs) 2024-10-08 17:12:44.327 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:4153 2024-10-08 17:12:44.332 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (5.217209ms) 2024-10-08 17:12:59.297 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (11.719µs) 2024-10-08 17:12:59.297 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:4204 2024-10-08 17:12:59.304 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (6.603989ms) 2024-10-08 17:13:14.371 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (23.099µs) 2024-10-08 17:13:14.371 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:4239 2024-10-08 17:13:14.379 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (8.114486ms) 2024-10-08 17:13:29.322 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (27.154µs) 2024-10-08 17:13:29.322 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:4275 2024-10-08 17:13:29.328 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (6.463879ms) 2024-10-08 17:13:44.352 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (26.965µs) 2024-10-08 17:13:44.353 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:4311 2024-10-08 17:13:44.360 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (6.986924ms) 2024-10-08 17:13:59.322 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (29.598µs) 2024-10-08 17:13:59.322 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:4348 2024-10-08 17:13:59.328 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (6.152645ms) 2024-10-08 17:14:14.323 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (33.162µs) 2024-10-08 17:14:14.323 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:4383 2024-10-08 17:14:14.334 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (11.088246ms) 2024-10-08 17:14:29.281 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (28.325µs) 2024-10-08 17:14:29.282 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:4419 2024-10-08 17:14:29.288 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (6.148793ms) 2024-10-08 17:14:44.317 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:4456 2024-10-08 17:14:44.317 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (23.094µs) 2024-10-08 17:14:44.325 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (7.67478ms) 2024-10-08 17:14:59.326 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (32.738µs) 2024-10-08 17:14:59.326 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:4493 2024-10-08 17:14:59.344 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (17.258845ms) 2024-10-08 17:15:14.333 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (13.26µs) 2024-10-08 17:15:14.333 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:4528 2024-10-08 17:15:14.341 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (8.060261ms) 2024-10-08 17:15:29.335 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (27.12µs) 2024-10-08 17:15:29.335 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:4566 2024-10-08 17:15:29.342 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (6.727511ms) 2024-10-08 17:15:44.337 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (28.267µs) 2024-10-08 17:15:44.337 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:4598 2024-10-08 17:15:44.344 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (6.904374ms) 2024-10-08 17:15:59.266 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (32.82µs) 2024-10-08 17:15:59.266 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:4632 2024-10-08 17:15:59.273 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (6.682632ms) 2024-10-08 17:16:14.370 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (20.854µs) 2024-10-08 17:16:14.370 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:4669 2024-10-08 17:16:14.376 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (5.905357ms) 2024-10-08 17:16:29.302 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (36.574µs) 2024-10-08 17:16:29.302 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:4706 2024-10-08 17:16:29.308 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (5.785543ms) 2024-10-08 17:16:44.335 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (30.645µs) 2024-10-08 17:16:44.335 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:4741 2024-10-08 17:16:44.341 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (5.391696ms) 2024-10-08 17:16:59.314 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (29.734µs) 2024-10-08 17:16:59.314 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:4779 2024-10-08 17:16:59.322 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (7.509758ms) 2024-10-08 17:17:14.342 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (33.878µs) 2024-10-08 17:17:14.342 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:4814 2024-10-08 17:17:14.347 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (5.209823ms) 2024-10-08 17:17:29.320 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (31.583µs) 2024-10-08 17:17:29.320 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:4851 2024-10-08 17:17:29.327 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (6.752848ms) 2024-10-08 17:17:44.291 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:4885 2024-10-08 17:17:44.298 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (32.899µs) 2024-10-08 17:17:44.301 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (10.0783ms) 2024-10-08 17:17:59.354 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (33.005µs) 2024-10-08 17:17:59.355 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:4922 2024-10-08 17:17:59.363 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (8.093024ms) 2024-10-08 17:18:14.322 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (24.364µs) 2024-10-08 17:18:14.322 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:4957 2024-10-08 17:18:14.329 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (7.432512ms) 2024-10-08 17:18:29.351 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (30.934µs) 2024-10-08 17:18:29.351 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:4995 2024-10-08 17:18:29.358 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (7.390049ms) 2024-10-08 17:18:44.381 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (28.495µs) 2024-10-08 17:18:44.382 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:5030 2024-10-08 17:18:44.398 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (16.931584ms) 2024-10-08 17:18:59.350 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (31.633µs) 2024-10-08 17:18:59.350 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:5066 2024-10-08 17:18:59.357 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (6.498789ms) 2024-10-08 17:19:09.321 INFO queue/queue.go:124 Finish processing item: queue id[kuscia-coredns-controller], key[bob/secretflow-task-20241008170216-single-psi-0-spu] (38.897µs) 2024-10-08 17:19:09.321 INFO queue/queue.go:124 Finish processing item: queue id[kuscia-coredns-controller], key[bob/secretflow-task-20241008170216-single-psi-0-fed] (16.219µs) 2024-10-08 17:19:09.321 INFO queue/queue.go:124 Finish processing item: queue id[kuscia-coredns-controller], key[bob/secretflow-task-20241008170216-single-psi-0-global] (14.292µs) 2024-10-08 17:19:09.341 INFO controller/endpoints.go:189 Updating endpoint bob/secretflow-task-20241008170216-single-psi-0-spu/2643 2024-10-08 17:19:09.341 INFO controller/endpoints.go:189 Updating endpoint bob/secretflow-task-20241008170216-single-psi-0-fed/2646 2024-10-08 17:19:09.341 INFO queue/queue.go:124 Finish processing item: queue id[endpoints-queue], key[bob/secretflow-task-20241008170216-single-psi-0-spu] (44.408µs) 2024-10-08 17:19:09.341 INFO queue/queue.go:124 Finish processing item: queue id[endpoints-queue], key[bob/secretflow-task-20241008170216-single-psi-0-fed] (15.156µs) 2024-10-08 17:19:09.341 INFO controller/endpoints.go:189 Updating endpoint bob/secretflow-task-20241008170216-single-psi-0-global/2648 2024-10-08 17:19:09.341 INFO queue/queue.go:124 Finish processing item: queue id[endpoints-queue], key[bob/secretflow-task-20241008170216-single-psi-0-global] (12.594µs) 2024-10-08 17:19:09.361 INFO controller/domain_route.go:357 Update DomainRoute bob/bob-kuscia-system revision:519 2024-10-08 17:19:09.368 INFO xds/xds.go:439 Add cluster:service-masterproxy 2024-10-08 17:19:09.368 INFO controller/domain_route.go:497 Update rule to master success 2024-10-08 17:19:09.368 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/bob-kuscia-system] (7.305397ms) 2024-10-08 17:19:09.368 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:5066 2024-10-08 17:19:09.371 INFO controller/domain_route.go:432 add cluster bob-to-alice name:http protocol:HTTP port:1080 2024-10-08 17:19:09.372 INFO xds/xds.go:439 Add cluster:bob-to-alice-http 2024-10-08 17:19:09.372 INFO controller/domain_route.go:357 Update DomainRoute bob/bob-alice revision:595 2024-10-08 17:19:09.375 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (6.885558ms) 2024-10-08 17:19:09.377 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (342.132µs) 2024-10-08 17:19:09.379 INFO xds/xds.go:439 Add cluster:bob-to-alice-http 2024-10-08 17:19:09.379 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/bob-alice] (7.483251ms) 2024-10-08 17:19:09.403 INFO queue/queue.go:124 Finish processing item: queue id[service-poll-queue], key[bob/secretflow-task-20241008170216-single-psi-0-spu] (45.983µs) 2024-10-08 17:19:09.404 INFO queue/queue.go:124 Finish processing item: queue id[service-poll-queue], key[bob/secretflow-task-20241008170216-single-psi-0-fed] (19.395µs) 2024-10-08 17:19:09.404 INFO queue/queue.go:124 Finish processing item: queue id[kuscia-coredns-controller], key[bob/secretflow-task-20241008170216-single-psi-0-spu] (84.26µs) 2024-10-08 17:19:09.404 INFO queue/queue.go:124 Finish processing item: queue id[kuscia-coredns-controller], key[bob/secretflow-task-20241008170216-single-psi-0-fed] (12.287µs) 2024-10-08 17:19:09.404 INFO queue/queue.go:124 Finish processing item: queue id[kuscia-coredns-controller], key[bob/secretflow-task-20241008170216-single-psi-0-global] (12.526µs) 2024-10-08 17:19:14.346 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:5102 2024-10-08 17:19:14.353 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (32.902µs) 2024-10-08 17:19:14.354 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (7.904231ms) 2024-10-08 17:19:29.338 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (24.217µs) 2024-10-08 17:19:29.339 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:5138 2024-10-08 17:19:29.356 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (17.082397ms) 2024-10-08 17:19:44.343 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (26.278µs) 2024-10-08 17:19:44.344 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:5172 2024-10-08 17:19:44.351 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (6.89681ms) 2024-10-08 17:19:59.333 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (30.384µs) 2024-10-08 17:19:59.334 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:5209 2024-10-08 17:19:59.340 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (6.536285ms) 2024-10-08 17:20:14.338 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (30.368µs) 2024-10-08 17:20:14.338 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:5244 2024-10-08 17:20:14.345 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (7.361038ms) 2024-10-08 17:20:29.433 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (30.218µs) 2024-10-08 17:20:29.434 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:5280 2024-10-08 17:20:29.476 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (42.153953ms) 2024-10-08 17:20:44.330 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (34.881µs) 2024-10-08 17:20:44.330 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:5315 2024-10-08 17:20:44.337 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (6.740174ms) 2024-10-08 17:20:59.415 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:5354 2024-10-08 17:20:59.421 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (47.055µs) 2024-10-08 17:20:59.422 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (7.112209ms) 2024-10-08 17:21:14.349 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (26.731µs) 2024-10-08 17:21:14.349 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:5387 2024-10-08 17:21:14.356 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (7.013762ms) 2024-10-08 17:21:29.286 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:5425 2024-10-08 17:21:29.294 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (33.284µs) 2024-10-08 17:21:29.296 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (10.006289ms) 2024-10-08 17:21:44.350 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:5460 2024-10-08 17:21:44.353 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (32.005µs) 2024-10-08 17:21:44.362 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (12.146167ms) 2024-10-08 17:21:59.343 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (32.585µs) 2024-10-08 17:21:59.343 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:5498 2024-10-08 17:21:59.350 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (7.703838ms) 2024-10-08 17:22:14.406 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (29.795µs) 2024-10-08 17:22:14.406 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:5531 2024-10-08 17:22:14.413 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (7.198181ms) 2024-10-08 17:22:29.349 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (28.179µs) 2024-10-08 17:22:29.349 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:5568 2024-10-08 17:22:29.359 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (9.741396ms) 2024-10-08 17:22:44.371 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (26.411µs) 2024-10-08 17:22:44.371 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:5601 2024-10-08 17:22:44.378 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (6.930448ms) 2024-10-08 17:22:59.286 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (15.818µs) 2024-10-08 17:22:59.286 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:5639 2024-10-08 17:22:59.292 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (6.431632ms) 2024-10-08 17:23:14.307 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (27.312µs) 2024-10-08 17:23:14.307 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:5674 2024-10-08 17:23:14.311 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (4.227457ms) 2024-10-08 17:23:29.326 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (35.696µs) 2024-10-08 17:23:29.326 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:5712 2024-10-08 17:23:29.334 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (7.6254ms) 2024-10-08 17:23:44.347 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (27.196µs) 2024-10-08 17:23:44.347 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:5747 2024-10-08 17:23:44.355 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (7.805113ms) 2024-10-08 17:23:59.346 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (18.127µs) 2024-10-08 17:23:59.346 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:5785 2024-10-08 17:23:59.350 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (4.3312ms) 2024-10-08 17:24:14.342 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (30.532µs) 2024-10-08 17:24:14.342 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:5819 2024-10-08 17:24:14.350 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (7.380306ms) 2024-10-08 17:24:29.329 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (27.637µs) 2024-10-08 17:24:29.329 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:5857 2024-10-08 17:24:29.336 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (6.54266ms) 2024-10-08 17:24:44.374 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (26.719µs) 2024-10-08 17:24:44.374 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:5892 2024-10-08 17:24:44.381 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (6.957996ms) 2024-10-08 17:24:59.283 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (28.419µs) 2024-10-08 17:24:59.283 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:5928 2024-10-08 17:24:59.291 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (7.861471ms) 2024-10-08 17:25:14.367 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:5964 2024-10-08 17:25:14.373 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (40.354µs) 2024-10-08 17:25:14.374 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (7.592107ms) 2024-10-08 17:25:29.362 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (32.544µs) 2024-10-08 17:25:29.362 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:6002 2024-10-08 17:25:29.370 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (7.844436ms) 2024-10-08 17:25:44.346 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (31.198µs) 2024-10-08 17:25:44.347 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:6034 2024-10-08 17:25:44.355 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (7.999263ms) 2024-10-08 17:25:59.342 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:6072 2024-10-08 17:25:59.345 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (24.21µs) 2024-10-08 17:25:59.347 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (4.885663ms) 2024-10-08 17:26:14.354 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (26.12µs) 2024-10-08 17:26:14.354 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:6105 2024-10-08 17:26:14.361 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (6.930727ms) 2024-10-08 17:26:29.362 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (28.497µs) 2024-10-08 17:26:29.362 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:6144 2024-10-08 17:26:29.371 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (9.125106ms) 2024-10-08 17:26:44.354 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (26.645µs) 2024-10-08 17:26:44.354 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:6179 2024-10-08 17:26:44.362 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (7.652251ms) 2024-10-08 17:26:59.367 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (31.826µs) 2024-10-08 17:26:59.368 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:6217 2024-10-08 17:26:59.384 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (16.1366ms) 2024-10-08 17:27:14.265 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (25.696µs) 2024-10-08 17:27:14.265 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:6248 2024-10-08 17:27:14.272 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (6.99459ms) 2024-10-08 17:27:29.342 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (36.307µs) 2024-10-08 17:27:29.342 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:6290 2024-10-08 17:27:29.350 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (8.103431ms) 2024-10-08 17:27:44.322 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (26.08µs) 2024-10-08 17:27:44.322 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:6322 2024-10-08 17:27:44.329 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (7.483982ms) 2024-10-08 17:27:59.363 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (26.498µs) 2024-10-08 17:27:59.363 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:6362 2024-10-08 17:27:59.378 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (14.905547ms) 2024-10-08 17:28:14.362 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:6396 2024-10-08 17:28:14.369 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (7.201187ms) 2024-10-08 17:28:14.369 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (26.87µs) 2024-10-08 17:28:29.370 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (31.586µs) 2024-10-08 17:28:29.370 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:6435 2024-10-08 17:28:29.378 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (7.405844ms) 2024-10-08 17:28:44.354 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (31.41µs) 2024-10-08 17:28:44.354 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:6467 2024-10-08 17:28:44.361 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (6.878325ms) 2024-10-08 17:28:59.333 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (28.699µs) 2024-10-08 17:28:59.333 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:6506 2024-10-08 17:28:59.339 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (5.773468ms) 2024-10-08 17:29:09.325 INFO queue/queue.go:124 Finish processing item: queue id[kuscia-coredns-controller], key[bob/secretflow-task-20241008170216-single-psi-0-global] (40.74µs) 2024-10-08 17:29:09.325 INFO queue/queue.go:124 Finish processing item: queue id[kuscia-coredns-controller], key[bob/secretflow-task-20241008170216-single-psi-0-spu] (33.705µs) 2024-10-08 17:29:09.325 INFO queue/queue.go:124 Finish processing item: queue id[kuscia-coredns-controller], key[bob/secretflow-task-20241008170216-single-psi-0-fed] (14.356µs) 2024-10-08 17:29:09.345 INFO controller/endpoints.go:189 Updating endpoint bob/secretflow-task-20241008170216-single-psi-0-spu/2643 2024-10-08 17:29:09.345 INFO controller/endpoints.go:189 Updating endpoint bob/secretflow-task-20241008170216-single-psi-0-fed/2646 2024-10-08 17:29:09.345 INFO queue/queue.go:124 Finish processing item: queue id[endpoints-queue], key[bob/secretflow-task-20241008170216-single-psi-0-spu] (47.191µs) 2024-10-08 17:29:09.345 INFO queue/queue.go:124 Finish processing item: queue id[endpoints-queue], key[bob/secretflow-task-20241008170216-single-psi-0-fed] (15.561µs) 2024-10-08 17:29:09.345 INFO controller/endpoints.go:189 Updating endpoint bob/secretflow-task-20241008170216-single-psi-0-global/2648 2024-10-08 17:29:09.346 INFO queue/queue.go:124 Finish processing item: queue id[endpoints-queue], key[bob/secretflow-task-20241008170216-single-psi-0-global] (12.329µs) 2024-10-08 17:29:09.365 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (55.476µs) 2024-10-08 17:29:09.366 INFO controller/domain_route.go:357 Update DomainRoute bob/bob-kuscia-system revision:519 2024-10-08 17:29:09.378 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:6506 2024-10-08 17:29:09.381 INFO controller/domain_route.go:432 add cluster bob-to-alice name:http protocol:HTTP port:1080 2024-10-08 17:29:09.385 INFO xds/xds.go:439 Add cluster:service-masterproxy 2024-10-08 17:29:09.385 INFO controller/domain_route.go:497 Update rule to master success 2024-10-08 17:29:09.385 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/bob-kuscia-system] (19.555774ms) 2024-10-08 17:29:09.386 INFO xds/xds.go:439 Add cluster:bob-to-alice-http 2024-10-08 17:29:09.386 INFO controller/domain_route.go:357 Update DomainRoute bob/bob-alice revision:595 2024-10-08 17:29:09.394 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (15.841374ms) 2024-10-08 17:29:09.398 INFO xds/xds.go:439 Add cluster:bob-to-alice-http 2024-10-08 17:29:09.398 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/bob-alice] (16.88963ms) 2024-10-08 17:29:09.403 INFO queue/queue.go:124 Finish processing item: queue id[service-poll-queue], key[bob/secretflow-task-20241008170216-single-psi-0-spu] (40.892µs) 2024-10-08 17:29:09.404 INFO queue/queue.go:124 Finish processing item: queue id[kuscia-coredns-controller], key[bob/secretflow-task-20241008170216-single-psi-0-spu] (18.025µs) 2024-10-08 17:29:09.404 INFO queue/queue.go:124 Finish processing item: queue id[kuscia-coredns-controller], key[bob/secretflow-task-20241008170216-single-psi-0-fed] (11.176µs) 2024-10-08 17:29:09.404 INFO queue/queue.go:124 Finish processing item: queue id[kuscia-coredns-controller], key[bob/secretflow-task-20241008170216-single-psi-0-global] (11.962µs) 2024-10-08 17:29:09.404 INFO queue/queue.go:124 Finish processing item: queue id[service-poll-queue], key[bob/secretflow-task-20241008170216-single-psi-0-fed] (16.134µs) 2024-10-08 17:29:14.366 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (33.679µs) 2024-10-08 17:29:14.366 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:6539 2024-10-08 17:29:14.379 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (12.517149ms) 2024-10-08 17:29:29.348 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (29.521µs) 2024-10-08 17:29:29.349 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:6578 2024-10-08 17:29:29.356 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (6.805944ms) 2024-10-08 17:29:44.321 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (25.74µs) 2024-10-08 17:29:44.321 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:6612 2024-10-08 17:29:44.328 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (7.304229ms) 2024-10-08 17:29:59.351 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (26.516µs) 2024-10-08 17:29:59.351 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:6651 2024-10-08 17:29:59.359 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (7.366195ms) 2024-10-08 17:30:14.323 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (29.737µs) 2024-10-08 17:30:14.324 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:6684 2024-10-08 17:30:14.333 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (8.867023ms) 2024-10-08 17:30:29.354 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (28.211µs) 2024-10-08 17:30:29.354 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:6724 2024-10-08 17:30:29.362 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (7.958769ms) 2024-10-08 17:30:44.364 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (34.836µs) 2024-10-08 17:30:44.364 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:6757 2024-10-08 17:30:44.373 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (8.721285ms) 2024-10-08 17:30:59.258 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (26.392µs) 2024-10-08 17:30:59.259 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:6793 2024-10-08 17:30:59.268 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (9.167602ms) 2024-10-08 17:31:14.282 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:6828 2024-10-08 17:31:14.285 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (38.21µs) 2024-10-08 17:31:14.289 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (6.993209ms) 2024-10-08 17:31:29.314 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (33.321µs) 2024-10-08 17:31:29.314 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:6868 2024-10-08 17:31:29.321 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (6.863974ms) 2024-10-08 17:31:44.331 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (24.798µs) 2024-10-08 17:31:44.331 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:6901 2024-10-08 17:31:44.341 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (9.197705ms) 2024-10-08 17:31:59.320 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (25.866µs) 2024-10-08 17:31:59.320 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:6940 2024-10-08 17:31:59.327 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (7.085151ms) 2024-10-08 17:32:14.370 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:6974 2024-10-08 17:32:14.371 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (32.002µs) 2024-10-08 17:32:14.377 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (7.402479ms) 2024-10-08 17:32:29.370 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (28.892µs) 2024-10-08 17:32:29.370 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:7013 2024-10-08 17:32:29.377 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (7.214295ms) 2024-10-08 17:32:44.383 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (29.238µs) 2024-10-08 17:32:44.383 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:7046 2024-10-08 17:32:44.390 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (6.864813ms) 2024-10-08 17:32:59.318 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (29.569µs) 2024-10-08 17:32:59.318 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:7084 2024-10-08 17:32:59.325 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (6.773517ms) 2024-10-08 17:33:14.343 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (33.209µs) 2024-10-08 17:33:14.343 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:7118 2024-10-08 17:33:14.353 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (10.196433ms) 2024-10-08 17:33:29.242 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (26.35µs) 2024-10-08 17:33:29.242 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:7155 2024-10-08 17:33:29.249 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (6.719617ms) 2024-10-08 17:33:44.338 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (22.194µs) 2024-10-08 17:33:44.338 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:7191 2024-10-08 17:33:44.340 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (2.493463ms) 2024-10-08 17:33:59.264 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (27.21µs) 2024-10-08 17:33:59.264 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:7225 2024-10-08 17:33:59.270 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (6.519889ms) 2024-10-08 17:34:14.358 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (23.433µs) 2024-10-08 17:34:14.358 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:7262 2024-10-08 17:34:14.365 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (6.901918ms) 2024-10-08 17:34:29.266 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (29.305µs) 2024-10-08 17:34:29.266 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:7296 2024-10-08 17:34:29.275 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (9.123964ms) 2024-10-08 17:34:44.346 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (33.283µs) 2024-10-08 17:34:44.346 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:7332 2024-10-08 17:34:44.354 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (7.558671ms) 2024-10-08 17:34:59.943 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:7371 2024-10-08 17:34:59.928 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (128.425µs) 2024-10-08 17:35:00.035 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (91.993805ms) 2024-10-08 17:35:14.391 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (28.825µs) 2024-10-08 17:35:14.391 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:7398 2024-10-08 17:35:14.421 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (29.871117ms) 2024-10-08 17:35:29.591 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:7439 2024-10-08 17:35:29.593 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (33.997µs) 2024-10-08 17:35:29.623 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (31.962883ms) 2024-10-08 17:40:58.217 INFO modules/modules.go:337 Start to init all secret backends ... 2024-10-08 17:40:58.220 WARN modules/modules.go:344 Init all secret backend but no provider found, creating default mem type 2024-10-08 17:40:58.220 INFO modules/modules.go:349 Finish Initializing all secret backends 2024-10-08 17:40:58.228 INFO modules/modules.go:115 Add destroyFunc [coredns] 2024-10-08 17:40:58.229 INFO modules/coredns.go:195 Start preparing coredns resolv.conf, root dir /home/kuscia/ 2024-10-08 17:40:58.230 INFO modules/coredns.go:218 Finish preparing coredns resolv.conf 2024-10-08 17:40:58.233 INFO modules/coredns.go:186 CoreDNS is ready 2024-10-08 17:40:58.235 INFO modules/modules.go:115 Add destroyFunc [containerd] 2024-10-08 17:40:58.285 INFO supervisor/supervisor.go:75 [containerd] start and watch subprocess 2024-10-08 17:40:58.286 INFO supervisor/supervisor.go:101 [containerd] try to start new process 2024-10-08 17:40:58.291 INFO process/process.go:76 Set pid[21] oom score adj to -700 2024-10-08 17:40:59.732 ERROR modules/containerd.go:117 Unable to import pause image: failed to run command "/home/kuscia/bin/ctr -a=/home/kuscia/containerd/run/containerd.sock -n=k8s.io images import /home/kuscia/pause/pause.tar", detail-> exit status 1 2024-10-08 17:41:01.201 INFO modules/containerd.go:180 Containerd is ready 2024-10-08 17:41:01.202 INFO modules/modules.go:115 Add destroyFunc [transport] 2024-10-08 17:41:01.202 INFO modules/modules.go:115 Add destroyFunc [envoy] 2024-10-08 17:41:01.203 INFO modules/modules.go:115 Add destroyFunc [domainroute] 2024-10-08 17:41:01.203 INFO supervisor/supervisor.go:75 [envoy] start and watch subprocess 2024-10-08 17:41:01.204 INFO supervisor/supervisor.go:101 [envoy] try to start new process 2024-10-08 17:41:01.204 INFO xds/xds.go:160 Management server listening on 10001 2024-10-08 17:41:01.205 INFO process/process.go:76 Set pid[56] oom score adj to -700 2024-10-08 17:41:01.216 INFO commands/root.go:62 Start xds success 2024-10-08 17:41:01.221 ERROR modules/domainroute.go:152 [PROBE] failed to probe master endpoint https://root-kuscia-master:1080, detail-> sending request error: Get "https://root-kuscia-master:1080": dial tcp 172.20.0.4:1080: connect: connection refused 2024-10-08 17:41:01.222 FATAL modules/domainroute.go:157 DomainRoute wait ready failed: context canceled 2024-10-08 17:41:07.062 INFO modules/modules.go:337 Start to init all secret backends ... 2024-10-08 17:41:07.104 WARN modules/modules.go:344 Init all secret backend but no provider found, creating default mem type 2024-10-08 17:41:07.104 INFO modules/modules.go:349 Finish Initializing all secret backends 2024-10-08 17:41:07.105 INFO modules/modules.go:115 Add destroyFunc [coredns] 2024-10-08 17:41:07.105 INFO modules/coredns.go:195 Start preparing coredns resolv.conf, root dir /home/kuscia/ 2024-10-08 17:41:07.106 INFO modules/coredns.go:218 Finish preparing coredns resolv.conf 2024-10-08 17:41:07.107 INFO modules/coredns.go:186 CoreDNS is ready 2024-10-08 17:41:07.108 INFO modules/modules.go:115 Add destroyFunc [containerd] 2024-10-08 17:41:07.181 INFO supervisor/supervisor.go:75 [containerd] start and watch subprocess 2024-10-08 17:41:07.182 INFO supervisor/supervisor.go:101 [containerd] try to start new process 2024-10-08 17:41:07.202 INFO process/process.go:76 Set pid[20] oom score adj to -700 2024-10-08 17:41:08.882 INFO modules/containerd.go:180 Containerd is ready 2024-10-08 17:41:08.882 INFO modules/modules.go:115 Add destroyFunc [transport] 2024-10-08 17:41:08.883 INFO modules/modules.go:115 Add destroyFunc [envoy] 2024-10-08 17:41:08.884 INFO modules/modules.go:115 Add destroyFunc [domainroute] 2024-10-08 17:41:08.886 INFO xds/xds.go:160 Management server listening on 10001 2024-10-08 17:41:08.894 INFO supervisor/supervisor.go:75 [envoy] start and watch subprocess 2024-10-08 17:41:08.894 INFO supervisor/supervisor.go:101 [envoy] try to start new process 2024-10-08 17:41:08.897 INFO process/process.go:76 Set pid[44] oom score adj to -700 2024-10-08 17:41:08.902 INFO commands/root.go:62 Start xds success 2024-10-08 17:41:08.910 ERROR modules/domainroute.go:152 [PROBE] failed to probe master endpoint https://root-kuscia-master:1080, detail-> sending request error: Get "https://root-kuscia-master:1080": dial tcp 172.20.0.4:1080: connect: connection refused 2024-10-08 17:41:08.910 FATAL modules/domainroute.go:157 DomainRoute wait ready failed: context canceled 2024-10-08 17:41:13.386 INFO modules/modules.go:337 Start to init all secret backends ... 2024-10-08 17:41:13.387 WARN modules/modules.go:344 Init all secret backend but no provider found, creating default mem type 2024-10-08 17:41:13.387 INFO modules/modules.go:349 Finish Initializing all secret backends 2024-10-08 17:41:13.388 INFO modules/modules.go:115 Add destroyFunc [coredns] 2024-10-08 17:41:13.389 INFO modules/coredns.go:195 Start preparing coredns resolv.conf, root dir /home/kuscia/ 2024-10-08 17:41:13.390 INFO modules/coredns.go:218 Finish preparing coredns resolv.conf 2024-10-08 17:41:13.392 INFO modules/coredns.go:186 CoreDNS is ready 2024-10-08 17:41:13.393 INFO modules/modules.go:115 Add destroyFunc [containerd] 2024-10-08 17:41:13.466 INFO supervisor/supervisor.go:75 [containerd] start and watch subprocess 2024-10-08 17:41:13.466 INFO supervisor/supervisor.go:101 [containerd] try to start new process 2024-10-08 17:41:13.478 INFO process/process.go:76 Set pid[18] oom score adj to -700 2024-10-08 17:41:15.019 INFO modules/containerd.go:180 Containerd is ready 2024-10-08 17:41:15.019 INFO modules/modules.go:115 Add destroyFunc [transport] 2024-10-08 17:41:15.020 INFO modules/modules.go:115 Add destroyFunc [envoy] 2024-10-08 17:41:15.021 INFO modules/modules.go:115 Add destroyFunc [domainroute] 2024-10-08 17:41:15.022 INFO supervisor/supervisor.go:75 [envoy] start and watch subprocess 2024-10-08 17:41:15.022 INFO supervisor/supervisor.go:101 [envoy] try to start new process 2024-10-08 17:41:15.023 INFO xds/xds.go:160 Management server listening on 10001 2024-10-08 17:41:15.024 INFO process/process.go:76 Set pid[45] oom score adj to -700 2024-10-08 17:41:15.037 INFO commands/root.go:62 Start xds success 2024-10-08 17:41:15.042 ERROR modules/domainroute.go:152 [PROBE] failed to probe master endpoint https://root-kuscia-master:1080, detail-> sending request error: Get "https://root-kuscia-master:1080": dial tcp 172.20.0.4:1080: connect: connection refused 2024-10-08 17:41:15.042 FATAL modules/domainroute.go:157 DomainRoute wait ready failed: context canceled 2024-10-08 17:41:18.183 INFO modules/modules.go:337 Start to init all secret backends ... 2024-10-08 17:41:18.184 WARN modules/modules.go:344 Init all secret backend but no provider found, creating default mem type 2024-10-08 17:41:18.184 INFO modules/modules.go:349 Finish Initializing all secret backends 2024-10-08 17:41:18.185 INFO modules/modules.go:115 Add destroyFunc [coredns] 2024-10-08 17:41:18.186 INFO modules/coredns.go:195 Start preparing coredns resolv.conf, root dir /home/kuscia/ 2024-10-08 17:41:18.187 INFO modules/coredns.go:218 Finish preparing coredns resolv.conf 2024-10-08 17:41:18.189 INFO modules/coredns.go:186 CoreDNS is ready 2024-10-08 17:41:18.189 INFO modules/modules.go:115 Add destroyFunc [containerd] 2024-10-08 17:41:18.218 INFO supervisor/supervisor.go:75 [containerd] start and watch subprocess 2024-10-08 17:41:18.219 INFO supervisor/supervisor.go:101 [containerd] try to start new process 2024-10-08 17:41:18.220 INFO process/process.go:76 Set pid[20] oom score adj to -700 2024-10-08 17:41:19.813 INFO modules/containerd.go:180 Containerd is ready 2024-10-08 17:41:19.813 INFO modules/modules.go:115 Add destroyFunc [transport] 2024-10-08 17:41:19.814 INFO modules/modules.go:115 Add destroyFunc [envoy] 2024-10-08 17:41:19.817 INFO supervisor/supervisor.go:75 [envoy] start and watch subprocess 2024-10-08 17:41:19.817 INFO supervisor/supervisor.go:101 [envoy] try to start new process 2024-10-08 17:41:19.818 INFO modules/modules.go:115 Add destroyFunc [domainroute] 2024-10-08 17:41:19.820 INFO xds/xds.go:160 Management server listening on 10001 2024-10-08 17:41:19.820 INFO process/process.go:76 Set pid[44] oom score adj to -700 2024-10-08 17:41:19.833 INFO commands/root.go:62 Start xds success 2024-10-08 17:41:19.839 ERROR modules/domainroute.go:152 [PROBE] failed to probe master endpoint https://root-kuscia-master:1080, detail-> sending request error: Get "https://root-kuscia-master:1080": dial tcp 172.20.0.4:1080: connect: connection refused 2024-10-08 17:41:19.839 FATAL modules/domainroute.go:157 DomainRoute wait ready failed: context canceled 2024-10-08 17:46:38.145 INFO modules/modules.go:337 Start to init all secret backends ... 2024-10-08 17:46:38.150 WARN modules/modules.go:344 Init all secret backend but no provider found, creating default mem type 2024-10-08 17:46:38.151 INFO modules/modules.go:349 Finish Initializing all secret backends 2024-10-08 17:46:38.154 INFO modules/modules.go:115 Add destroyFunc [coredns] 2024-10-08 17:46:38.155 INFO modules/coredns.go:195 Start preparing coredns resolv.conf, root dir /home/kuscia/ 2024-10-08 17:46:38.156 INFO modules/coredns.go:218 Finish preparing coredns resolv.conf 2024-10-08 17:46:38.160 INFO modules/coredns.go:186 CoreDNS is ready 2024-10-08 17:46:38.161 INFO modules/modules.go:115 Add destroyFunc [containerd] 2024-10-08 17:46:38.541 INFO supervisor/supervisor.go:75 [containerd] start and watch subprocess 2024-10-08 17:46:38.541 INFO supervisor/supervisor.go:101 [containerd] try to start new process 2024-10-08 17:46:38.552 INFO process/process.go:76 Set pid[20] oom score adj to -700 2024-10-08 17:46:39.827 ERROR modules/containerd.go:117 Unable to import pause image: failed to run command "/home/kuscia/bin/ctr -a=/home/kuscia/containerd/run/containerd.sock -n=k8s.io images import /home/kuscia/pause/pause.tar", detail-> exit status 1 2024-10-08 17:46:41.384 INFO modules/containerd.go:180 Containerd is ready 2024-10-08 17:46:41.384 INFO modules/modules.go:115 Add destroyFunc [transport] 2024-10-08 17:46:41.385 INFO modules/modules.go:115 Add destroyFunc [envoy] 2024-10-08 17:46:41.385 INFO modules/modules.go:115 Add destroyFunc [domainroute] 2024-10-08 17:46:41.386 INFO xds/xds.go:160 Management server listening on 10001 2024-10-08 17:46:41.387 INFO supervisor/supervisor.go:75 [envoy] start and watch subprocess 2024-10-08 17:46:41.387 INFO supervisor/supervisor.go:101 [envoy] try to start new process 2024-10-08 17:46:41.388 INFO process/process.go:76 Set pid[58] oom score adj to -700 2024-10-08 17:46:41.401 INFO commands/root.go:62 Start xds success 2024-10-08 17:46:41.406 ERROR modules/domainroute.go:152 [PROBE] failed to probe master endpoint https://root-kuscia-master:1080, detail-> sending request error: Get "https://root-kuscia-master:1080": dial tcp 172.20.0.4:1080: connect: connection refused 2024-10-08 17:46:41.406 FATAL modules/domainroute.go:157 DomainRoute wait ready failed: context canceled 2024-10-08 17:46:46.126 INFO modules/modules.go:337 Start to init all secret backends ... 2024-10-08 17:46:46.126 WARN modules/modules.go:344 Init all secret backend but no provider found, creating default mem type 2024-10-08 17:46:46.126 INFO modules/modules.go:349 Finish Initializing all secret backends 2024-10-08 17:46:46.127 INFO modules/modules.go:115 Add destroyFunc [coredns] 2024-10-08 17:46:46.127 INFO modules/coredns.go:195 Start preparing coredns resolv.conf, root dir /home/kuscia/ 2024-10-08 17:46:46.128 INFO modules/coredns.go:218 Finish preparing coredns resolv.conf 2024-10-08 17:46:46.129 INFO modules/coredns.go:186 CoreDNS is ready 2024-10-08 17:46:46.129 INFO modules/modules.go:115 Add destroyFunc [containerd] 2024-10-08 17:46:46.161 INFO supervisor/supervisor.go:75 [containerd] start and watch subprocess 2024-10-08 17:46:46.161 INFO supervisor/supervisor.go:101 [containerd] try to start new process 2024-10-08 17:46:46.171 INFO process/process.go:76 Set pid[19] oom score adj to -700 2024-10-08 17:46:48.191 INFO modules/containerd.go:180 Containerd is ready 2024-10-08 17:46:48.191 INFO modules/modules.go:115 Add destroyFunc [transport] 2024-10-08 17:46:48.191 INFO modules/modules.go:115 Add destroyFunc [envoy] 2024-10-08 17:46:48.192 INFO modules/modules.go:115 Add destroyFunc [domainroute] 2024-10-08 17:46:48.192 INFO xds/xds.go:160 Management server listening on 10001 2024-10-08 17:46:48.193 INFO supervisor/supervisor.go:75 [envoy] start and watch subprocess 2024-10-08 17:46:48.193 INFO supervisor/supervisor.go:101 [envoy] try to start new process 2024-10-08 17:46:48.212 INFO process/process.go:76 Set pid[46] oom score adj to -700 2024-10-08 17:46:48.208 INFO commands/root.go:62 Start xds success 2024-10-08 17:46:48.218 ERROR modules/domainroute.go:152 [PROBE] failed to probe master endpoint https://root-kuscia-master:1080, detail-> sending request error: Get "https://root-kuscia-master:1080": dial tcp 172.20.0.4:1080: connect: connection refused 2024-10-08 17:46:48.219 FATAL modules/domainroute.go:157 DomainRoute wait ready failed: context canceled 2024-10-08 17:46:51.305 INFO modules/modules.go:337 Start to init all secret backends ... 2024-10-08 17:46:51.306 WARN modules/modules.go:344 Init all secret backend but no provider found, creating default mem type 2024-10-08 17:46:51.306 INFO modules/modules.go:349 Finish Initializing all secret backends 2024-10-08 17:46:51.307 INFO modules/modules.go:115 Add destroyFunc [coredns] 2024-10-08 17:46:51.307 INFO modules/coredns.go:195 Start preparing coredns resolv.conf, root dir /home/kuscia/ 2024-10-08 17:46:51.308 INFO modules/coredns.go:218 Finish preparing coredns resolv.conf 2024-10-08 17:46:51.310 INFO modules/coredns.go:186 CoreDNS is ready 2024-10-08 17:46:51.311 INFO modules/modules.go:115 Add destroyFunc [containerd] 2024-10-08 17:46:51.384 INFO supervisor/supervisor.go:75 [containerd] start and watch subprocess 2024-10-08 17:46:51.384 INFO supervisor/supervisor.go:101 [containerd] try to start new process 2024-10-08 17:46:51.396 INFO process/process.go:76 Set pid[19] oom score adj to -700 2024-10-08 17:46:53.936 INFO modules/containerd.go:180 Containerd is ready 2024-10-08 17:46:53.936 INFO modules/modules.go:115 Add destroyFunc [transport] 2024-10-08 17:46:53.937 INFO modules/modules.go:115 Add destroyFunc [envoy] 2024-10-08 17:46:53.937 INFO modules/modules.go:115 Add destroyFunc [domainroute] 2024-10-08 17:46:53.938 INFO xds/xds.go:160 Management server listening on 10001 2024-10-08 17:46:53.938 INFO supervisor/supervisor.go:75 [envoy] start and watch subprocess 2024-10-08 17:46:53.938 INFO supervisor/supervisor.go:101 [envoy] try to start new process 2024-10-08 17:46:53.942 INFO process/process.go:76 Set pid[46] oom score adj to -700 2024-10-08 17:46:53.959 INFO commands/root.go:62 Start xds success 2024-10-08 17:46:53.970 ERROR modules/domainroute.go:152 [PROBE] failed to probe master endpoint https://root-kuscia-master:1080, detail-> sending request error: Get "https://root-kuscia-master:1080": dial tcp 172.20.0.4:1080: connect: connection refused 2024-10-08 17:46:53.970 FATAL modules/domainroute.go:157 DomainRoute wait ready failed: context canceled 2024-10-08 17:46:56.881 INFO modules/modules.go:337 Start to init all secret backends ... 2024-10-08 17:46:56.882 WARN modules/modules.go:344 Init all secret backend but no provider found, creating default mem type 2024-10-08 17:46:56.882 INFO modules/modules.go:349 Finish Initializing all secret backends 2024-10-08 17:46:56.882 INFO modules/modules.go:115 Add destroyFunc [coredns] 2024-10-08 17:46:56.882 INFO modules/coredns.go:195 Start preparing coredns resolv.conf, root dir /home/kuscia/ 2024-10-08 17:46:56.884 INFO modules/coredns.go:218 Finish preparing coredns resolv.conf 2024-10-08 17:46:56.885 INFO modules/coredns.go:186 CoreDNS is ready 2024-10-08 17:46:56.885 INFO modules/modules.go:115 Add destroyFunc [containerd] 2024-10-08 17:46:56.912 INFO supervisor/supervisor.go:75 [containerd] start and watch subprocess 2024-10-08 17:46:56.912 INFO supervisor/supervisor.go:101 [containerd] try to start new process 2024-10-08 17:46:56.920 INFO process/process.go:76 Set pid[21] oom score adj to -700 2024-10-08 17:46:58.386 INFO modules/containerd.go:180 Containerd is ready 2024-10-08 17:46:58.387 INFO modules/modules.go:115 Add destroyFunc [transport] 2024-10-08 17:46:58.387 INFO modules/modules.go:115 Add destroyFunc [envoy] 2024-10-08 17:46:58.387 INFO modules/modules.go:115 Add destroyFunc [domainroute] 2024-10-08 17:46:58.388 INFO supervisor/supervisor.go:75 [envoy] start and watch subprocess 2024-10-08 17:46:58.388 INFO supervisor/supervisor.go:101 [envoy] try to start new process 2024-10-08 17:46:58.389 INFO xds/xds.go:160 Management server listening on 10001 2024-10-08 17:46:58.389 INFO process/process.go:76 Set pid[45] oom score adj to -700 2024-10-08 17:46:58.396 INFO commands/root.go:62 Start xds success 2024-10-08 17:46:58.400 ERROR modules/domainroute.go:152 [PROBE] failed to probe master endpoint https://root-kuscia-master:1080, detail-> sending request error: Get "https://root-kuscia-master:1080": dial tcp 172.20.0.4:1080: connect: connection refused 2024-10-08 17:46:58.400 FATAL modules/domainroute.go:157 DomainRoute wait ready failed: context canceled 2024-10-08 17:47:00.393 INFO modules/modules.go:337 Start to init all secret backends ... 2024-10-08 17:47:00.393 WARN modules/modules.go:344 Init all secret backend but no provider found, creating default mem type 2024-10-08 17:47:00.393 INFO modules/modules.go:349 Finish Initializing all secret backends 2024-10-08 17:47:00.394 INFO modules/modules.go:115 Add destroyFunc [coredns] 2024-10-08 17:47:00.395 INFO modules/coredns.go:195 Start preparing coredns resolv.conf, root dir /home/kuscia/ 2024-10-08 17:47:00.396 INFO modules/coredns.go:218 Finish preparing coredns resolv.conf 2024-10-08 17:47:00.398 INFO modules/coredns.go:186 CoreDNS is ready 2024-10-08 17:47:00.399 INFO modules/modules.go:115 Add destroyFunc [containerd] 2024-10-08 17:47:00.451 INFO supervisor/supervisor.go:75 [containerd] start and watch subprocess 2024-10-08 17:47:00.451 INFO supervisor/supervisor.go:101 [containerd] try to start new process 2024-10-08 17:47:00.452 INFO process/process.go:76 Set pid[20] oom score adj to -700 2024-10-08 17:47:01.965 INFO modules/containerd.go:180 Containerd is ready 2024-10-08 17:47:01.965 INFO modules/modules.go:115 Add destroyFunc [transport] 2024-10-08 17:47:01.966 INFO modules/modules.go:115 Add destroyFunc [envoy] 2024-10-08 17:47:01.966 INFO modules/modules.go:115 Add destroyFunc [domainroute] 2024-10-08 17:47:01.968 INFO xds/xds.go:160 Management server listening on 10001 2024-10-08 17:47:01.969 INFO supervisor/supervisor.go:75 [envoy] start and watch subprocess 2024-10-08 17:47:01.969 INFO supervisor/supervisor.go:101 [envoy] try to start new process 2024-10-08 17:47:01.970 INFO process/process.go:76 Set pid[45] oom score adj to -700 2024-10-08 17:47:01.979 INFO commands/root.go:62 Start xds success 2024-10-08 17:47:01.997 INFO commands/root.go:86 [PROBE] success to probe master endpoint https://root-kuscia-master:1080 2024-10-08 17:47:01.997 INFO xds/cluster_config.go:131 Generate tls config for service-masterproxy 2024-10-08 17:47:01.998 INFO xds/cluster_config.go:275 disable keep-alive for cluster:service-masterproxy 2024-10-08 17:47:01.998 INFO xds/xds.go:439 Add cluster:service-masterproxy 2024-10-08 17:47:01.998 INFO clusters/master.go:61 add Master cluster:masterproxy 2024-10-08 17:47:02.068 ERROR modules/envoy.go:59 Get ready err:Get "http://127.0.0.1:10000/ready": dial tcp 127.0.0.1:10000: connect: connection refused 2024-10-08 17:47:02.284 INFO modules/envoy.go:226 Envoy is ready 2024-10-08 17:47:02.968 INFO modules/transport.go:144 transport is ready 2024-10-08 17:47:04.017 INFO clusters/master.go:278 Get master gateway namespace: kuscia-system 2024-10-08 17:47:14.056 WARN controller/handshake.go:749 send request error, detail -> Post "http://127.0.0.1:80/handshake": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 2024-10-08 17:47:17.097 INFO xds/cluster_config.go:275 enable keep-alive for cluster:service-masterproxy 2024-10-08 17:47:17.097 INFO xds/xds.go:439 Add cluster:service-masterproxy 2024-10-08 17:47:18.128 INFO commands/root.go:218 Check MasterProxy ready 2024-10-08 17:47:18.166 INFO commands/root.go:91 Add master proxy clusters successfully 2024-10-08 17:47:18.166 INFO xds/xds.go:439 Add cluster:service-transport 2024-10-08 17:47:18.167 INFO clusters/interconn.go:70 Add Transport Cluster success 2024-10-08 17:47:18.167 INFO commands/root.go:104 Add interconn clusters success 2024-10-08 17:47:18.169 INFO commands/root.go:170 Gateway running 2024-10-08 17:47:18.169 INFO modules/domainroute.go:159 DomainRoute is ready 2024-10-08 17:47:18.169 INFO modules/modules.go:115 Add destroyFunc [confmanager] 2024-10-08 17:47:18.170 INFO controller/domain_route.go:183 Starting DomainRoute controller 2024-10-08 17:47:18.170 INFO controller/domain_route.go:186 Waiting for informer caches to sync 2024-10-08 17:47:18.170 INFO controller/gateway.go:105 Starting Gateway controller 2024-10-08 17:47:18.170 INFO controller/endpoints.go:122 Waiting for informer caches to sync 2024-10-08 17:47:18.170 INFO nlog/nlog.go:77 I1008 17:47:18.170777 7 shared_informer.go:270] Waiting for caches to sync for endpoints 2024-10-08 17:47:18.171 INFO poller/poll_manager.go:127 Waiting for informer caches to sync 2024-10-08 17:47:18.171 INFO nlog/nlog.go:77 I1008 17:47:18.171636 7 shared_informer.go:270] Waiting for caches to sync for poll manager 2024-10-08 17:47:18.173 INFO nlog/nlog.go:77 I1008 17:47:18.173695 7 shared_informer.go:270] Waiting for caches to sync for endpoints 2024-10-08 17:47:18.175 INFO nlog/nlog.go:77 I1008 17:47:18.175261 7 shared_informer.go:270] Waiting for caches to sync for pod 2024-10-08 17:47:18.213 INFO controller/endpoints.go:189 Updating endpoint bob/secretflow-task-20241008170216-single-psi-0-global/2648 2024-10-08 17:47:18.213 INFO controller/endpoints.go:189 Updating endpoint bob/secretflow-task-20241008170216-single-psi-0-spu/2643 2024-10-08 17:47:18.213 INFO controller/endpoints.go:189 Updating endpoint bob/secretflow-task-20241008170216-single-psi-0-fed/2646 2024-10-08 17:47:18.271 INFO nlog/nlog.go:77 I1008 17:47:18.271495 7 shared_informer.go:277] Caches are synced for endpoints 2024-10-08 17:47:18.271 INFO controller/endpoints.go:127 Starting endpoints Controller 2024-10-08 17:47:18.272 INFO queue/queue.go:124 Finish processing item: queue id[endpoints-queue], key[bob/secretflow-task-20241008170216-single-psi-0-global] (39.61µs) 2024-10-08 17:47:18.272 INFO queue/queue.go:124 Finish processing item: queue id[endpoints-queue], key[bob/secretflow-task-20241008170216-single-psi-0-spu] (12.988µs) 2024-10-08 17:47:18.272 INFO queue/queue.go:124 Finish processing item: queue id[endpoints-queue], key[bob/secretflow-task-20241008170216-single-psi-0-fed] (10.398µs) 2024-10-08 17:47:18.272 INFO controller/domain_route.go:193 Starting workers 2024-10-08 17:47:18.272 INFO controller/domain_route.go:198 Started workers 2024-10-08 17:47:18.272 INFO controller/domain_route.go:432 add cluster bob-to-alice name:http protocol:HTTP port:1080 2024-10-08 17:47:18.272 INFO controller/domain_route.go:916 disable keep-alive for cluster:bob-to-alice-http 2024-10-08 17:47:18.272 INFO xds/xds.go:439 Add cluster:bob-to-alice-http 2024-10-08 17:47:18.272 INFO controller/domain_route.go:357 Update DomainRoute bob/bob-alice revision:595 2024-10-08 17:47:18.273 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:7398 2024-10-08 17:47:18.280 INFO controller/domain_route.go:357 Update DomainRoute bob/bob-kuscia-system revision:7469 2024-10-08 17:47:18.280 INFO xds/cluster_config.go:275 enable keep-alive for cluster:bob-to-alice-http 2024-10-08 17:47:18.280 INFO xds/xds.go:439 Add cluster:bob-to-alice-http 2024-10-08 17:47:18.280 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/bob-alice] (7.986417ms) 2024-10-08 17:47:18.282 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (8.770251ms) 2024-10-08 17:47:18.283 INFO nlog/nlog.go:77 I1008 17:47:18.283566 7 shared_informer.go:277] Caches are synced for pod 2024-10-08 17:47:18.283 INFO coredns/controller.go:52 Starting pod controller, namespace: bob 2024-10-08 17:47:18.283 INFO nlog/nlog.go:77 I1008 17:47:18.283832 7 shared_informer.go:277] Caches are synced for poll manager 2024-10-08 17:47:18.283 INFO poller/poll_manager.go:142 Starting poll manager 2024-10-08 17:47:18.284 INFO queue/queue.go:124 Finish processing item: queue id[service-poll-queue], key[bob/secretflow-task-20241008170216-single-psi-0-fed] (49.696µs) 2024-10-08 17:47:18.284 INFO nlog/nlog.go:77 I1008 17:47:18.284404 7 shared_informer.go:277] Caches are synced for endpoints 2024-10-08 17:47:18.284 INFO coredns/controller.go:42 Starting endpoint controller, namespace: bob 2024-10-08 17:47:18.284 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (13.895µs) 2024-10-08 17:47:18.284 INFO queue/queue.go:124 Finish processing item: queue id[kuscia-coredns-controller], key[bob/secretflow-task-20241008170216-single-psi-0-spu] (77.947µs) 2024-10-08 17:47:18.284 INFO queue/queue.go:124 Finish processing item: queue id[kuscia-coredns-controller], key[bob/secretflow-task-20241008170216-single-psi-0-fed] (33.519µs) 2024-10-08 17:47:18.284 INFO queue/queue.go:124 Finish processing item: queue id[kuscia-coredns-controller], key[bob/secretflow-task-20241008170216-single-psi-0-global] (12.67µs) 2024-10-08 17:47:18.284 INFO queue/queue.go:124 Finish processing item: queue id[service-poll-queue], key[bob/secretflow-task-20241008170216-single-psi-0-spu] (33.872µs) 2024-10-08 17:47:18.291 INFO xds/xds.go:439 Add cluster:service-masterproxy 2024-10-08 17:47:18.291 INFO controller/domain_route.go:497 Update rule to master success 2024-10-08 17:47:18.291 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/bob-kuscia-system] (11.71177ms) 2024-10-08 17:47:18.326 INFO controller/gateway.go:194 Envoy cluster changed, old: [] new: [172.20.0.3] 2024-10-08 17:47:18.326 INFO xds/xds.go:439 Add cluster:envoy-cluster 2024-10-08 17:47:18.739 INFO engine/engine.go:192 default server bean not exist, skip router regist 2024-10-08 17:47:18.739 INFO engine/engine.go:215 Starting bean.httpServerBean Bean: ConfManagerHttpServer 2024-10-08 17:47:18.739 INFO engine/engine.go:215 Starting bean.grpcServerBean Bean: ConfManagerGrpcServer 2024-10-08 17:47:18.740 INFO bean/grpc_server_bean.go:89 Grpc server listening on :8061 2024-10-08 17:47:18.740 INFO beans/gin_bean.go:133 https server started on :8060 2024-10-08 17:47:18.867 INFO controller/domain_route.go:357 Update DomainRoute bob/bob-kuscia-system revision:7474 2024-10-08 17:47:18.879 INFO xds/xds.go:439 Add cluster:service-masterproxy 2024-10-08 17:47:18.880 INFO controller/domain_route.go:497 Update rule to master success 2024-10-08 17:47:18.912 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/bob-kuscia-system] (44.579796ms) 2024-10-08 17:47:18.914 INFO controller/domain_route.go:357 Update DomainRoute bob/bob-kuscia-system revision:7478 2024-10-08 17:47:18.927 INFO xds/xds.go:439 Add cluster:service-masterproxy 2024-10-08 17:47:18.927 INFO controller/domain_route.go:497 Update rule to master success 2024-10-08 17:47:18.927 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/bob-kuscia-system] (13.126063ms) 2024-10-08 17:47:19.757 INFO decorator/proto_decorator.go:165 REQUEST Path: /healthZ Content-Type: application/json Body: {} 2024-10-08 17:47:19.758 INFO decorator/proto_decorator.go:216 RESPONSE{"status":{"message":"success"},"data":{"ready":true}} 2024-10-08 17:47:19.758 INFO modules/confmanager.go:184 http/https server is ready 2024-10-08 17:47:19.758 INFO modules/confmanager.go:221 ConfManager is ready 2024-10-08 17:47:19.758 INFO modules/modules.go:115 Add destroyFunc [kusciaapi] 2024-10-08 17:47:19.767 INFO engine/engine.go:192 default server bean not exist, skip router regist 2024-10-08 17:47:19.767 INFO engine/engine.go:215 Starting bean.httpServerBean Bean: kusciaAPIHttpServer 2024-10-08 17:47:19.768 INFO engine/engine.go:215 Starting bean.grpcServerBean Bean: kusciaAPIGrpcServer 2024-10-08 17:47:19.768 INFO beans/gin_bean.go:133 https server started on :8082 2024-10-08 17:47:19.768 INFO bean/grpc_server_bean.go:113 grpc server listening on :8083 2024-10-08 17:47:19.768 INFO beans/gin_bean.go:137 http server started 127.0.0.1:8092 2024-10-08 17:47:19.885 INFO decorator/proto_decorator.go:165 REQUEST Path: /healthZ Content-Type: application/json Body: {} 2024-10-08 17:47:19.885 INFO decorator/proto_decorator.go:216 RESPONSE{"status":{"message":"success"},"data":{"ready":true}} 2024-10-08 17:47:19.889 INFO modules/kusciaapi.go:229 http server is ready 2024-10-08 17:47:19.911 INFO modules/kusciaapi.go:293 KusciaApi is ready 2024-10-08 17:47:19.911 INFO modules/modules.go:115 Add destroyFunc [agent] 2024-10-08 17:47:19.911 INFO commands/root.go:46 Run root command, Namespace=bob 2024-10-08 17:47:19.912 INFO plugin/plugin.go:71 Init plugin hook:image-security succeed 2024-10-08 17:47:19.912 INFO plugin/plugin.go:71 Init plugin hook:env-import succeed 2024-10-08 17:47:19.912 INFO plugin/plugin.go:71 Init plugin hook:cert-issuance succeed 2024-10-08 17:47:19.912 INFO plugin/plugin.go:71 Init plugin hook:config-render succeed 2024-10-08 17:47:19.927 INFO node/capacity_manager.go:56 Capacity Manager, runtime: runc, capacityCfg:&{ }, reservedResCfg: &{0.5 500Mi}, rootDir: /home/kuscia/, localCapacity:true 2024-10-08 17:47:19.928 INFO node/capacity_manager.go:195 Total cpu: 8, available cpu: 8, cpu quota: 750000, cpu period: 100000 2024-10-08 17:47:19.928 INFO node/capacity_manager.go:214 Total memory: 4294967296, available memory: 3770679296 2024-10-08 17:47:19.930 INFO framework/node_controller.go:166 Configure node root-kuscia-lite-bob 2024-10-08 17:47:19.933 INFO node/generic_node.go:70 Configure generic node "root-kuscia-lite-bob" successfully 2024-10-08 17:47:19.947 WARN framework/node_controller.go:396 There is another node with the same name and machine id is running and will be replaced by the current node, node_name=root-kuscia-lite-bob, machine_id=aab24d56-353b-0dc1-4f04-b88b0adbed7f, another_node_boot_id=1728375041-1728377353254263979 2024-10-08 17:47:20.051 INFO framework/node_controller.go:567 Created new lease, name=root-kuscia-lite-bob 2024-10-08 17:47:20.051 INFO framework/node_controller.go:203 Node controller started 2024-10-08 17:47:20.065 INFO kuberuntime/kuberuntime_manager.go:190 Container runtime initialized, containerRuntime=containerd, version=v1.7.7-k3s1.26, apiVersion=v1 2024-10-08 17:47:20.080 INFO source/apiserver.go:82 Start running apiserver source 2024-10-08 17:47:20.095 INFO source/apiserver.go:58 Receive pod "secretflow-task-20241008170216-single-psi-0_bob(e7655aa2-dda6-4e0b-8e15-0d8bd4d8104b)" add event from apiserver 2024-10-08 17:47:20.096 INFO source/config.go:100 Pod change merged, source=api, adds=[secretflow-task-20241008170216-single-psi-0_bob(e7655aa2-dda6-4e0b-8e15-0d8bd4d8104b)], updates=[], deletes=[], removes=[], reconciles=[] 2024-10-08 17:47:20.195 INFO source/apiserver.go:91 Pod Informer: cache sync finished 2024-10-08 17:47:20.195 INFO framework/pods_controller.go:204 Starting Pods controller ... 2024-10-08 17:47:20.195 INFO status/status_manager.go:164 Starting to sync pod status with apiserver 2024-10-08 17:47:20.195 INFO framework/pods_controller.go:311 Starting agent main sync loop 2024-10-08 17:47:20.195 INFO framework/pods_controller.go:263 SyncLoop ADD, source=api, pods=[secretflow-task-20241008170216-single-psi-0_bob(e7655aa2-dda6-4e0b-8e15-0d8bd4d8104b)] 2024-10-08 17:47:20.196 INFO pod/cri_provider.go:310 Starting CRI provider ... 2024-10-08 17:47:20.196 INFO commands/root.go:190 Agent started 2024-10-08 17:47:20.197 INFO modules/agent.go:175 Agent is ready 2024-10-08 17:47:20.197 INFO modules/modules.go:115 Add destroyFunc [datamesh] 2024-10-08 17:47:20.231 INFO pleg/generic.go:303 Generic (PLEG): container finished, podID=e7655aa2-dda6-4e0b-8e15-0d8bd4d8104b, containerID=0e2fd5be280766df6ab3317af500a83a46d124bd254520effaf431e4f69792cd, exitCode=255 2024-10-08 17:47:20.232 INFO pod/cri_provider.go:348 Receive pleg event: &{ID:e7655aa2-dda6-4e0b-8e15-0d8bd4d8104b Type:ContainerDied Data:0e2fd5be280766df6ab3317af500a83a46d124bd254520effaf431e4f69792cd} 2024-10-08 17:47:20.232 INFO pod/cri_provider.go:348 Receive pleg event: &{ID:e7655aa2-dda6-4e0b-8e15-0d8bd4d8104b Type:ContainerDied Data:80669c65b6a10e69276b52a889ec7483587e02497938a11848b7ed258ebcd489} 2024-10-08 17:47:20.250 INFO framework/node_controller.go:206 Node is ready 2024-10-08 17:47:20.502 INFO modules/datamesh.go:60 Datamesh namespace:bob. 2024-10-08 17:47:20.502 INFO bean/operator_bean.go:53 OperatorBean init 2024-10-08 17:47:20.502 INFO engine/engine.go:192 default server bean not exist, skip router regist 2024-10-08 17:47:20.502 INFO engine/engine.go:215 Starting bean.httpServerBean Bean: DataMeshHttpServer 2024-10-08 17:47:20.503 INFO engine/engine.go:215 Starting bean.grpcServerBean Bean: DataMeshGrpcServer 2024-10-08 17:47:20.503 INFO engine/engine.go:215 Starting *bean.operatorBean Bean: DataMeshOperator 2024-10-08 17:47:20.503 INFO service/operator.go:47 DataMesh operator service start 2024-10-08 17:47:20.503 INFO beans/gin_bean.go:133 https server started on :8070 2024-10-08 17:47:20.503 INFO bean/grpc_server_bean.go:99 Grpc server listening on :8071 2024-10-08 17:47:20.520 INFO service/operator.go:91 Datasource default-data-source has been created successful. 2024-10-08 17:47:20.532 INFO service/operator.go:106 Datasource default-dp-data-source has been created successful. 2024-10-08 17:47:20.621 INFO decorator/proto_decorator.go:165 REQUEST Path: /healthZ Content-Type: application/json Body: {} 2024-10-08 17:47:20.621 INFO decorator/proto_decorator.go:216 RESPONSE{"status":{"message":"success"},"data":{"ready":true}} 2024-10-08 17:47:20.621 INFO modules/datamesh.go:140 http server is ready 2024-10-08 17:47:20.622 INFO modules/datamesh.go:177 DataMesh is ready 2024-10-08 17:47:20.622 INFO modules/modules.go:115 Add destroyFunc [nodeexporter] 2024-10-08 17:47:20.622 INFO supervisor/supervisor.go:75 [node_exporter] start and watch subprocess 2024-10-08 17:47:20.622 INFO supervisor/supervisor.go:101 [node_exporter] try to start new process 2024-10-08 17:47:20.623 INFO process/process.go:76 Set pid[78] oom score adj to -600 2024-10-08 17:47:21.624 INFO modules/nodeexporter.go:142 Node_exporter is ready 2024-10-08 17:47:21.625 INFO modules/modules.go:115 Add destroyFunc [ssexporter] 2024-10-08 17:47:22.626 INFO modules/ssexporter.go:128 Ss exporter is ready 2024-10-08 17:47:22.626 INFO modules/modules.go:115 Add destroyFunc [metricexporter] 2024-10-08 17:47:22.627 INFO metricexporter/metricexporter.go:87 Start to export metrics... 2024-10-08 17:47:23.629 INFO modules/metricexporter.go:132 Metric exporter is ready 2024-10-08 17:47:23.630 INFO process/process.go:76 Set pid[1] oom score adj to -999 2024-10-08 17:47:23.630 INFO process/process.go:76 Set pid[7] oom score adj to -900 2024-10-08 17:47:25.198 INFO framework/pod.go:704 Pod "secretflow-task-20241008170216-single-psi-0_bob(e7655aa2-dda6-4e0b-8e15-0d8bd4d8104b)" is restartable after termination due to UID reuse, but pod phase is terminal 2024-10-08 17:47:32.944 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (23.187µs) 2024-10-08 17:47:32.944 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:7524 2024-10-08 17:47:32.951 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (6.681876ms) 2024-10-08 17:47:33.284 INFO poller/poll_manager.go:194 Current gateway member count is 0, while alive gateway count is 1 2024-10-08 17:47:33.285 INFO poller/poll_manager.go:214 Gateway Members changed, old: [] new: [root-kuscia-lite-bob] 2024-10-08 17:47:33.285 INFO queue/queue.go:124 Finish processing item: queue id[service-poll-queue], key[bob/secretflow-task-20241008170216-single-psi-0-fed] (35.46µs) 2024-10-08 17:47:33.285 INFO queue/queue.go:124 Finish processing item: queue id[service-poll-queue], key[bob/secretflow-task-20241008170216-single-psi-0-spu] (5.961µs) 2024-10-08 17:47:33.285 INFO queue/queue.go:124 Finish processing item: queue id[service-poll-queue], key[kuscia-handshake] (3.806µs) 2024-10-08 17:47:47.923 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (60.395µs) 2024-10-08 17:47:47.923 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:7559 2024-10-08 17:47:47.930 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (6.735928ms) 2024-10-08 17:48:02.930 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (64.792µs) 2024-10-08 17:48:02.930 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:7595 2024-10-08 17:48:02.938 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (7.683139ms) 2024-10-08 17:48:17.929 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (20.101µs) 2024-10-08 17:48:17.930 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:7627 2024-10-08 17:48:17.936 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (6.818276ms) 2024-10-08 17:48:20.458 INFO pod/cri_provider.go:348 Receive pleg event: &{ID:e7655aa2-dda6-4e0b-8e15-0d8bd4d8104b Type:ContainerRemoved Data:0e2fd5be280766df6ab3317af500a83a46d124bd254520effaf431e4f69792cd} 2024-10-08 17:48:20.458 INFO pod/cri_provider.go:348 Receive pleg event: &{ID:e7655aa2-dda6-4e0b-8e15-0d8bd4d8104b Type:ContainerRemoved Data:80669c65b6a10e69276b52a889ec7483587e02497938a11848b7ed258ebcd489} 2024-10-08 17:48:32.931 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (21.038µs) 2024-10-08 17:48:32.931 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:7668 2024-10-08 17:48:32.938 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (7.027591ms) 2024-10-08 17:48:47.930 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:7700 2024-10-08 17:48:47.930 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (25.707µs) 2024-10-08 17:48:47.937 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (6.665247ms) 2024-10-08 17:49:02.923 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (19.099µs) 2024-10-08 17:49:02.924 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:7738 2024-10-08 17:49:02.931 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (7.631793ms) 2024-10-08 17:49:17.922 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (20.662µs) 2024-10-08 17:49:17.922 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:7770 2024-10-08 17:49:17.929 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (6.88743ms) 2024-10-08 17:49:32.925 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (22.739µs) 2024-10-08 17:49:32.925 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:7807 2024-10-08 17:49:32.932 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (7.010716ms) 2024-10-08 17:49:47.925 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (33.005µs) 2024-10-08 17:49:47.925 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:7839 2024-10-08 17:49:47.932 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (6.731155ms) 2024-10-08 17:50:02.896 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (15.17µs) 2024-10-08 17:50:02.896 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:7872 2024-10-08 17:50:02.899 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (2.871277ms) 2024-10-08 17:50:17.902 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (24.718µs) 2024-10-08 17:50:17.902 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:7905 2024-10-08 17:50:17.909 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (6.863969ms) 2024-10-08 17:50:32.903 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (12.772µs) 2024-10-08 17:50:32.903 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:7941 2024-10-08 17:50:32.907 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (4.046024ms) 2024-10-08 17:50:47.899 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (17.198µs) 2024-10-08 17:50:47.900 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:7974 2024-10-08 17:50:47.905 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (5.613504ms) 2024-10-08 17:51:02.956 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (22.671µs) 2024-10-08 17:51:02.956 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:8013 2024-10-08 17:51:02.963 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (6.776826ms) 2024-10-08 17:51:17.923 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:8046 2024-10-08 17:51:17.923 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (29.319µs) 2024-10-08 17:51:17.931 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (7.571239ms) 2024-10-08 17:51:32.932 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:8081 2024-10-08 17:51:32.933 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (23.212µs) 2024-10-08 17:51:32.939 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (6.882602ms) 2024-10-08 17:51:47.929 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:8114 2024-10-08 17:51:47.930 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (31.774µs) 2024-10-08 17:51:47.936 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (7.096499ms) 2024-10-08 17:52:02.897 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (23.081µs) 2024-10-08 17:52:02.897 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:8150 2024-10-08 17:52:02.903 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (6.690379ms) 2024-10-08 17:52:17.927 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (22.957µs) 2024-10-08 17:52:17.928 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:8186 2024-10-08 17:52:17.935 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (7.002928ms) 2024-10-08 17:52:32.931 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (31.808µs) 2024-10-08 17:52:32.931 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:8222 2024-10-08 17:52:32.938 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (6.617508ms) 2024-10-08 17:52:47.926 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (23.977µs) 2024-10-08 17:52:47.926 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:8255 2024-10-08 17:52:47.934 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (8.363894ms) 2024-10-08 17:53:02.931 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (28.09µs) 2024-10-08 17:53:02.931 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:8291 2024-10-08 17:53:02.938 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (6.541428ms) 2024-10-08 17:53:17.926 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (21.761µs) 2024-10-08 17:53:17.927 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:8324 2024-10-08 17:53:17.932 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (5.574613ms) 2024-10-08 17:53:32.926 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (23.83µs) 2024-10-08 17:53:32.927 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:8359 2024-10-08 17:53:32.934 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (7.353937ms) 2024-10-08 17:53:47.917 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (23.184µs) 2024-10-08 17:53:47.917 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:8392 2024-10-08 17:53:47.923 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (6.446377ms) 2024-10-08 17:54:02.926 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (19.505µs) 2024-10-08 17:54:02.926 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:8427 2024-10-08 17:54:02.933 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (6.821493ms) 2024-10-08 17:54:17.936 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (23.336µs) 2024-10-08 17:54:17.936 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:8462 2024-10-08 17:54:17.943 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (6.791324ms) 2024-10-08 17:54:32.924 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:8496 2024-10-08 17:54:32.924 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (22.087µs) 2024-10-08 17:54:32.930 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (5.280763ms) 2024-10-08 17:54:48.017 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (30.004µs) 2024-10-08 17:54:48.017 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:8529 2024-10-08 17:54:48.024 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (7.09793ms) 2024-10-08 17:55:02.933 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (26.7µs) 2024-10-08 17:55:02.933 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:8565 2024-10-08 17:55:02.940 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (6.787003ms) 2024-10-08 17:55:17.897 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (22.109µs) 2024-10-08 17:55:17.897 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:8595 2024-10-08 17:55:17.904 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (6.902991ms) 2024-10-08 17:55:32.923 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:8634 2024-10-08 17:55:32.925 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (26.51µs) 2024-10-08 17:55:32.930 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (6.668843ms) 2024-10-08 17:55:47.905 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (19.252µs) 2024-10-08 17:55:47.905 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:8663 2024-10-08 17:55:47.912 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (6.637096ms) 2024-10-08 17:56:02.898 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (23.575µs) 2024-10-08 17:56:02.898 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:8700 2024-10-08 17:56:02.905 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (6.73889ms) 2024-10-08 17:56:17.924 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (19.145µs) 2024-10-08 17:56:17.924 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:8735 2024-10-08 17:56:17.931 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (6.465542ms) 2024-10-08 17:56:32.932 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:8772 2024-10-08 17:56:32.934 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (25.593µs) 2024-10-08 17:56:32.939 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (6.549148ms) 2024-10-08 17:56:47.904 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (23.625µs) 2024-10-08 17:56:47.905 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:8801 2024-10-08 17:56:47.913 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (7.916939ms) 2024-10-08 17:57:02.921 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (19.747µs) 2024-10-08 17:57:02.921 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:8844 2024-10-08 17:57:02.928 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (7.084222ms) 2024-10-08 17:57:17.922 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:8880 2024-10-08 17:57:17.928 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (26.113µs) 2024-10-08 17:57:17.930 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (8.268089ms) 2024-10-08 17:57:18.194 INFO queue/queue.go:124 Finish processing item: queue id[kuscia-coredns-controller], key[bob/secretflow-task-20241008170216-single-psi-0-fed] (28.432µs) 2024-10-08 17:57:18.194 INFO queue/queue.go:124 Finish processing item: queue id[kuscia-coredns-controller], key[bob/secretflow-task-20241008170216-single-psi-0-global] (13.839µs) 2024-10-08 17:57:18.194 INFO queue/queue.go:124 Finish processing item: queue id[kuscia-coredns-controller], key[bob/secretflow-task-20241008170216-single-psi-0-spu] (9.2µs) 2024-10-08 17:57:18.214 INFO controller/endpoints.go:189 Updating endpoint bob/secretflow-task-20241008170216-single-psi-0-fed/2646 2024-10-08 17:57:18.215 INFO controller/endpoints.go:189 Updating endpoint bob/secretflow-task-20241008170216-single-psi-0-global/2648 2024-10-08 17:57:18.215 INFO queue/queue.go:124 Finish processing item: queue id[endpoints-queue], key[bob/secretflow-task-20241008170216-single-psi-0-fed] (53.038µs) 2024-10-08 17:57:18.215 INFO controller/endpoints.go:189 Updating endpoint bob/secretflow-task-20241008170216-single-psi-0-spu/2643 2024-10-08 17:57:18.215 INFO queue/queue.go:124 Finish processing item: queue id[endpoints-queue], key[bob/secretflow-task-20241008170216-single-psi-0-spu] (20.154µs) 2024-10-08 17:57:18.215 INFO queue/queue.go:124 Finish processing item: queue id[endpoints-queue], key[bob/secretflow-task-20241008170216-single-psi-0-global] (39.736µs) 2024-10-08 17:57:18.219 INFO queue/queue.go:124 Finish processing item: queue id[service-poll-queue], key[bob/secretflow-task-20241008170216-single-psi-0-spu] (19.7µs) 2024-10-08 17:57:18.219 INFO queue/queue.go:124 Finish processing item: queue id[service-poll-queue], key[bob/secretflow-task-20241008170216-single-psi-0-fed] (20.03µs) 2024-10-08 17:57:18.220 INFO queue/queue.go:124 Finish processing item: queue id[kuscia-coredns-controller], key[bob/secretflow-task-20241008170216-single-psi-0-fed] (11.56µs) 2024-10-08 17:57:18.220 INFO queue/queue.go:124 Finish processing item: queue id[kuscia-coredns-controller], key[bob/secretflow-task-20241008170216-single-psi-0-global] (8.468µs) 2024-10-08 17:57:18.220 INFO queue/queue.go:124 Finish processing item: queue id[kuscia-coredns-controller], key[bob/secretflow-task-20241008170216-single-psi-0-spu] (4.459µs) 2024-10-08 17:57:18.223 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (13.346µs) 2024-10-08 17:57:18.223 INFO controller/domain_route.go:357 Update DomainRoute bob/bob-kuscia-system revision:7478 2024-10-08 17:57:18.223 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:8880 2024-10-08 17:57:18.226 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (3.569511ms) 2024-10-08 17:57:18.226 INFO controller/domain_route.go:432 add cluster bob-to-alice name:http protocol:HTTP port:1080 2024-10-08 17:57:18.226 INFO xds/xds.go:439 Add cluster:bob-to-alice-http 2024-10-08 17:57:18.226 INFO controller/domain_route.go:357 Update DomainRoute bob/bob-alice revision:595 2024-10-08 17:57:18.228 INFO xds/xds.go:439 Add cluster:service-masterproxy 2024-10-08 17:57:18.228 INFO controller/domain_route.go:497 Update rule to master success 2024-10-08 17:57:18.228 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/bob-kuscia-system] (5.372476ms) 2024-10-08 17:57:18.229 INFO xds/xds.go:439 Add cluster:bob-to-alice-http 2024-10-08 17:57:18.229 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/bob-alice] (2.872884ms) 2024-10-08 17:57:32.897 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (23.309µs) 2024-10-08 17:57:32.898 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:8916 2024-10-08 17:57:32.905 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (7.480566ms) 2024-10-08 17:57:47.929 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:8954 2024-10-08 17:57:47.936 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (6.407976ms) 2024-10-08 17:57:47.936 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (24.7µs) 2024-10-08 17:58:02.923 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (19.694µs) 2024-10-08 17:58:02.923 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:8991 2024-10-08 17:58:02.930 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (6.625019ms) 2024-10-08 17:58:17.897 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (25.615µs) 2024-10-08 17:58:17.897 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:9021 2024-10-08 17:58:17.904 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (6.697389ms) 2024-10-08 17:58:32.925 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (23.27µs) 2024-10-08 17:58:32.925 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:9060 2024-10-08 17:58:32.932 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (7.311917ms) 2024-10-08 17:58:47.929 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (23.009µs) 2024-10-08 17:58:47.929 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:9093 2024-10-08 17:58:47.936 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (6.695235ms) 2024-10-08 17:59:02.925 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (20.811µs) 2024-10-08 17:59:02.926 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:9129 2024-10-08 17:59:02.932 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (6.505263ms) 2024-10-08 17:59:17.898 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (20.574µs) 2024-10-08 17:59:17.898 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:9158 2024-10-08 17:59:17.905 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (6.58714ms) 2024-10-08 17:59:32.926 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (21.637µs) 2024-10-08 17:59:32.926 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:9198 2024-10-08 17:59:32.933 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (6.714564ms) 2024-10-08 17:59:47.929 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:9230 2024-10-08 17:59:47.929 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (29.223µs) 2024-10-08 17:59:47.936 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (6.629017ms) 2024-10-08 18:00:02.918 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (12.858µs) 2024-10-08 18:00:02.918 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:9265 2024-10-08 18:00:02.925 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (7.275823ms) 2024-10-08 18:00:17.923 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (22.026µs) 2024-10-08 18:00:17.923 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:9298 2024-10-08 18:00:17.930 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (6.701974ms) 2024-10-08 18:00:32.925 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:9334 2024-10-08 18:00:32.925 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (29.641µs) 2024-10-08 18:00:32.931 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (6.744239ms) 2024-10-08 18:00:47.924 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:9367 2024-10-08 18:00:47.925 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (26.991µs) 2024-10-08 18:00:47.931 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (6.769533ms) 2024-10-08 18:01:02.903 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:9399 2024-10-08 18:01:02.904 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (27.012µs) 2024-10-08 18:01:02.910 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (7.003594ms) 2024-10-08 18:01:17.929 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (21.486µs) 2024-10-08 18:01:17.929 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:9436 2024-10-08 18:01:17.935 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (6.554062ms) 2024-10-08 18:01:32.927 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (28.839µs) 2024-10-08 18:01:32.927 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:9471 2024-10-08 18:01:32.934 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (6.935099ms) 2024-10-08 18:01:47.926 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (23.166µs) 2024-10-08 18:01:47.926 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:9505 2024-10-08 18:01:47.933 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (6.681219ms) 2024-10-08 18:02:02.929 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:9544 2024-10-08 18:02:02.929 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (33.213µs) 2024-10-08 18:02:02.935 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (6.712244ms) 2024-10-08 18:02:17.924 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (19.451µs) 2024-10-08 18:02:17.924 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:9577 2024-10-08 18:02:17.931 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (6.498575ms) 2024-10-08 18:02:32.921 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (19.163µs) 2024-10-08 18:02:32.921 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:9613 2024-10-08 18:02:32.927 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (5.711018ms) 2024-10-08 18:02:47.897 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (17.354µs) 2024-10-08 18:02:47.897 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:9642 2024-10-08 18:02:47.903 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (5.695726ms) 2024-10-08 18:03:02.925 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (21.914µs) 2024-10-08 18:03:02.925 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:9681 2024-10-08 18:03:02.932 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (6.546306ms) 2024-10-08 18:03:17.932 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (20.876µs) 2024-10-08 18:03:17.932 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:9714 2024-10-08 18:03:17.940 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (8.045423ms) 2024-10-08 18:03:32.933 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:9750 2024-10-08 18:03:32.933 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (30.155µs) 2024-10-08 18:03:32.940 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (6.872098ms) 2024-10-08 18:03:47.902 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (22.935µs) 2024-10-08 18:03:47.902 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:9780 2024-10-08 18:03:47.909 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (7.516339ms) 2024-10-08 18:04:02.937 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (27.747µs) 2024-10-08 18:04:02.938 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:9819 2024-10-08 18:04:02.944 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (6.925488ms) 2024-10-08 18:04:17.931 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (23.316µs) 2024-10-08 18:04:17.931 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:9850 2024-10-08 18:04:17.938 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (6.933507ms) 2024-10-08 18:04:32.922 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (22.546µs) 2024-10-08 18:04:32.922 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:9886 2024-10-08 18:04:32.929 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (7.087259ms) 2024-10-08 18:04:47.903 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:9917 2024-10-08 18:04:47.909 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (6.853662ms) 2024-10-08 18:04:47.903 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (28.695µs) 2024-10-08 18:05:02.924 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (26.354µs) 2024-10-08 18:05:02.924 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:9955 2024-10-08 18:05:02.932 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (7.869557ms) 2024-10-08 18:05:17.931 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (22.547µs) 2024-10-08 18:05:17.931 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:9989 2024-10-08 18:05:17.938 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (6.662372ms) 2024-10-08 18:05:32.926 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (26.883µs) 2024-10-08 18:05:32.926 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:10025 2024-10-08 18:05:32.933 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (6.672124ms) 2024-10-08 18:05:47.933 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (22.062µs) 2024-10-08 18:05:47.934 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:10058 2024-10-08 18:05:47.940 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (6.692917ms) 2024-10-08 18:06:02.922 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (18.498µs) 2024-10-08 18:06:02.923 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:10094 2024-10-08 18:06:02.928 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (5.073288ms) 2024-10-08 18:06:17.936 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-bob:alice] (25.894µs) 2024-10-08 18:06:17.936 INFO controller/domain_route.go:357 Update DomainRoute bob/alice-bob revision:10126 2024-10-08 18:06:17.943 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[bob/alice-bob] (6.659078ms) [root@root-kuscia-lite-bob kuscia]#

[wangzul]

看一下路由信息，在master节点执行kubectl get cdr

[byeyang]

看一下路由信息，在master节点执行kubectl get cdr

[root@root-kuscia-master kuscia]# kubectl get cdr NAME SOURCE DESTINATION HOST AUTHENTICATION READY alice-kuscia-system alice kuscia-system Token True alice-bob alice bob root-kuscia-lite-bob Token True bob-alice bob alice root-kuscia-lite-alice Token True bob-kuscia-system bob kuscia-system Token True [root@root-kuscia-master kuscia]#

[wangzul]

很抱歉经技术人员确认NSJAIL_PSI需要等secretflow模块适配，目前暂时不可用。

[byeyang]

很抱歉经技术人员确认NSJAIL_PSI需要等secretflow模块适配，目前暂时不可用。

好的，感谢答复！

[UniqueMarvin]

hi @byeyang 你好，方便问一下具体是什么场景需要使用nsjail么

[byeyang]

hi @byeyang 你好，方便问一下具体是什么场景需要使用nsjail么

暂时不明确，请问现在能用了吗 有适配了吗