search

secretflow / kuscia

Kuscia(Kubernetes-based Secure Collaborative InfrA) is a K8s-based privacy-preserving computing task orchestration framework.
https://www.secretflow.org.cn/docs/kuscia/latest/zh-Hans
Apache License 2.0
73 stars 53 forks source link

kuscia中心化集群K8S部署执行任务失败 #426

Closed Meng-xiangkun closed 2 months ago

Meng-xiangkun commented 2 months ago

Issue Type

Feature

Search for existing issues similar to yours

Yes

Kuscia Version

0.10.0b0

Link to Relevant Documentation

No response

Question Details

k8s方式部署中心化集群，执行任务失败，请问怎么集成secretflow执行计算，Runp模式使用的是kuscia0.10.0b0镜像

2024-09-12 14:52:20.261 INFO interceptor/common.go:92 [GRPC] [GRPC /kuscia.proto.api.v1alpha1.kusciaapi.DomainService/QueryDomain] Duration: 9.76729ms, StatusCode: 0, ForwardHost: [], ContextType: [], Request: {"domain_id":"bob"}, Response: {"data":{"annotations":{"domain/bob":"kuscia.secretflow/domain-type=embedded","kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"kuscia.secretflow/v1alpha1\",\"kind\":\"Domain\",\"metadata\":{\"annotations\":{\"domain/bob\":\"kuscia.secretflow/domain-type=embedded\"},\"name\":\"bob\"},\"spec\":{\"authCenter\":{\"authenticationType\":\"Token\",\"tokenGenMethod\":\"UID-RSA-GEN\"},\"cert\":null,\"master\":null,\"role\":null}}\n"},"auth_center":{"authentication_type":"Token","token_gen_method":"UID-RSA-GEN"},"cert":"LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURBVENDQWVtZ0F3SUJBZ0lCQVRBTkJna3Foa2lHOXcwQkFRc0ZBREFZTVJZd0ZBWURWUVFERXcxcmRYTmoKYVdFdGMzbHpkR1Z0TUI0WERUY3dNREV3TVRBd01EQXdNRm9YRFRnd01ERXdNVEF3TURBd01Gb3dEakVNTUFvRwpBMVVFQXhNRFltOWlNSUlCSWpBTkJna3Foa2lHOXcwQkFRRUZBQU9DQVE4QU1JSUJDZ0tDQVFFQXlWKzAyT052Ck9SKy8xVE9IYjl3N0hRRlNiRmxUNUtkeHhLN3ZwU3MwWjdXcnRjeld0ZXBjcmsrVUhTWHREdUhpV0tBcTJpQksKK3drWGhBUzA0WDNySWxHQjhtRDVwbEMrMWlaaFg4NnV4eUFFZzB5MkdicCtrajVRamhBWC9LbDBsL1liSTQyaQpOWmV0SENvdDJQbXh

2024-09-12 14:52:20.273 INFO interceptor/common.go:92 [GRPC] [GRPC /kuscia.proto.api.v1alpha1.kusciaapi.DomainService/QueryDomain] Duration: 9.22525ms, StatusCode: 0, ForwardHost: [], ContextType: [], Request: {"domain_id":"alice"}, Response: {"data":{"annotations":{"domain/alice":"kuscia.secretflow/domain-type=embedded","kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"kuscia.secretflow/v1alpha1\",\"kind\":\"Domain\",\"metadata\":{\"annotations\":{\"domain/alice\":\"kuscia.secretflow/domain-type=embedded\"},\"name\":\"alice\"},\"spec\":{\"authCenter\":{\"authenticationType\":\"Token\",\"tokenGenMethod\":\"UID-RSA-GEN\"},\"cert\":null,\"master\":null,\"role\":null}}\n"},"auth_center":{"authentication_type":"Token","token_gen_method":"UID-RSA-GEN"},"cert":"LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURBekNDQWV1Z0F3SUJBZ0lCQVRBTkJna3Foa2lHOXcwQkFRc0ZBREFZTVJZd0ZBWURWUVFERXcxcmRYTmoKYVdFdGMzbHpkR1Z0TUI0WERUY3dNREV3TVRBd01EQXdNRm9YRFRnd01ERXdNVEF3TURBd01Gb3dFREVPTUF3RwpBMVVFQXhNRllXeHBZMlV3Z2dFaU1BMEdDU3FHU0liM0RRRUJBUVVBQTRJQkR3QXdnZ0VLQW9JQkFRREpYN1RZCjQyODVINy9WTTRkdjNEc2RBVkpzV1ZQa3AzSEVydStsS3pSbnRhdTF6TmExNmx5dVQ1UWRKZTBPNGVKWW9DcmEKSUVyN0NSZUVCTFRoZmVzaVVZSHlZUG1tVUw3V0ptRmZ6cTdISUFTRFRMWVp1bjZTUGxDT0VCZjhxWFNYOWhzagpqYUkxbDYwY

2024-09-12 14:52:20.286 INFO interceptor/common.go:92 [GRPC] [GRPC /kuscia.proto.api.v1alpha1.kusciaapi.DomainRouteService/QueryDomainRoute] Duration: 9.929205ms, StatusCode: 0, ForwardHost: [], ContextType: [], Request: {"destination":"alice","source":"bob"}, Response: {"data":{"authentication_type":"Token","destination":"alice","endpoint":{"host":"kuscia-lite-alice.data-develop-operate-dev.svc.cluster.local","ports":[{"name":"http","port":1080,"protocol":"HTTP","path_prefix":"/"}]},"name":"bob-alice","source":"bob","status":{"status":"Succeeded"},"token_config":{"destination_public_key":"LS0tLS1CRUdJTiBSU0EgUFVCTElDIEtFWS0tLS0tCk1JSUJDZ0tDQVFFQXlWKzAyT052T1IrLzFUT0hiOXc3SFFGU2JGbFQ1S2R4eEs3dnBTczBaN1dydGN6V3RlcGMKcmsrVUhTWHREdUhpV0tBcTJpQksrd2tYaEFTMDRYM3JJbEdCOG1ENXBsQysxaVpoWDg2dXh5QUVnMHkyR2JwKwprajVRamhBWC9LbDBsL1liSTQyaU5aZXRIQ290MlBteEVXaTlId1puY01OQTM0MWxBWXRGNUM5SzBWQVpOSHZICnRHSzN2S1dTQjZ6Mk83ekY3NXJ0RjZiWTA2azdzTm82bzhvMFJxbGt2OGdCaXJuemprREh4eXBjRWNndldMMGgKTVkxUTVualN5OW5uV1JpMmFnc0kvMlVRSUgxYnFJWjlnVXVUTko0WGZmdWFDSzFaS0tGY3dRSitmTGdMYVYwbgpzekFrSEgxRkxmdWFZbHA0MjV4ZHl5OE1RTWlQa1NkYVdRSURBUUFCCi0tLS0tRU5EIFJTQSBQVUJMSUMgS0VZLS0tLS0K","rolling_update_period":86400,"source_public_key":"LS0tLS1CRUdJTiBSU0EgUFVCTElDIEtFWS0tLS0tCk1JSUJDZ0tDQVFFQXlWKzAyT052T1IrLzFUT

2024-09-12 14:52:20.299 INFO interceptor/common.go:92 [GRPC] [GRPC /kuscia.proto.api.v1alpha1.kusciaapi.DomainRouteService/QueryDomainRoute] Duration: 9.918146ms, StatusCode: 0, ForwardHost: [], ContextType: [], Request: {"destination":"bob","source":"alice"}, Response: {"data":{"authentication_type":"Token","destination":"bob","endpoint":{"host":"kuscia-lite-bob.data-develop-operate-dev.svc.cluster.local","ports":[{"name":"http","port":1080,"protocol":"HTTP","path_prefix":"/"}]},"name":"alice-bob","source":"alice","status":{"status":"Succeeded"},"token_config":{"destination_public_key":"LS0tLS1CRUdJTiBSU0EgUFVCTElDIEtFWS0tLS0tCk1JSUJDZ0tDQVFFQXlWKzAyT052T1IrLzFUT0hiOXc3SFFGU2JGbFQ1S2R4eEs3dnBTczBaN1dydGN6V3RlcGMKcmsrVUhTWHREdUhpV0tBcTJpQksrd2tYaEFTMDRYM3JJbEdCOG1ENXBsQysxaVpoWDg2dXh5QUVnMHkyR2JwKwprajVRamhBWC9LbDBsL1liSTQyaU5aZXRIQ290MlBteEVXaTlId1puY01OQTM0MWxBWXRGNUM5SzBWQVpOSHZICnRHSzN2S1dTQjZ6Mk83ekY3NXJ0RjZiWTA2azdzTm82bzhvMFJxbGt2OGdCaXJuemprREh4eXBjRWNndldMMGgKTVkxUTVualN5OW5uV1JpMmFnc0kvMlVRSUgxYnFJWjlnVXVUTko0WGZmdWFDSzFaS0tGY3dRSitmTGdMYVYwbgpzekFrSEgxRkxmdWFZbHA0MjV4ZHl5OE1RTWlQa1NkYVdRSURBUUFCCi0tLS0tRU5EIFJTQSBQVUJMSUMgS0VZLS0tLS0K","rolling_update_period":86400,"source_public_key":"LS0tLS1CRUdJTiBSU0EgUFVCTElDIEtFWS0tLS0tCk1JSUJDZ0tDQVFFQXlWKzAyT052T1IrLzFUT0h

2024-09-12 14:52:20.347 INFO interceptor/common.go:92 [GRPC] [GRPC /kuscia.proto.api.v1alpha1.kusciaapi.DomainDataService/QueryDomainData] Duration: 10.429097ms, StatusCode: 0, ForwardHost: [], ContextType: [], Request: {"data":{"domain_id":"alice","domaindata_id":"alice-table"}}, Response: {"data":{"attributes":{"description":"alice demo data"},"author":"alice","columns":[{"name":"id1","type":"str"},{"name":"age","type":"float"},{"name":"education","type":"float"},{"name":"default","type":"float"},{"name":"balance","type":"float"},{"name":"housing","type":"float"},{"name":"loan","type":"float"},{"name":"day","type":"float"},{"name":"duration","type":"float"},{"name":"campaign","type":"float"},{"name":"pdays","type":"float"},{"name":"previous","type":"float"},{"name":"job_blue-collar","type":"float"},{"name":"job_entrepreneur","type":"float"},{"name":"job_housemaid","type":"float"},{"name":"job_management","type":"float"},{"name":"job_retired","type":"float"},{"name":"job_self-employed","type":"float"},{"name":"job_services","type":"float"},{"name":"job_student","type":"float"},{"name":"job_technician","type":"float"},{"name":"job_unemployed","type":"float"},{"name":"marital_divorced","type":"float"},{"name":"marital_married","type":"float"},{"name":"marital_single","type":"float"}],"datasource_

2024-09-12 14:52:20.366 INFO interceptor/common.go:92 [GRPC] [GRPC /kuscia.proto.api.v1alpha1.kusciaapi.DomainDataService/QueryDomainData] Duration: 9.936527ms, StatusCode: 0, ForwardHost: [], ContextType: [], Request: {"data":{"domain_id":"bob","domaindata_id":"bob-table"}}, Response: {"data":{"attributes":{"description":"bob demo data"},"author":"bob","columns":[{"name":"id2","type":"str"},{"name":"contact_cellular","type":"float"},{"name":"contact_telephone","type":"float"},{"name":"contact_unknown","type":"float"},{"name":"month_apr","type":"float"},{"name":"month_aug","type":"float"},{"name":"month_dec","type":"float"},{"name":"month_feb","type":"float"},{"name":"month_jan","type":"float"},{"name":"month_jul","type":"float"},{"name":"month_jun","type":"float"},{"name":"month_mar","type":"float"},{"name":"month_may","type":"float"},{"name":"month_nov","type":"float"},{"name":"month_oct","type":"float"},{"name":"month_sep","type":"float"},{"name":"poutcome_failure","type":"float"},{"name":"poutcome_other","type":"float"},{"name":"poutcome_success","type":"float"},{"name":"poutcome_unknown","type":"float"},{"name":"y","type":"int"}],"datasource_id":"default-data-source","domain_id":"bob","domaindata_id":"bob-table","file_format":0,"name":"bob.csv","relative_uri":"bob.csv","status":"Availa

2024-09-12 14:52:20.413 INFO interceptor/common.go:92 [GRPC] [GRPC /kuscia.proto.api.v1alpha1.kusciaapi.JobService/CreateJob] Duration: 20.248363ms, StatusCode: 0, ForwardHost: [], ContextType: [], Request: {"custom_fields":{},"initiator":"bob","job_id":"yyqt","max_parallelism":1,"tasks":[{"app_image":"secretflow-image","parties":[{"domain_id":"bob"},{"domain_id":"alice"}],"alias":"yyqt-ubryppxk-node-3","task_id":"yyqt-ubryppxk-node-3","task_input_config":"{\n \"sf_datasource_config\": {\n \"bob\": {\n \"id\": \"default-data-source\"\n },\n \"alice\": {\n \"id\": \"default-data-source\"\n }\n },\n \"sf_cluster_desc\": {\n \"parties\": [\"bob\", \"alice\"],\n \"devices\": [{\n \"name\": \"spu\",\n \"type\": \"spu\",\n \"parties\": [\"bob\", \"alice\"],\n \"config\": \"{\\\"runtime_config\\\":{\\\"protocol\\\":\\\"SEMI2K\\\",\\\"field\\\":\\\"FM128\\\"},\\\"link_desc\\\":{\\\"connect_retry_times\\\":60,\\\"connect_retry_interval_ms\\\":1000,\\\"brpc_channel_protocol\\\":\\\"http\\\",\\\"brpc_channel_connection_type\\\":\\\"pooled\\\",\\\"recv_timeout_ms\\\":1200000,\\\"http_timeout_ms\\\":1200000}}\"\n }, {\n \"name\": \"heu\",\n \"type\": \"heu\",\n , Response: {"data":{"job_id":"yyqt"},"status":{"code":0,"details":null,"message":"success"}}

2024-09-12 14:52:20.415 ERROR grpchandler/job_handler.go:82 Send job event error: rpc error: code = Unavailable desc = transport is closing

2024-09-12 14:52:20.416 ERROR grpchandler/job_handler.go:82 Send job event error: rpc error: code = Unavailable desc = transport is closing

2024-09-12 14:52:20.416 ERROR grpchandler/job_handler.go:82 Send job event error: rpc error: code = Unavailable desc = transport is closing

2024-09-12 14:52:20.416 INFO resources/kusciajob.go:91 update kuscia job yyqt

2024-09-12 14:52:20.416 ERROR grpchandler/job_handler.go:82 Send job event error: rpc error: code = Unavailable desc = transport is closing

2024-09-12 14:52:20.416 ERROR grpchandler/job_handler.go:82 Send job event error: rpc error: code = Unavailable desc = transport is closing

2024-09-12 14:52:20.417 ERROR grpchandler/job_handler.go:82 Send job event error: rpc error: code = Unavailable desc = transport is closing

2024-09-12 14:52:20.417 ERROR grpchandler/job_handler.go:82 Send job event error: rpc error: code = Unavailable desc = transport is closing

2024-09-12 14:52:20.430 INFO queue/queue.go:124 Finish processing item: queue id[kuscia-job-controller], key[cross-domain/yyqt] (14.644165ms)

2024-09-12 14:52:20.432 ERROR grpchandler/job_handler.go:82 Send job event error: rpc error: code = Unavailable desc = transport is closing

2024-09-12 14:52:20.432 INFO resources/kusciajob.go:121 Start updating kuscia job "yyqt" status

2024-09-12 14:52:20.433 ERROR grpchandler/job_handler.go:82 Send job event error: rpc error: code = Unavailable desc = transport is closing

2024-09-12 14:52:20.433 ERROR grpchandler/job_handler.go:82 Send job event error: rpc error: code = Unavailable desc = transport is closing

2024-09-12 14:52:20.433 ERROR grpchandler/job_handler.go:82 Send job event error: rpc error: code = Unavailable desc = transport is closing

2024-09-12 14:52:20.433 ERROR grpchandler/job_handler.go:82 Send job event error: rpc error: code = Unavailable desc = transport is closing

2024-09-12 14:52:20.433 ERROR grpchandler/job_handler.go:82 Send job event error: rpc error: code = Unavailable desc = transport is closing

2024-09-12 14:52:20.434 ERROR grpchandler/job_handler.go:82 Send job event error: rpc error: code = Unavailable desc = transport is closing

2024-09-12 14:52:20.451 INFO resources/kusciajob.go:125 Finish updating kuscia job "yyqt" status

2024-09-12 14:52:20.451 INFO kusciajob/controller.go:304 Finished syncing KusciaJob "cross-domain/yyqt" (19.256398ms)

2024-09-12 14:52:20.452 INFO queue/queue.go:124 Finish processing item: queue id[kuscia-job-controller], key[cross-domain/yyqt] (19.331887ms)

2024-09-12 14:52:20.453 ERROR grpchandler/job_handler.go:82 Send job event error: rpc error: code = Unavailable desc = transport is closing

2024-09-12 14:52:20.455 INFO resources/kusciajob.go:121 Start updating kuscia job "yyqt" status

2024-09-12 14:52:20.457 ERROR grpchandler/job_handler.go:82 Send job event error: rpc error: code = Unavailable desc = transport is closing

2024-09-12 14:52:20.458 ERROR grpchandler/job_handler.go:82 Send job event error: rpc error: code = Unavailable desc = transport is closing

2024-09-12 14:52:20.466 ERROR grpchandler/job_handler.go:82 Send job event error: rpc error: code = Unavailable desc = transport is closing

2024-09-12 14:52:20.467 ERROR grpchandler/job_handler.go:82 Send job event error: rpc error: code = Unavailable desc = transport is closing

2024-09-12 14:52:20.467 ERROR grpchandler/job_handler.go:82 Send job event error: rpc error: code = Unavailable desc = transport is closing

2024-09-12 14:52:20.468 INFO resources/kusciajob.go:125 Finish updating kuscia job "yyqt" status

2024-09-12 14:52:20.468 INFO kusciajob/controller.go:304 Finished syncing KusciaJob "cross-domain/yyqt" (12.966191ms)

2024-09-12 14:52:20.468 INFO queue/queue.go:124 Finish processing item: queue id[kuscia-job-controller], key[cross-domain/yyqt] (13.011873ms)

2024-09-12 14:52:20.468 ERROR grpchandler/job_handler.go:82 Send job event error: rpc error: code = Unavailable desc = transport is closing

2024-09-12 14:52:20.471 ERROR grpchandler/job_handler.go:82 Send job event error: rpc error: code = Unavailable desc = transport is closing

2024-09-12 14:52:20.471 ERROR grpchandler/job_handler.go:82 Send job event error: rpc error: code = Unavailable desc = transport is closing

2024-09-12 14:52:20.471 ERROR grpchandler/job_handler.go:82 Send job event error: rpc error: code = Unavailable desc = transport is closing

2024-09-12 14:52:20.471 ERROR grpchandler/job_handler.go:82 Send job event error: rpc error: code = Unavailable desc = transport is closing

2024-09-12 14:52:20.471 ERROR grpchandler/job_handler.go:82 Send job event error: rpc error: code = Unavailable desc = transport is closing

2024-09-12 14:52:20.471 ERROR grpchandler/job_handler.go:82 Send job event error: rpc error: code = Unavailable desc = transport is closing

2024-09-12 14:52:20.471 ERROR grpchandler/job_handler.go:82 Send job event error: rpc error: code = Unavailable desc = transport is closing

2024-09-12 14:52:20.473 WARN handler/scheduler.go:1166 Can not get appImage secretflow-image. error: appimages.kuscia.secretflow "secretflow-image" not found

2024-09-12 14:52:20.473 WARN handler/scheduler.go:1116 Can not get suitable deployTemplate. err: appimages.kuscia.secretflow "secretflow-image" not found

2024-09-12 14:52:20.477 WARN handler/scheduler.go:1166 Can not get appImage secretflow-image. error: appimages.kuscia.secretflow "secretflow-image" not found

2024-09-12 14:52:20.477 WARN handler/scheduler.go:1116 Can not get suitable deployTemplate. err: appimages.kuscia.secretflow "secretflow-image" not found

2024-09-12 14:52:20.477 INFO handler/running.go:97 Create kuscia tasks: yyqt-ubryppxk-node-3

2024-09-12 14:52:20.486 INFO resources/kusciajob.go:121 Start updating kuscia job "yyqt" status

2024-09-12 14:52:20.494 ERROR kusciatask/controller.go:493 Failed to process object: error handling "cross-domain/yyqt-ubryppxk-node-3", failed to process kusciaTask "cross-domain/yyqt-ubryppxk-node-3", failed to build domain bob kit info, failed to get appImage "secretflow-image" from cache, appimage.kuscia.secretflow "secretflow-image" not found, retry

2024-09-12 14:52:20.500 INFO resources/kusciajob.go:125 Finish updating kuscia job "yyqt" status

2024-09-12 14:52:20.501 INFO kusciajob/controller.go:304 Finished syncing KusciaJob "cross-domain/yyqt" (32.499072ms)

2024-09-12 14:52:20.501 INFO queue/queue.go:124 Finish processing item: queue id[kuscia-job-controller], key[cross-domain/yyqt] (32.561314ms)

2024-09-12 14:52:20.501 INFO handler/scheduler.go:885 JobStatusPhaseFrom readyTasks={}, tasks={{taskId=yyqt-ubryppxk-node-3, dependencies=[], tolerable=false, phase=Pending},{taskId=yyqt-ubryppxk-node-4, dependencies=[yyqt-ubryppxk-node-3], tolerable=false, phase=nil}}, kusciaJobId=yyqt

2024-09-12 14:52:20.501 INFO resources/kusciajob.go:121 Start updating kuscia job "yyqt" status

2024-09-12 14:52:20.507 ERROR kusciatask/controller.go:493 Failed to process object: error handling "cross-domain/yyqt-ubryppxk-node-3", failed to process kusciaTask "cross-domain/yyqt-ubryppxk-node-3", failed to build domain bob kit info, failed to get appImage "secretflow-image" from cache, appimage.kuscia.secretflow "secretflow-image" not found, retry

2024-09-12 14:52:20.518 INFO resources/kusciajob.go:125 Finish updating kuscia job "yyqt" status

2024-09-12 14:52:20.518 INFO kusciajob/controller.go:304 Finished syncing KusciaJob "cross-domain/yyqt" (17.450155ms)

2024-09-12 14:52:20.518 INFO queue/queue.go:124 Finish processing item: queue id[kuscia-job-controller], key[cross-domain/yyqt] (17.527306ms)

2024-09-12 14:52:20.520 INFO handler/scheduler.go:885 JobStatusPhaseFrom readyTasks={}, tasks={{taskId=yyqt-ubryppxk-node-3, dependencies=[], tolerable=false, phase=Pending},{taskId=yyqt-ubryppxk-node-4, dependencies=[yyqt-ubryppxk-node-3], tolerable=false, phase=nil}}, kusciaJobId=yyqt

2024-09-12 14:52:20.520 INFO queue/queue.go:124 Finish processing item: queue id[kuscia-job-controller], key[cross-domain/yyqt] (305.532µs)

2024-09-12 14:52:20.527 ERROR kusciatask/controller.go:493 Failed to process object: error handling "cross-domain/yyqt-ubryppxk-node-3", failed to process kusciaTask "cross-domain/yyqt-ubryppxk-node-3", failed to build domain bob kit info, failed to get appImage "secretflow-image" from cache, appimage.kuscia.secretflow "secretflow-image" not found, retry

2024-09-12 14:52:20.537 ERROR grpchandler/job_handler.go:82 Send job event error: rpc error: code = Unavailable desc = transport is closing

2024-09-12 14:52:20.538 ERROR grpchandler/job_handler.go:82 Send job event error: rpc error: code = Unavailable desc = transport is closing

2024-09-12 14:52:20.542 ERROR grpchandler/job_handler.go:82 Send job event error: rpc error: code = Unavailable desc = transport is closing

2024-09-12 14:52:20.542 ERROR grpchandler/job_handler.go:82 Send job event error: rpc error: code = Unavailable desc = transport is closing

2024-09-12 14:52:20.542 ERROR grpchandler/job_handler.go:82 Send job event error: rpc error: code = Unavailable desc = transport is closing

2024-09-12 14:52:20.542 ERROR grpchandler/job_handler.go:82 Send job event error: rpc error: code = Unavailable desc = transport is closing

2024-09-12 14:52:20.543 ERROR grpchandler/job_handler.go:82 Send job event error: rpc error: code = Unavailable desc = transport is closing

2024-09-12 14:52:20.554 ERROR kusciatask/controller.go:493 Failed to process object: error handling "cross-domain/yyqt-ubryppxk-node-3", failed to process kusciaTask "cross-domain/yyqt-ubryppxk-node-3", failed to build domain bob kit info, failed to get appImage "secretflow-image" from cache, appimage.kuscia.secretflow "secretflow-image" not found, retry

2024-09-12 14:52:20.604 INFO resources/kusciatask.go:66 Start updating kuscia task "yyqt-ubryppxk-node-3" status

2024-09-12 14:52:20.620 INFO resources/kusciatask.go:70 Finish updating kuscia task "yyqt-ubryppxk-node-3" status

2024-09-12 14:52:20.620 INFO kusciatask/controller.go:579 Finish syncing KusciaTask "cross-domain/yyqt-ubryppxk-node-3" (25.048848ms)

2024-09-12 14:52:20.621 WARN handler/failed_handler.go:62 Get task resource group yyqt-ubryppxk-node-3 failed, skip setting its status to failed, taskresourcegroup.kuscia.secretflow "yyqt-ubryppxk-node-3" not found

2024-09-12 14:52:20.621 INFO resources/kusciatask.go:66 Start updating kuscia task "yyqt-ubryppxk-node-3" status

2024-09-12 14:52:20.622 INFO handler/scheduler.go:885 JobStatusPhaseFrom readyTasks={}, tasks={{taskId=yyqt-ubryppxk-node-3, dependencies=[], tolerable=false, phase=Failed},{taskId=yyqt-ubryppxk-node-4, dependencies=[yyqt-ubryppxk-node-3], tolerable=false, phase=nil}}, kusciaJobId=yyqt

2024-09-12 14:52:20.622 INFO handler/scheduler.go:910 JobStatusPhaseFrom failed readyTasks={}, tasks={{taskId=yyqt-ubryppxk-node-3, dependencies=[], tolerable=false, phase=Failed},{taskId=yyqt-ubryppxk-node-4, dependencies=[yyqt-ubryppxk-node-3], tolerable=false, phase=nil}}, kusciaJobId=yyqt

2024-09-12 14:52:20.622 INFO resources/kusciajob.go:121 Start updating kuscia job "yyqt" status

2024-09-12 14:52:20.638 INFO resources/kusciatask.go:70 Finish updating kuscia task "yyqt-ubryppxk-node-3" status

2024-09-12 14:52:20.639 INFO kusciatask/controller.go:579 Finish syncing KusciaTask "cross-domain/yyqt-ubryppxk-node-3" (17.221162ms)

2024-09-12 14:52:20.639 INFO resources/kusciajob.go:125 Finish updating kuscia job "yyqt" status

2024-09-12 14:52:20.639 INFO kusciajob/controller.go:304 Finished syncing KusciaJob "cross-domain/yyqt" (17.501169ms)

2024-09-12 14:52:20.639 INFO queue/queue.go:124 Finish processing item: queue id[kuscia-job-controller], key[cross-domain/yyqt] (17.541546ms)

2024-09-12 14:52:20.640 INFO handler/scheduler.go:885 JobStatusPhaseFrom readyTasks={}, tasks={{taskId=yyqt-ubryppxk-node-3, dependencies=[], tolerable=false, phase=Failed},{taskId=yyqt-ubryppxk-node-4, dependencies=[yyqt-ubryppxk-node-3], tolerable=false, phase=nil}}, kusciaJobId=yyqt

2024-09-12 14:52:20.641 INFO handler/scheduler.go:910 JobStatusPhaseFrom failed readyTasks={}, tasks={{taskId=yyqt-ubryppxk-node-4, dependencies=[yyqt-ubryppxk-node-3], tolerable=false, phase=nil},{taskId=yyqt-ubryppxk-node-3, dependencies=[], tolerable=false, phase=Failed}}, kusciaJobId=yyqt

2024-09-12 14:52:20.641 INFO resources/kusciajob.go:121 Start updating kuscia job "yyqt" status

2024-09-12 14:52:20.641 INFO kusciatask/controller.go:547 KusciaTask "cross-domain/yyqt-ubryppxk-node-3" was finished, skipping

2024-09-12 14:52:20.655 ERROR grpchandler/job_handler.go:82 Send job event error: rpc error: code = Unavailable desc = transport is closing

2024-09-12 14:52:20.664 ERROR grpchandler/job_handler.go:82 Send job event error: rpc error: code = Unavailable desc = transport is closing

2024-09-12 14:52:20.665 ERROR grpchandler/job_handler.go:82 Send job event error: rpc error: code = Unavailable desc = transport is closing

2024-09-12 14:52:20.665 ERROR grpchandler/job_handler.go:82 Send job event error: rpc error: code = Unavailable desc = transport is closing

2024-09-12 14:52:20.665 ERROR grpchandler/job_handler.go:82 Send job event error: rpc error: code = Unavailable desc = transport is closing

2024-09-12 14:52:20.665 INFO resources/kusciajob.go:125 Finish updating kuscia job "yyqt" status

2024-09-12 14:52:20.665 INFO kusciajob/controller.go:304 Finished syncing KusciaJob "cross-domain/yyqt" (25.254405ms)

2024-09-12 14:52:20.666 INFO queue/queue.go:124 Finish processing item: queue id[kuscia-job-controller], key[cross-domain/yyqt] (25.316298ms)

2024-09-12 14:52:20.666 INFO resources/kusciajob.go:121 Start updating kuscia job "yyqt" status

2024-09-12 14:52:20.667 ERROR grpchandler/job_handler.go:82 Send job event error: rpc error: code = Unavailable desc = transport is closing

2024-09-12 14:52:20.667 ERROR grpchandler/job_handler.go:82 Send job event error: rpc error: code = Unavailable desc = transport is closing

2024-09-12 14:52:20.680 INFO resources/kusciajob.go:125 Finish updating kuscia job "yyqt" status

2024-09-12 14:52:20.680 INFO kusciajob/controller.go:304 Finished syncing KusciaJob "cross-domain/yyqt" (14.670979ms)

2024-09-12 14:52:20.680 INFO queue/queue.go:124 Finish processing item: queue id[kuscia-job-controller], key[cross-domain/yyqt] (14.713565ms)

2024-09-12 14:52:20.683 INFO queue/queue.go:124 Finish processing item: queue id[kuscia-job-controller], key[cross-domain/yyqt] (52.728µs)

2024-09-12 14:52:20.695 ERROR grpchandler/job_handler.go:82 Send job event error: rpc error: code = Unavailable desc = transport is closing

2024-09-12 14:52:20.697 ERROR grpchandler/job_handler.go:82 Send job event error: rpc error: code = Unavailable desc = transport is closing

2024-09-12 14:52:20.703 ERROR grpchandler/job_handler.go:82 Send job event error: rpc error: code = Unavailable desc = transport is closing

2024-09-12 14:52:20.703 ERROR grpchandler/job_handler.go:82 Send job event error: rpc error: code = Unavailable desc = transport is closing

2024-09-12 14:52:20.705 ERROR grpchandler/job_handler.go:82 Send job event error: rpc error: code = Unavailable desc = transport is closing

2024-09-12 14:52:20.706 ERROR grpchandler/job_handler.go:82 Send job event error: rpc error: code = Unavailable desc = transport is closing

2024-09-12 14:52:20.706 ERROR grpchandler/job_handler.go:82 Send job event error: rpc error: code = Unavailable desc = transport is closing

2024-09-12 14:52:26.327 INFO interceptor/common.go:92 [GRPC] [GRPC /kuscia.proto.api.v1alpha1.kusciaapi.DomainRouteService/BatchQueryDomainRouteStatus] Duration: 41.771µs, StatusCode: 0, ForwardHost: [], ContextType: [], Request: {"route_keys":null}, Response: {"status":{"code":11100,"details":null,"message":"DomainRoute keys can not be empty"}}

2024-09-12 14:52:30.613 INFO queue/queue.go:176 Finish processing item: queue id[domain-route-controller], key[alice/bob-alice] (26.274µs)

2024-09-12 14:52:30.635 INFO queue/queue.go:176 Finish processing item: queue id[domain-route-controller], key[kuscia-system/bob-kuscia-system] (23.62µs)

2024-09-12 14:52:30.635 INFO controller/domain_route.go:357 Update DomainRoute kuscia-system/bob-kuscia-system revision:1118836

2024-09-12 14:52:30.635 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[bob-kuscia-system:bob] (29.473µs)

2024-09-12 14:52:30.636 INFO clusterdomainroute/domainroute.go:148 ClusterDomainRoute bob-alice update status

2024-09-12 14:52:30.636 INFO queue/queue.go:176 Finish processing item: queue id[cluster-domain-route-controller], key[alice/bob-alice] (23.003615ms)

2024-09-12 14:52:30.638 INFO queue/queue.go:176 Finish processing item: queue id[cluster-domain-route-controller], key[bob-alice] (363.962µs)

2024-09-12 14:52:30.647 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[kuscia-system/bob-kuscia-system] (12.075411ms)

2024-09-12 14:52:30.655 INFO clusterdomainroute/domainroute.go:148 ClusterDomainRoute bob-kuscia-system update status

2024-09-12 14:52:30.655 INFO queue/queue.go:176 Finish processing item: queue id[cluster-domain-route-controller], key[kuscia-system/bob-kuscia-system] (18.612649ms)

2024-09-12 14:52:30.657 INFO queue/queue.go:176 Finish processing item: queue id[cluster-domain-route-controller], key[bob-kuscia-system] (334.961µs)

2024-09-12 14:52:31.701 INFO queue/queue.go:176 Finish processing item: queue id[domain-route-controller], key[kuscia-system/alice-kuscia-system] (25.035µs)

2024-09-12 14:52:31.701 INFO controller/domain_route.go:357 Update DomainRoute kuscia-system/alice-kuscia-system revision:1118842

2024-09-12 14:52:31.701 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-poll-queue], key[alice-kuscia-system:alice] (33.983µs)

2024-09-12 14:52:31.713 INFO queue/queue.go:124 Finish processing item: queue id[domain-route-queue], key[kuscia-system/alice-kuscia-system] (12.428434ms)

2024-09-12 14:52:31.721 INFO clusterdomainroute/domainroute.go:148 ClusterDomainRoute alice-kuscia-system update status

2024-09-12 14:52:31.721 INFO queue/queue.go:176 Finish processing item: queue id[cluster-domain-route-controller], key[kuscia-system/alice-kuscia-system] (20.438651ms)

2024-09-12 14:52:31.722 INFO queue/queue.go:176 Finish processing item: queue id[domain-route-controller], key[bob/alice-bob] (35.886µs)

2024-09-12 14:52:31.724 INFO queue/queue.go:176 Finish processing item: queue id[cluster-domain-route-controller], key[alice-kuscia-system] (356.808µs)

2024-09-12 14:52:31.740 INFO clusterdomainroute/domainroute.go:148 ClusterDomainRoute alice-bob update status

2024-09-12 14:52:31.741 INFO queue/queue.go:176 Finish processing item: queue id[cluster-domain-route-controller], key[bob/alice-bob] (19.02182ms)

2024-09-12 14:52:31.743 INFO queue/queue.go:176 Finish processing item: queue id[cluster-domain-route-controller], key[alice-bob] (315.3µs)