search

secretflow / kuscia

Kuscia(Kubernetes-based Secure Collaborative InfrA) is a K8s-based privacy-preserving computing task orchestration framework.
https://www.secretflow.org.cn/docs/kuscia/latest/zh-Hans
Apache License 2.0
72 stars 50 forks source link

kuscia 以runp方式在k8s上启动报错 #328

Open secretflowcode opened 4 months ago

secretflowcode commented 4 months ago

Issue Type

Install/Deploy

Search for existing issues similar to yours

Yes

OS Platform and Distribution

centos 7

Kuscia Version

kuscia 0.7.0b0

Deployment

k8s

deployment Version

k8s v1.22.12

App Running type

secretflow

App Running version

1.5.0b0

Configuration file used to run kuscia.

kind: Deployment
apiVersion: apps/v1
metadata:
  name: kuscia-autonomy-sp01
  namespace: ns
  annotations:
    deployment.kubernetes.io/revision: '11'
    kubesphere.io/creator: admin
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kuscia-autonomy-sp01
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: kuscia-autonomy-sp01
      annotations:
        kubesphere.io/creator: admin
        kubesphere.io/restartedAt: '2024-05-24T09:03:51.543Z'
        logging.kubesphere.io/logsidecar-config: '{}'
    spec:
      volumes:
        - name: kuscia-var-tmp
          emptyDir: {}
        - name: kuscia-config
          configMap:
            name: kuscia-autonomy-sp01-cm
            defaultMode: 420
      containers:
        - name: kuscia
          image: 'XXX/kuscia-secretflow:latest'
          command:
            - tini
            - '--'
            - kuscia
            - start
            - '-c'
            - etc/conf/kuscia.yaml
          workingDir: /home/kuscia
          env:
            - name: REGISTRY_ENDPOINT
              value: XXX/secretflow-lite-anolis8
            - name: NAMESPACE
              value: ns
            - name: TZ
              value: Asia/Shanghai
            - name: KUBERNETES_SERVICE_HOST
              value: 10.79.188.13
            - name: KUBERNETES_SERVICE_PORT
              value: '8443'
          resources: {}
          volumeMounts:
            - name: kuscia-var-tmp
              mountPath: /home/kuscia/var/tmp
            - name: kuscia-config
              mountPath: /home/kuscia/etc/conf/kuscia.yaml
              subPath: kuscia.yaml
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          imagePullPolicy: Always
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      dnsPolicy: ClusterFirst
      nodeSelector:
        kubernetes.io/hostname: sp-node
      automountServiceAccountToken: false
      securityContext: {}
      imagePullSecrets:
        - name: harbor-secret
      schedulerName: default-scheduler
      tolerations:
        - key: node-type/edge
          operator: Equal
          effect: NoSchedule
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 25%
      maxSurge: 25%
  revisionHistoryLimit: 10
  progressDeadlineSeconds: 600

What happend and What you expected to happen.

2024-05-28 15:47:22.818 INFO modules/modules.go:204 Start to init all secret backends ...
2024-05-28 15:47:22.818 WARN modules/modules.go:211 Init all secret backend but no provider found, creating default mem type
2024-05-28 15:47:22.818 INFO modules/modules.go:216 Finish Initializing all secret backends
2024-05-28 15:47:22.819 INFO tls/crypt.go:313 Generate cert with key, subject[dip-edge]
2024-05-28 15:47:22.822 INFO tls/crypt.go:313 Generate cert with key, subject[dip-edge]
2024-05-28 15:47:22.824 INFO modules/coredns.go:180 Start preparing coredns resolv.conf, root dir /home/kuscia/
2024-05-28 15:47:22.825 INFO modules/coredns.go:203 Finish preparing coredns resolv.conf
.:53
2024-05-28 15:47:22.825 INFO modules/coredns.go:169 coredns is ready
2024-05-28 15:47:22.825 INFO supervisor/supervisor.go:72 [k3s] start and watch subprocess
2024-05-28 15:47:22.825 INFO supervisor/supervisor.go:79 [k3s] try to start new process
2024-05-28 15:47:28.008 INFO modules/k3s.go:369 apply /home/kuscia/crds/v1alpha1/kuscia.secretflow_kusciabetajobs.yaml
2024-05-28 15:47:28.055 INFO modules/k3s.go:369 apply /home/kuscia/crds/v1alpha1/kuscia.secretflow_domains.yaml
2024-05-28 15:47:28.055 INFO modules/k3s.go:369 apply /home/kuscia/crds/v1alpha1/kuscia.secretflow_clusterdomainroutes.yaml
2024-05-28 15:47:28.086 INFO modules/k3s.go:369 apply /home/kuscia/crds/v1alpha1/kuscia.secretflow_domainappimages.yaml
2024-05-28 15:47:28.124 INFO modules/k3s.go:369 apply /home/kuscia/crds/v1alpha1/kuscia.secretflow_taskresources.yaml
2024-05-28 15:47:28.136 INFO modules/k3s.go:369 apply /home/kuscia/crds/v1alpha1/kuscia.secretflow_domaindatas.yaml
2024-05-28 15:47:28.257 INFO modules/k3s.go:369 apply /home/kuscia/crds/v1alpha1/kuscia.secretflow_appimages.yaml
2024-05-28 15:47:28.312 INFO modules/k3s.go:369 apply /home/kuscia/crds/v1alpha1/kuscia.secretflow_taskresourcegroups.yaml
2024-05-28 15:47:28.312 INFO modules/k3s.go:369 apply /home/kuscia/crds/v1alpha1/kuscia.secretflow_interopconfigs.yaml
2024-05-28 15:47:28.316 INFO modules/k3s.go:369 apply /home/kuscia/crds/v1alpha1/kuscia.secretflow_domaindatasources.yaml
2024-05-28 15:47:28.339 INFO modules/k3s.go:369 apply /home/kuscia/crds/v1alpha1/kuscia.secretflow_kusciabetatasks.yaml
2024-05-28 15:47:28.368 INFO modules/k3s.go:369 apply /home/kuscia/crds/v1alpha1/kuscia.secretflow_kusciajobsummaries.yaml
2024-05-28 15:47:28.407 INFO modules/k3s.go:369 apply /home/kuscia/crds/v1alpha1/kuscia.secretflow_domaindatagrants.yaml
2024-05-28 15:47:28.432 INFO modules/k3s.go:369 apply /home/kuscia/crds/v1alpha1/kuscia.secretflow_gateways.yaml
2024-05-28 15:47:28.457 INFO modules/k3s.go:369 apply /home/kuscia/crds/v1alpha1/kuscia.secretflow_kusciatasksummaries.yaml
2024-05-28 15:47:28.462 INFO modules/k3s.go:369 apply /home/kuscia/crds/v1alpha1/kuscia.secretflow_kusciadeploymentsummaries.yaml
2024-05-28 15:47:28.476 INFO modules/k3s.go:369 apply /home/kuscia/crds/v1alpha1/kuscia.secretflow_kusciabetadeployments.yaml
2024-05-28 15:47:28.479 INFO modules/k3s.go:369 apply /home/kuscia/crds/v1alpha1/kuscia.secretflow_domainroutes.yaml
The CustomResourceDefinition "kusciajobs.kuscia.secretflow" is invalid: spec.scope: Invalid value: "Cluster": field is immutable
2024-05-28 15:47:28.488 FATAL modules/k3s.go:366 apply /home/kuscia/crds/v1alpha1/kuscia.secretflow_kusciajobs.yaml err:exit status 1

Kuscia log output.

2024-05-28 15:47:22.818 INFO modules/modules.go:204 Start to init all secret backends ...
2024-05-28 15:47:22.818 WARN modules/modules.go:211 Init all secret backend but no provider found, creating default mem type
2024-05-28 15:47:22.818 INFO modules/modules.go:216 Finish Initializing all secret backends
2024-05-28 15:47:22.819 INFO tls/crypt.go:313 Generate cert with key, subject[dip-edge]
2024-05-28 15:47:22.822 INFO tls/crypt.go:313 Generate cert with key, subject[dip-edge]
2024-05-28 15:47:22.824 INFO modules/coredns.go:180 Start preparing coredns resolv.conf, root dir /home/kuscia/
2024-05-28 15:47:22.825 INFO modules/coredns.go:203 Finish preparing coredns resolv.conf
.:53
2024-05-28 15:47:22.825 INFO modules/coredns.go:169 coredns is ready
2024-05-28 15:47:22.825 INFO supervisor/supervisor.go:72 [k3s] start and watch subprocess
2024-05-28 15:47:22.825 INFO supervisor/supervisor.go:79 [k3s] try to start new process
2024-05-28 15:47:28.008 INFO modules/k3s.go:369 apply /home/kuscia/crds/v1alpha1/kuscia.secretflow_kusciabetajobs.yaml
2024-05-28 15:47:28.055 INFO modules/k3s.go:369 apply /home/kuscia/crds/v1alpha1/kuscia.secretflow_domains.yaml
2024-05-28 15:47:28.055 INFO modules/k3s.go:369 apply /home/kuscia/crds/v1alpha1/kuscia.secretflow_clusterdomainroutes.yaml
2024-05-28 15:47:28.086 INFO modules/k3s.go:369 apply /home/kuscia/crds/v1alpha1/kuscia.secretflow_domainappimages.yaml
2024-05-28 15:47:28.124 INFO modules/k3s.go:369 apply /home/kuscia/crds/v1alpha1/kuscia.secretflow_taskresources.yaml
2024-05-28 15:47:28.136 INFO modules/k3s.go:369 apply /home/kuscia/crds/v1alpha1/kuscia.secretflow_domaindatas.yaml
2024-05-28 15:47:28.257 INFO modules/k3s.go:369 apply /home/kuscia/crds/v1alpha1/kuscia.secretflow_appimages.yaml
2024-05-28 15:47:28.312 INFO modules/k3s.go:369 apply /home/kuscia/crds/v1alpha1/kuscia.secretflow_taskresourcegroups.yaml
2024-05-28 15:47:28.312 INFO modules/k3s.go:369 apply /home/kuscia/crds/v1alpha1/kuscia.secretflow_interopconfigs.yaml
2024-05-28 15:47:28.316 INFO modules/k3s.go:369 apply /home/kuscia/crds/v1alpha1/kuscia.secretflow_domaindatasources.yaml
2024-05-28 15:47:28.339 INFO modules/k3s.go:369 apply /home/kuscia/crds/v1alpha1/kuscia.secretflow_kusciabetatasks.yaml
2024-05-28 15:47:28.368 INFO modules/k3s.go:369 apply /home/kuscia/crds/v1alpha1/kuscia.secretflow_kusciajobsummaries.yaml
2024-05-28 15:47:28.407 INFO modules/k3s.go:369 apply /home/kuscia/crds/v1alpha1/kuscia.secretflow_domaindatagrants.yaml
2024-05-28 15:47:28.432 INFO modules/k3s.go:369 apply /home/kuscia/crds/v1alpha1/kuscia.secretflow_gateways.yaml
2024-05-28 15:47:28.457 INFO modules/k3s.go:369 apply /home/kuscia/crds/v1alpha1/kuscia.secretflow_kusciatasksummaries.yaml
2024-05-28 15:47:28.462 INFO modules/k3s.go:369 apply /home/kuscia/crds/v1alpha1/kuscia.secretflow_kusciadeploymentsummaries.yaml
2024-05-28 15:47:28.476 INFO modules/k3s.go:369 apply /home/kuscia/crds/v1alpha1/kuscia.secretflow_kusciabetadeployments.yaml
2024-05-28 15:47:28.479 INFO modules/k3s.go:369 apply /home/kuscia/crds/v1alpha1/kuscia.secretflow_domainroutes.yaml
The CustomResourceDefinition "kusciajobs.kuscia.secretflow" is invalid: spec.scope: Invalid value: "Cluster": field is immutable
2024-05-28 15:47:28.488 FATAL modules/k3s.go:366 apply /home/kuscia/crds/v1alpha1/kuscia.secretflow_kusciajobs.yaml err:exit status 1
secretflowcode commented 4 months ago

大家有没有遇到这样的问题

gshilei commented 4 months ago

Hi @secretflowcode 从你的日志中看，应该是使用的 Kuscia 镜像比较老。 请问上述模版中的image镜像是自己本地构建的还是使用kuscia官方提供的？ image: 'XXX/kuscia-secretflow:latest‘

建议使用 secretflow-registry.cn-hangzhou.cr.aliyuncs.com/secretflow/kuscia-secretflow:0.8.0b0 镜像试一下