search

karmada-io / karmada

Open, Multi-Cloud, Multi-Cluster Kubernetes Orchestration
https://karmada.io
Apache License 2.0
4.11k stars 805 forks source link

how to configure the Propagating policy in public cloud and private cloud? #3284

Open soulseen opened 1 year ago

soulseen commented 1 year ago

Please provide an in-depth description of the question you have: I am in a scenario in which I have two member clusters, member-1 is a cluster based private data center and member-2 is using the public cloud. When deploying workloads, I expect to propagate to member-1 first, and when member-1 runs out of resources, the remaining replicas will be propagated to member-2. if scaling down the deployment, it should decrease the pods in member-2 first, not only according to the cluster weight to propagate deployment.

is there any Propagation Policy that can be used in this scenario?

chaunceyjiang commented 1 year ago

When deploying workloads, I expect to propagate to member-1 first, and when member-1 runs out of resources, the remaining replicas will be propagated to member-2

You can take a look at this proposal to see if it can solve your problem.

soulseen commented 1 year ago

When deploying workloads, I expect to propagate to member-1 first, and when member-1 runs out of resources, the remaining replicas will be propagated to member-2

You can take a look at this proposal to see if it can solve your problem.

looks good, @chaunceyjiang is this proposal can split deployment into two clusters (local cluster and public cloud cluster)? not only select a feasible cluster and schedule all pods into it.

soulseen commented 1 year ago

is there anyone can help to take a look? thanks a lot.

XiShanYongYe-Chang commented 1 year ago

Hi @soulseen, the functionality in this proposal has been implemented in the release-1.5 branch, try the following configuration:

apiVersion: policy.karmada.io/v1alpha1
kind: PropagationPolicy
metadata:
  name: nginx
spec:
  resourceSelectors:
    - apiVersion: apps/v1
      kind: Deployment
      name: nginx
  placement:
    replicaScheduling:
      replicaSchedulingType: Divided
      replicaDivisionPreference: Aggregated
    clusterAffinities:
      - affinityName: private
        clusterNames:
          - member-1
      - affinityName: private-and-public
        clusterNames:
          - member1
          - member2

The above can divide replicas into clusters as few as possible. Or you can set a weight reference to divide the replicas.

soulseen commented 1 year ago

thanks @XiShanYongYe-Chang, I used this policy config:

apiVersion: policy.karmada.io/v1alpha1
kind: PropagationPolicy
metadata:
  name: nginx-1-private-cloud
spec:
  resourceSelectors:
    - apiVersion: apps/v1
      kind: Deployment
      name: nginx-1
  placement:
    replicaScheduling:
      replicaSchedulingType: Divided
      replicaDivisionPreference: Weighted
      weightPreference:
        staticWeightList:
          - targetCluster:
              clusterNames:
                - member-1
            weight: 4
          - targetCluster:
              clusterNames:
                - member-2
            weight: 2
    clusterAffinities:
      - affinityName: private
        clusterNames:
          - member-1
          - member-2
      - affinityName: private-and-public
        clusterNames:
          - member-1
          - member-2

and all 6 replicas are deployed into member-1, but what I expect is to divide the replicas like member-1=4 and member-2=2 Is there something wrong with my configuration?

RainbowMango commented 1 year ago

I just reproduced it on my side with a similar policy as follows:

# cat policy.yaml 
apiVersion: policy.karmada.io/v1alpha1
kind: PropagationPolicy
metadata:
  name: nginx-1-private-cloud
spec:
  resourceSelectors:
    - apiVersion: apps/v1
      kind: Deployment
      name: nginx
  placement:
    replicaScheduling:
      replicaSchedulingType: Divided
      replicaDivisionPreference: Weighted
      weightPreference:
        staticWeightList:
          - targetCluster:
              clusterNames:
                - member1
            weight: 4
          - targetCluster:
              clusterNames:
                - member2
            weight: 2
    clusterAffinities:
      - affinityName: private
        clusterNames:
          - member1
          - member2
      - affinityName: private-and-public
        clusterNames:
          - member1
          - member2

And then, apply a Deployment as follows:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  replicas: 6
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - image: nginx
        name: nginx

The scheduling result is(works as expected) :

spec:
  clusters:
  - name: member1
    replicas: 4
  - name: member2
    replicas: 2
RainbowMango commented 1 year ago

and all 6 replicas are deployed into member-1, but what I expect is to divide the replicas like member-1=4 and member-2=2 Is there something wrong with my configuration?

@soulseen Whether the resources in member-2 are sufficient?

soulseen commented 1 year ago

@RainbowMango I don't think so, because I if removed member-1 in propagation policy, all replicas are scheduled into member-2, but imagePull failed, and I tried again, the result is the same:

spec:
  clusters:
  - name: member-1
    replicas: 6
  - name: member-2
    replicas: 0
soulseen commented 1 year ago

Environment:

RainbowMango commented 1 year ago

It will make things easier if this issue could be reproduced in an environment launched by hack/local-up-karmada.sh. Can you help to try it?

Or could you please share the logs of karmada-scheduler? As well as the yaml of nginx-1?

soulseen commented 1 year ago

@RainbowMango Ohh, I tried it again, and it success. the result is:

spec:
  clusters:
  - name: member1
    replicas: 4
  - name: member2
    replicas: 2

but I don't change any propagation policy config, just redeploy nginx.

soulseen commented 1 year ago

karmada-scheduler log(k8s-danlu-vm-dev-new=member-1, qa-live=member-2):

I0320 08:40:41.005138       1 generic_scheduler.go:84] Feasible clusters scores: [{qa-live 100} {k8s-danlu-vm-dev-new 0}]
I0320 08:40:41.005183       1 general.go:61] cluster qa-live has max available replicas: 241 according to cluster resource models
I0320 08:40:41.005225       1 general.go:61] cluster k8s-danlu-vm-dev-new has max available replicas: 19 according to cluster resource models
E0320 08:40:41.005329       1 util.go:53] Max cluster available replicas error: [cluster k8s-danlu-vm-dev-new does not exist in estimator cache, cluster qa-live does not exist in estimator cache]
I0320 08:40:41.005360       1 util.go:74] Target cluster: [{qa-live 241} {k8s-danlu-vm-dev-new 19}]
I0320 08:40:41.005387       1 select_clusters.go:19] Select all clusters
I0320 08:40:41.005404       1 generic_scheduler.go:90] Selected clusters: [qa-live k8s-danlu-vm-dev-new]
I0320 08:40:41.005434       1 scheduler.go:492] ResourceBinding(440-zxy-test-8-6214/nginx-1-deployment) scheduled to clusters [{k8s-danlu-vm-dev-new 4} {qa-live 2}]
I0320 08:40:41.022343       1 scheduler.go:529] Patch schedule to ResourceBinding(440-zxy-test-8-6214/nginx-1-deployment) succeed
I0320 08:40:41.022367       1 scheduler.go:735] Begin to patch status with affinityName(private) to ResourceBinding(440-zxy-test-8-6214/nginx-1-deployment).
I0320 08:40:41.022437       1 scheduler.go:497] "End scheduling resourceBinding with ClusterAffinities" resourceBinding="440-zxy-test-8-6214/nginx-1-deployment"
I0320 08:40:41.022461       1 scheduler.go:721] Begin to patch status condition to ResourceBinding(440-zxy-test-8-6214/nginx-1-deployment)
I0320 08:40:41.022573       1 event.go:294] "Event occurred" object="440-zxy-test-8-6214/nginx-1-deployment" fieldPath="" kind="ResourceBinding" apiVersion="work.karmada.io/v1alpha2" type="Normal" reason="ScheduleBindingSucceed" message="Binding has been scheduled"
I0320 08:40:41.022615       1 event.go:294] "Event occurred" object="440-zxy-test-8-6214/nginx-1" fieldPath="" kind="Deployment" apiVersion="apps/v1" type="Normal" reason="ScheduleBindingSucceed" message="Binding has been scheduled"
I0320 08:40:41.031795       1 scheduler.go:759] Patch schedule status to ResourceBinding(440-zxy-test-8-6214/nginx-1-deployment) succeed
I0320 08:40:41.031899       1 scheduler.go:337] Don't need to schedule ResourceBinding(440-zxy-test-8-6214/nginx-1-deployment)
I0320 08:40:41.032254       1 scheduler.go:337] Don't need to schedule ResourceBinding(440-zxy-test-8-6214/nginx-1-deployment)
I0320 08:40:41.079077       1 scheduler.go:337] Don't need to schedule ResourceBinding(440-zxy-test-8-6214/nginx-1-deployment)
I0320 08:40:41.170195       1 scheduler.go:337] Don't need to schedule ResourceBinding(440-zxy-test-8-6214/nginx-1-deployment)
I0320 08:40:41.235731       1 scheduler.go:337] Don't need to schedule ResourceBinding(440-zxy-test-8-6214/nginx-1-deployment)
I0320 08:40:42.748097       1 scheduler.go:337] Don't need to schedule ResourceBinding(440-zxy-test-8-6214/nginx-1-deployment)
I0320 08:40:43.786142       1 scheduler.go:337] Don't need to schedule ResourceBinding(440-zxy-test-8-6214/nginx-1-deployment)
I0320 08:40:45.762113       1 scheduler.go:337] Don't need to schedule ResourceBinding(440-zxy-test-8-6214/nginx-1-deployment)
I0320 08:40:46.766773       1 scheduler.go:337] Don't need to schedule ResourceBinding(440-zxy-test-8-6214/nginx-1-deployment)
I0320 08:40:49.235125       1 event_handler.go:161] Update event for cluster k8s-danlu-vm-dev-new
I0320 08:40:49.238299       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-k8s-danlu-vm-dev-new:10352) of cluster(k8s-danlu-vm-dev-new).
I0320 08:40:51.359628       1 event_handler.go:161] Update event for cluster qa-live
E0320 08:40:54.239482       1 cache.go:97] Failed to dial cluster(k8s-danlu-vm-dev-new): dial karmada-scheduler-estimator-k8s-danlu-vm-dev-new:10352 error: context deadline exceeded.
I0320 08:40:54.241971       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-qa-live:10352) of cluster(qa-live).
E0320 08:40:59.244844       1 cache.go:97] Failed to dial cluster(qa-live): dial karmada-scheduler-estimator-qa-live:10352 error: context deadline exceeded.
I0320 08:40:59.275416       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-k8s-danlu-vm-dev-new:10352) of cluster(k8s-danlu-vm-dev-new).
E0320 08:41:04.276820       1 cache.go:97] Failed to dial cluster(k8s-danlu-vm-dev-new): dial karmada-scheduler-estimator-k8s-danlu-vm-dev-new:10352 error: context deadline exceeded.
I0320 08:41:04.280997       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-qa-live:10352) of cluster(qa-live).
E0320 08:41:09.281645       1 cache.go:97] Failed to dial cluster(qa-live): dial karmada-scheduler-estimator-qa-live:10352 error: context deadline exceeded.
I0320 08:41:09.291204       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-k8s-danlu-vm-dev-new:10352) of cluster(k8s-danlu-vm-dev-new).
E0320 08:41:14.291663       1 cache.go:97] Failed to dial cluster(k8s-danlu-vm-dev-new): dial karmada-scheduler-estimator-k8s-danlu-vm-dev-new:10352 error: context deadline exceeded.
I0320 08:41:14.294525       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-qa-live:10352) of cluster(qa-live).
E0320 08:41:19.294930       1 cache.go:97] Failed to dial cluster(qa-live): dial karmada-scheduler-estimator-qa-live:10352 error: context deadline exceeded.
I0320 08:41:19.298152       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-k8s-danlu-vm-dev-new:10352) of cluster(k8s-danlu-vm-dev-new).
E0320 08:41:24.299250       1 cache.go:97] Failed to dial cluster(k8s-danlu-vm-dev-new): dial karmada-scheduler-estimator-k8s-danlu-vm-dev-new:10352 error: context deadline exceeded.
I0320 08:41:24.302850       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-qa-live:10352) of cluster(qa-live).
I0320 08:41:26.422989       1 reflector.go:559] pkg/generated/informers/externalversions/factory.go:121: Watch close - *v1alpha2.ClusterResourceBinding total 8 items received
E0320 08:41:29.303995       1 cache.go:97] Failed to dial cluster(qa-live): dial karmada-scheduler-estimator-qa-live:10352 error: context deadline exceeded.
I0320 08:41:29.309527       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-k8s-danlu-vm-dev-new:10352) of cluster(k8s-danlu-vm-dev-new).
E0320 08:41:34.309932       1 cache.go:97] Failed to dial cluster(k8s-danlu-vm-dev-new): dial karmada-scheduler-estimator-k8s-danlu-vm-dev-new:10352 error: context deadline exceeded.
I0320 08:41:34.313424       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-qa-live:10352) of cluster(qa-live).
E0320 08:41:39.313612       1 cache.go:97] Failed to dial cluster(qa-live): dial karmada-scheduler-estimator-qa-live:10352 error: context deadline exceeded.
I0320 08:41:39.317607       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-k8s-danlu-vm-dev-new:10352) of cluster(k8s-danlu-vm-dev-new).
E0320 08:41:44.318815       1 cache.go:97] Failed to dial cluster(k8s-danlu-vm-dev-new): dial karmada-scheduler-estimator-k8s-danlu-vm-dev-new:10352 error: context deadline exceeded.
I0320 08:41:44.322082       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-qa-live:10352) of cluster(qa-live).
E0320 08:41:49.322600       1 cache.go:97] Failed to dial cluster(qa-live): dial karmada-scheduler-estimator-qa-live:10352 error: context deadline exceeded.
I0320 08:41:49.325659       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-k8s-danlu-vm-dev-new:10352) of cluster(k8s-danlu-vm-dev-new).
I0320 08:41:53.074273       1 reflector.go:559] pkg/generated/informers/externalversions/factory.go:121: Watch close - *v1alpha1.Cluster total 21 items received
E0320 08:41:54.326690       1 cache.go:97] Failed to dial cluster(k8s-danlu-vm-dev-new): dial karmada-scheduler-estimator-k8s-danlu-vm-dev-new:10352 error: context deadline exceeded.
I0320 08:41:54.329775       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-qa-live:10352) of cluster(qa-live).
E0320 08:41:59.330880       1 cache.go:97] Failed to dial cluster(qa-live): dial karmada-scheduler-estimator-qa-live:10352 error: context deadline exceeded.
I0320 08:41:59.335154       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-k8s-danlu-vm-dev-new:10352) of cluster(k8s-danlu-vm-dev-new).
I0320 08:42:04.269713       1 scheduler.go:337] Don't need to schedule ResourceBinding(440-zxy-test-8-6214/nginx-1-deployment)
E0320 08:42:04.336137       1 cache.go:97] Failed to dial cluster(k8s-danlu-vm-dev-new): dial karmada-scheduler-estimator-k8s-danlu-vm-dev-new:10352 error: context deadline exceeded.
I0320 08:42:04.339906       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-qa-live:10352) of cluster(qa-live).
I0320 08:42:04.513074       1 reflector.go:559] pkg/generated/informers/externalversions/factory.go:121: Watch close - *v1alpha2.ResourceBinding total 14 items received
E0320 08:42:09.340151       1 cache.go:97] Failed to dial cluster(qa-live): dial karmada-scheduler-estimator-qa-live:10352 error: context deadline exceeded.
I0320 08:42:09.342979       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-k8s-danlu-vm-dev-new:10352) of cluster(k8s-danlu-vm-dev-new).
I0320 08:42:11.682990       1 event_handler.go:161] Update event for cluster qa-live
E0320 08:42:14.343708       1 cache.go:97] Failed to dial cluster(k8s-danlu-vm-dev-new): dial karmada-scheduler-estimator-k8s-danlu-vm-dev-new:10352 error: context deadline exceeded.
I0320 08:42:14.347148       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-qa-live:10352) of cluster(qa-live).
E0320 08:42:19.348091       1 cache.go:97] Failed to dial cluster(qa-live): dial karmada-scheduler-estimator-qa-live:10352 error: context deadline exceeded.
I0320 08:42:19.351932       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-k8s-danlu-vm-dev-new:10352) of cluster(k8s-danlu-vm-dev-new).
E0320 08:42:24.353686       1 cache.go:97] Failed to dial cluster(k8s-danlu-vm-dev-new): dial karmada-scheduler-estimator-k8s-danlu-vm-dev-new:10352 error: context deadline exceeded.
I0320 08:42:24.357050       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-qa-live:10352) of cluster(qa-live).
E0320 08:42:29.357634       1 cache.go:97] Failed to dial cluster(qa-live): dial karmada-scheduler-estimator-qa-live:10352 error: context deadline exceeded.
I0320 08:42:29.360245       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-k8s-danlu-vm-dev-new:10352) of cluster(k8s-danlu-vm-dev-new).
E0320 08:42:34.361298       1 cache.go:97] Failed to dial cluster(k8s-danlu-vm-dev-new): dial karmada-scheduler-estimator-k8s-danlu-vm-dev-new:10352 error: context deadline exceeded.
I0320 08:42:34.364209       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-qa-live:10352) of cluster(qa-live).
E0320 08:42:39.365046       1 cache.go:97] Failed to dial cluster(qa-live): dial karmada-scheduler-estimator-qa-live:10352 error: context deadline exceeded.
I0320 08:42:39.486366       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-k8s-danlu-vm-dev-new:10352) of cluster(k8s-danlu-vm-dev-new).
E0320 08:42:44.487477       1 cache.go:97] Failed to dial cluster(k8s-danlu-vm-dev-new): dial karmada-scheduler-estimator-k8s-danlu-vm-dev-new:10352 error: context deadline exceeded.
I0320 08:42:44.492783       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-qa-live:10352) of cluster(qa-live).
E0320 08:42:49.493227       1 cache.go:97] Failed to dial cluster(qa-live): dial karmada-scheduler-estimator-qa-live:10352 error: context deadline exceeded.
I0320 08:42:54.731801       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-k8s-danlu-vm-dev-new:10352) of cluster(k8s-danlu-vm-dev-new).
I0320 08:42:59.440834       1 event_handler.go:161] Update event for cluster k8s-danlu-vm-dev-new
E0320 08:42:59.733033       1 cache.go:97] Failed to dial cluster(k8s-danlu-vm-dev-new): dial karmada-scheduler-estimator-k8s-danlu-vm-dev-new:10352 error: context deadline exceeded.
I0320 08:42:59.736741       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-k8s-danlu-vm-dev-new:10352) of cluster(k8s-danlu-vm-dev-new).
E0320 08:43:04.737158       1 cache.go:97] Failed to dial cluster(k8s-danlu-vm-dev-new): dial karmada-scheduler-estimator-k8s-danlu-vm-dev-new:10352 error: context deadline exceeded.
I0320 08:43:04.740734       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-qa-live:10352) of cluster(qa-live).
E0320 08:43:09.742026       1 cache.go:97] Failed to dial cluster(qa-live): dial karmada-scheduler-estimator-qa-live:10352 error: context deadline exceeded.
I0320 08:43:20.216385       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-k8s-danlu-vm-dev-new:10352) of cluster(k8s-danlu-vm-dev-new).
E0320 08:43:25.216794       1 cache.go:97] Failed to dial cluster(k8s-danlu-vm-dev-new): dial karmada-scheduler-estimator-k8s-danlu-vm-dev-new:10352 error: context deadline exceeded.
I0320 08:43:30.226105       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-qa-live:10352) of cluster(qa-live).
I0320 08:43:31.986259       1 event_handler.go:161] Update event for cluster qa-live
E0320 08:43:35.226601       1 cache.go:97] Failed to dial cluster(qa-live): dial karmada-scheduler-estimator-qa-live:10352 error: context deadline exceeded.
I0320 08:43:35.229717       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-qa-live:10352) of cluster(qa-live).
E0320 08:43:40.230317       1 cache.go:97] Failed to dial cluster(qa-live): dial karmada-scheduler-estimator-qa-live:10352 error: context deadline exceeded.
I0320 08:44:16.189875       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-qa-live:10352) of cluster(qa-live).
E0320 08:44:21.190379       1 cache.go:97] Failed to dial cluster(qa-live): dial karmada-scheduler-estimator-qa-live:10352 error: context deadline exceeded.
I0320 08:44:21.190420       1 worker.go:127] Dropping resource "qa-live" out of the queue: dial karmada-scheduler-estimator-qa-live:10352 error: context deadline exceeded
I0320 08:44:22.176826       1 event_handler.go:161] Update event for cluster qa-live
I0320 08:44:22.180272       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-qa-live:10352) of cluster(qa-live).
E0320 08:44:27.181271       1 cache.go:97] Failed to dial cluster(qa-live): dial karmada-scheduler-estimator-qa-live:10352 error: context deadline exceeded.
I0320 08:44:27.189759       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-qa-live:10352) of cluster(qa-live).
E0320 08:44:32.190933       1 cache.go:97] Failed to dial cluster(qa-live): dial karmada-scheduler-estimator-qa-live:10352 error: context deadline exceeded.
I0320 08:44:32.204542       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-qa-live:10352) of cluster(qa-live).
E0320 08:44:37.205829       1 cache.go:97] Failed to dial cluster(qa-live): dial karmada-scheduler-estimator-qa-live:10352 error: context deadline exceeded.
I0320 08:44:37.230166       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-qa-live:10352) of cluster(qa-live).
E0320 08:44:42.230925       1 cache.go:97] Failed to dial cluster(qa-live): dial karmada-scheduler-estimator-qa-live:10352 error: context deadline exceeded.
I0320 08:44:42.275078       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-qa-live:10352) of cluster(qa-live).
E0320 08:44:47.276015       1 cache.go:97] Failed to dial cluster(qa-live): dial karmada-scheduler-estimator-qa-live:10352 error: context deadline exceeded.
I0320 08:44:47.279543       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-k8s-danlu-vm-dev-new:10352) of cluster(k8s-danlu-vm-dev-new).
E0320 08:44:52.279812       1 cache.go:97] Failed to dial cluster(k8s-danlu-vm-dev-new): dial karmada-scheduler-estimator-k8s-danlu-vm-dev-new:10352 error: context deadline exceeded.
I0320 08:44:52.279843       1 worker.go:127] Dropping resource "k8s-danlu-vm-dev-new" out of the queue: dial karmada-scheduler-estimator-k8s-danlu-vm-dev-new:10352 error: context deadline exceeded
I0320 08:44:52.283348       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-qa-live:10352) of cluster(qa-live).
E0320 08:44:57.284118       1 cache.go:97] Failed to dial cluster(qa-live): dial karmada-scheduler-estimator-qa-live:10352 error: context deadline exceeded.
I0320 08:44:57.448653       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-qa-live:10352) of cluster(qa-live).
E0320 08:45:02.449580       1 cache.go:97] Failed to dial cluster(qa-live): dial karmada-scheduler-estimator-qa-live:10352 error: context deadline exceeded.
I0320 08:45:02.773767       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-qa-live:10352) of cluster(qa-live).
E0320 08:45:07.774228       1 cache.go:97] Failed to dial cluster(qa-live): dial karmada-scheduler-estimator-qa-live:10352 error: context deadline exceeded.
I0320 08:45:08.418204       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-qa-live:10352) of cluster(qa-live).
E0320 08:45:13.418833       1 cache.go:97] Failed to dial cluster(qa-live): dial karmada-scheduler-estimator-qa-live:10352 error: context deadline exceeded.
I0320 08:45:14.702945       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-qa-live:10352) of cluster(qa-live).
E0320 08:45:19.703640       1 cache.go:97] Failed to dial cluster(qa-live): dial karmada-scheduler-estimator-qa-live:10352 error: context deadline exceeded.
I0320 08:45:22.268005       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-qa-live:10352) of cluster(qa-live).
E0320 08:45:27.268490       1 cache.go:97] Failed to dial cluster(qa-live): dial karmada-scheduler-estimator-qa-live:10352 error: context deadline exceeded.
I0320 08:45:32.391887       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-qa-live:10352) of cluster(qa-live).
E0320 08:45:37.392175       1 cache.go:97] Failed to dial cluster(qa-live): dial karmada-scheduler-estimator-qa-live:10352 error: context deadline exceeded.
I0320 08:45:47.635565       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-qa-live:10352) of cluster(qa-live).
E0320 08:45:52.636486       1 cache.go:97] Failed to dial cluster(qa-live): dial karmada-scheduler-estimator-qa-live:10352 error: context deadline exceeded.
I0320 08:46:12.583727       1 event_handler.go:161] Update event for cluster qa-live
I0320 08:46:12.586707       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-qa-live:10352) of cluster(qa-live).
I0320 08:46:15.049452       1 scheduler.go:321] Reschedule ResourceBinding(default/nginx-deployment) as replicas scaled down or scaled up
I0320 08:46:15.049500       1 scheduler.go:413] "Begin scheduling resource binding with ClusterAffinity" resourceBinding="default/nginx-deployment"
I0320 08:46:15.049643       1 generic_scheduler.go:78] Feasible clusters found: [k8s-danlu-vm-dev-new qa-live]
I0320 08:46:15.049682       1 generic_scheduler.go:149] Plugin ClusterAffinity scores on default/nginx => [{k8s-danlu-vm-dev-new 0} {qa-live 0}]
I0320 08:46:15.049703       1 generic_scheduler.go:149] Plugin ClusterLocality scores on default/nginx => [{k8s-danlu-vm-dev-new 100} {qa-live 100}]
I0320 08:46:15.049721       1 generic_scheduler.go:84] Feasible clusters scores: [{k8s-danlu-vm-dev-new 100} {qa-live 100}]
E0320 08:46:15.049822       1 util.go:53] Max cluster available replicas error: [cluster qa-live does not exist in estimator cache, cluster k8s-danlu-vm-dev-new does not exist in estimator cache]
I0320 08:46:15.049856       1 util.go:74] Target cluster: [{k8s-danlu-vm-dev-new 872} {qa-live 1751}]
I0320 08:46:15.049880       1 select_clusters.go:19] Select all clusters
I0320 08:46:15.049895       1 generic_scheduler.go:90] Selected clusters: [k8s-danlu-vm-dev-new qa-live]
I0320 08:46:15.049919       1 scheduler.go:431] ResourceBinding(default/nginx-deployment) scheduled to clusters [{k8s-danlu-vm-dev-new 4} {qa-live 2}]
I0320 08:46:15.062353       1 scheduler.go:529] Patch schedule to ResourceBinding(default/nginx-deployment) succeed
I0320 08:46:15.062396       1 scheduler.go:437] "End scheduling resource binding with ClusterAffinity" resourceBinding="default/nginx-deployment"
I0320 08:46:15.062407       1 scheduler.go:721] Begin to patch status condition to ResourceBinding(default/nginx-deployment)
I0320 08:46:15.062926       1 event.go:294] "Event occurred" object="default/nginx-deployment" fieldPath="" kind="ResourceBinding" apiVersion="work.karmada.io/v1alpha2" type="Normal" reason="ScheduleBindingSucceed" message="Binding has been scheduled"
I0320 08:46:15.062963       1 event.go:294] "Event occurred" object="default/nginx" fieldPath="" kind="Deployment" apiVersion="apps/v1" type="Normal" reason="ScheduleBindingSucceed" message="Binding has been scheduled"
I0320 08:46:15.071789       1 scheduler.go:759] Patch schedule status to ResourceBinding(default/nginx-deployment) succeed
I0320 08:46:15.071903       1 scheduler.go:337] Don't need to schedule ResourceBinding(default/nginx-deployment)
I0320 08:46:15.072627       1 scheduler.go:337] Don't need to schedule ResourceBinding(default/nginx-deployment)
I0320 08:46:15.107614       1 scheduler.go:337] Don't need to schedule ResourceBinding(default/nginx-deployment)
I0320 08:46:15.240155       1 scheduler.go:337] Don't need to schedule ResourceBinding(default/nginx-deployment)
I0320 08:46:15.278652       1 scheduler.go:337] Don't need to schedule ResourceBinding(default/nginx-deployment)
I0320 08:46:15.320245       1 scheduler.go:337] Don't need to schedule ResourceBinding(default/nginx-deployment)
E0320 08:46:17.587476       1 cache.go:97] Failed to dial cluster(qa-live): dial karmada-scheduler-estimator-qa-live:10352 error: context deadline exceeded.
I0320 08:46:17.592041       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-qa-live:10352) of cluster(qa-live).
I0320 08:46:17.781565       1 scheduler.go:337] Don't need to schedule ResourceBinding(default/nginx-deployment)
I0320 08:46:19.741638       1 event_handler.go:161] Update event for cluster k8s-danlu-vm-dev-new
E0320 08:46:22.592663       1 cache.go:97] Failed to dial cluster(qa-live): dial karmada-scheduler-estimator-qa-live:10352 error: context deadline exceeded.
I0320 08:46:22.596203       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-k8s-danlu-vm-dev-new:10352) of cluster(k8s-danlu-vm-dev-new).
I0320 08:46:22.630555       1 event_handler.go:161] Update event for cluster qa-live
E0320 08:46:27.596608       1 cache.go:97] Failed to dial cluster(k8s-danlu-vm-dev-new): dial karmada-scheduler-estimator-k8s-danlu-vm-dev-new:10352 error: context deadline exceeded.
I0320 08:46:27.600197       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-qa-live:10352) of cluster(qa-live).
E0320 08:46:32.616742       1 cache.go:97] Failed to dial cluster(qa-live): dial karmada-scheduler-estimator-qa-live:10352 error: context deadline exceeded.
I0320 08:46:32.616790       1 worker.go:127] Dropping resource "qa-live" out of the queue: dial karmada-scheduler-estimator-qa-live:10352 error: context deadline exceeded
I0320 08:46:32.620610       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-k8s-danlu-vm-dev-new:10352) of cluster(k8s-danlu-vm-dev-new).
E0320 08:46:37.621423       1 cache.go:97] Failed to dial cluster(k8s-danlu-vm-dev-new): dial karmada-scheduler-estimator-k8s-danlu-vm-dev-new:10352 error: context deadline exceeded.
I0320 08:46:37.635381       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-k8s-danlu-vm-dev-new:10352) of cluster(k8s-danlu-vm-dev-new).
E0320 08:46:42.635578       1 cache.go:97] Failed to dial cluster(k8s-danlu-vm-dev-new): dial karmada-scheduler-estimator-k8s-danlu-vm-dev-new:10352 error: context deadline exceeded.
I0320 08:46:42.660314       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-k8s-danlu-vm-dev-new:10352) of cluster(k8s-danlu-vm-dev-new).
E0320 08:46:47.660892       1 cache.go:97] Failed to dial cluster(k8s-danlu-vm-dev-new): dial karmada-scheduler-estimator-k8s-danlu-vm-dev-new:10352 error: context deadline exceeded.
I0320 08:46:47.725127       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-k8s-danlu-vm-dev-new:10352) of cluster(k8s-danlu-vm-dev-new).
E0320 08:46:52.725314       1 cache.go:97] Failed to dial cluster(k8s-danlu-vm-dev-new): dial karmada-scheduler-estimator-k8s-danlu-vm-dev-new:10352 error: context deadline exceeded.
I0320 08:46:52.809416       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-k8s-danlu-vm-dev-new:10352) of cluster(k8s-danlu-vm-dev-new).
E0320 08:46:57.810092       1 cache.go:97] Failed to dial cluster(k8s-danlu-vm-dev-new): dial karmada-scheduler-estimator-k8s-danlu-vm-dev-new:10352 error: context deadline exceeded.
I0320 08:46:57.973850       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-k8s-danlu-vm-dev-new:10352) of cluster(k8s-danlu-vm-dev-new).
E0320 08:47:02.974702       1 cache.go:97] Failed to dial cluster(k8s-danlu-vm-dev-new): dial karmada-scheduler-estimator-k8s-danlu-vm-dev-new:10352 error: context deadline exceeded.
I0320 08:47:02.978430       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-qa-live:10352) of cluster(qa-live).
E0320 08:47:07.979275       1 cache.go:97] Failed to dial cluster(qa-live): dial karmada-scheduler-estimator-qa-live:10352 error: context deadline exceeded.
I0320 08:47:07.982997       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-k8s-danlu-vm-dev-new:10352) of cluster(k8s-danlu-vm-dev-new).
I0320 08:47:12.827193       1 event_handler.go:161] Update event for cluster qa-live
E0320 08:47:12.983940       1 cache.go:97] Failed to dial cluster(k8s-danlu-vm-dev-new): dial karmada-scheduler-estimator-k8s-danlu-vm-dev-new:10352 error: context deadline exceeded.
I0320 08:47:12.987649       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-qa-live:10352) of cluster(qa-live).
E0320 08:47:17.988731       1 cache.go:97] Failed to dial cluster(qa-live): dial karmada-scheduler-estimator-qa-live:10352 error: context deadline exceeded.
I0320 08:47:17.991872       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-k8s-danlu-vm-dev-new:10352) of cluster(k8s-danlu-vm-dev-new).
I0320 08:47:22.878686       1 event_handler.go:161] Update event for cluster qa-live
E0320 08:47:22.992136       1 cache.go:97] Failed to dial cluster(k8s-danlu-vm-dev-new): dial karmada-scheduler-estimator-k8s-danlu-vm-dev-new:10352 error: context deadline exceeded.
I0320 08:47:22.995965       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-qa-live:10352) of cluster(qa-live).
E0320 08:47:27.997031       1 cache.go:97] Failed to dial cluster(qa-live): dial karmada-scheduler-estimator-qa-live:10352 error: context deadline exceeded.
I0320 08:47:28.000360       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-k8s-danlu-vm-dev-new:10352) of cluster(k8s-danlu-vm-dev-new).
E0320 08:47:33.001435       1 cache.go:97] Failed to dial cluster(k8s-danlu-vm-dev-new): dial karmada-scheduler-estimator-k8s-danlu-vm-dev-new:10352 error: context deadline exceeded.
I0320 08:47:33.004656       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-qa-live:10352) of cluster(qa-live).
E0320 08:47:38.005886       1 cache.go:97] Failed to dial cluster(qa-live): dial karmada-scheduler-estimator-qa-live:10352 error: context deadline exceeded.
I0320 08:47:38.009055       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-k8s-danlu-vm-dev-new:10352) of cluster(k8s-danlu-vm-dev-new).
E0320 08:47:43.009758       1 cache.go:97] Failed to dial cluster(k8s-danlu-vm-dev-new): dial karmada-scheduler-estimator-k8s-danlu-vm-dev-new:10352 error: context deadline exceeded.
I0320 08:47:43.013395       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-qa-live:10352) of cluster(qa-live).
E0320 08:47:48.013666       1 cache.go:97] Failed to dial cluster(qa-live): dial karmada-scheduler-estimator-qa-live:10352 error: context deadline exceeded.
I0320 08:47:48.098922       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-qa-live:10352) of cluster(qa-live).
I0320 08:47:49.077822       1 reflector.go:559] pkg/generated/informers/externalversions/factory.go:121: Watch close - *v1alpha1.Cluster total 16 items received
E0320 08:47:53.100023       1 cache.go:97] Failed to dial cluster(qa-live): dial karmada-scheduler-estimator-qa-live:10352 error: context deadline exceeded.
I0320 08:47:53.103064       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-k8s-danlu-vm-dev-new:10352) of cluster(k8s-danlu-vm-dev-new).
E0320 08:47:58.103855       1 cache.go:97] Failed to dial cluster(k8s-danlu-vm-dev-new): dial karmada-scheduler-estimator-k8s-danlu-vm-dev-new:10352 error: context deadline exceeded.
I0320 08:47:58.107131       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-qa-live:10352) of cluster(qa-live).
E0320 08:48:03.107741       1 cache.go:97] Failed to dial cluster(qa-live): dial karmada-scheduler-estimator-qa-live:10352 error: context deadline exceeded.
I0320 08:48:03.431479       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-qa-live:10352) of cluster(qa-live).
E0320 08:48:08.431980       1 cache.go:97] Failed to dial cluster(qa-live): dial karmada-scheduler-estimator-qa-live:10352 error: context deadline exceeded.
I0320 08:48:08.435478       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-k8s-danlu-vm-dev-new:10352) of cluster(k8s-danlu-vm-dev-new).
E0320 08:48:13.436302       1 cache.go:97] Failed to dial cluster(k8s-danlu-vm-dev-new): dial karmada-scheduler-estimator-k8s-danlu-vm-dev-new:10352 error: context deadline exceeded.
I0320 08:48:13.440243       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-qa-live:10352) of cluster(qa-live).
I0320 08:48:15.425428       1 reflector.go:559] pkg/generated/informers/externalversions/factory.go:121: Watch close - *v1alpha2.ClusterResourceBinding total 4 items received
E0320 08:48:18.440708       1 cache.go:97] Failed to dial cluster(qa-live): dial karmada-scheduler-estimator-qa-live:10352 error: context deadline exceeded.
I0320 08:48:19.725759       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-qa-live:10352) of cluster(qa-live).
E0320 08:48:24.726718       1 cache.go:97] Failed to dial cluster(qa-live): dial karmada-scheduler-estimator-qa-live:10352 error: context deadline exceeded.
I0320 08:48:27.295312       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-qa-live:10352) of cluster(qa-live).
E0320 08:48:32.296175       1 cache.go:97] Failed to dial cluster(qa-live): dial karmada-scheduler-estimator-qa-live:10352 error: context deadline exceeded.
I0320 08:48:33.926518       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-k8s-danlu-vm-dev-new:10352) of cluster(k8s-danlu-vm-dev-new).
E0320 08:48:38.927461       1 cache.go:97] Failed to dial cluster(k8s-danlu-vm-dev-new): dial karmada-scheduler-estimator-k8s-danlu-vm-dev-new:10352 error: context deadline exceeded.
I0320 08:48:38.931372       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-qa-live:10352) of cluster(qa-live).
I0320 08:48:42.518730       1 reflector.go:559] pkg/generated/informers/externalversions/factory.go:121: Watch close - *v1alpha2.ResourceBinding total 11 items received
E0320 08:48:43.931722       1 cache.go:97] Failed to dial cluster(qa-live): dial karmada-scheduler-estimator-qa-live:10352 error: context deadline exceeded.
I0320 08:48:54.175426       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-qa-live:10352) of cluster(qa-live).
I0320 08:48:57.215478       1 scheduler.go:314] Start to schedule ResourceBinding(440-zxy-test-8-6214/nginx-1-deployment) as placement changed
I0320 08:48:57.215513       1 scheduler.go:441] "Begin scheduling resourceBinding with ClusterAffinities" resourceBinding="440-zxy-test-8-6214/nginx-1-deployment"
I0320 08:48:57.215571       1 scheduler.go:458] Schedule ResourceBinding(440-zxy-test-8-6214/nginx-1-deployment) with clusterAffiliates index(0)
I0320 08:48:57.215672       1 generic_scheduler.go:78] Feasible clusters found: [qa-live k8s-danlu-vm-dev-new]
I0320 08:48:57.215697       1 generic_scheduler.go:149] Plugin ClusterAffinity scores on 440-zxy-test-8-6214/nginx-1 => [{qa-live 0} {k8s-danlu-vm-dev-new 0}]
I0320 08:48:57.215715       1 generic_scheduler.go:149] Plugin ClusterLocality scores on 440-zxy-test-8-6214/nginx-1 => [{qa-live 0} {k8s-danlu-vm-dev-new 0}]
I0320 08:48:57.215731       1 generic_scheduler.go:84] Feasible clusters scores: [{qa-live 0} {k8s-danlu-vm-dev-new 0}]
I0320 08:48:57.215772       1 general.go:61] cluster qa-live has max available replicas: 243 according to cluster resource models
I0320 08:48:57.215810       1 general.go:61] cluster k8s-danlu-vm-dev-new has max available replicas: 19 according to cluster resource models
E0320 08:48:57.215936       1 util.go:53] Max cluster available replicas error: [cluster k8s-danlu-vm-dev-new does not exist in estimator cache, cluster qa-live does not exist in estimator cache]
I0320 08:48:57.215978       1 util.go:74] Target cluster: [{qa-live 243} {k8s-danlu-vm-dev-new 19}]
I0320 08:48:57.216010       1 select_clusters.go:19] Select all clusters
I0320 08:48:57.216027       1 generic_scheduler.go:90] Selected clusters: [k8s-danlu-vm-dev-new qa-live]
I0320 08:48:57.216058       1 scheduler.go:492] ResourceBinding(440-zxy-test-8-6214/nginx-1-deployment) scheduled to clusters [{k8s-danlu-vm-dev-new 4} {qa-live 2}]
I0320 08:48:57.236499       1 scheduler.go:529] Patch schedule to ResourceBinding(440-zxy-test-8-6214/nginx-1-deployment) succeed
I0320 08:48:57.236531       1 scheduler.go:735] Begin to patch status with affinityName(private) to ResourceBinding(440-zxy-test-8-6214/nginx-1-deployment).
I0320 08:48:57.247750       1 scheduler.go:759] Patch schedule status to ResourceBinding(440-zxy-test-8-6214/nginx-1-deployment) succeed
I0320 08:48:57.247824       1 scheduler.go:497] "End scheduling resourceBinding with ClusterAffinities" resourceBinding="440-zxy-test-8-6214/nginx-1-deployment"
I0320 08:48:57.247863       1 scheduler.go:721] Begin to patch status condition to ResourceBinding(440-zxy-test-8-6214/nginx-1-deployment)
I0320 08:48:57.247990       1 event.go:294] "Event occurred" object="440-zxy-test-8-6214/nginx-1-deployment" fieldPath="" kind="ResourceBinding" apiVersion="work.karmada.io/v1alpha2" type="Normal" reason="ScheduleBindingSucceed" message="Binding has been scheduled"
I0320 08:48:57.248029       1 event.go:294] "Event occurred" object="440-zxy-test-8-6214/nginx-1" fieldPath="" kind="Deployment" apiVersion="apps/v1" type="Normal" reason="ScheduleBindingSucceed" message="Binding has been scheduled"
I0320 08:48:57.266673       1 scheduler.go:759] Patch schedule status to ResourceBinding(440-zxy-test-8-6214/nginx-1-deployment) succeed
I0320 08:48:57.266789       1 scheduler.go:337] Don't need to schedule ResourceBinding(440-zxy-test-8-6214/nginx-1-deployment)
I0320 08:48:57.267858       1 scheduler.go:337] Don't need to schedule ResourceBinding(440-zxy-test-8-6214/nginx-1-deployment)
I0320 08:48:57.294235       1 scheduler.go:337] Don't need to schedule ResourceBinding(440-zxy-test-8-6214/nginx-1-deployment)
I0320 08:48:57.338882       1 scheduler.go:337] Don't need to schedule ResourceBinding(440-zxy-test-8-6214/nginx-1-deployment)
I0320 08:48:57.395042       1 scheduler.go:337] Don't need to schedule ResourceBinding(440-zxy-test-8-6214/nginx-1-deployment)
I0320 08:48:57.447337       1 scheduler.go:337] Don't need to schedule ResourceBinding(440-zxy-test-8-6214/nginx-1-deployment)
I0320 08:48:57.495011       1 scheduler.go:337] Don't need to schedule ResourceBinding(440-zxy-test-8-6214/nginx-1-deployment)
I0320 08:48:59.001714       1 scheduler.go:337] Don't need to schedule ResourceBinding(440-zxy-test-8-6214/nginx-1-deployment)
E0320 08:48:59.176440       1 cache.go:97] Failed to dial cluster(qa-live): dial karmada-scheduler-estimator-qa-live:10352 error: context deadline exceeded.
I0320 08:48:59.970372       1 scheduler.go:337] Don't need to schedule ResourceBinding(440-zxy-test-8-6214/nginx-1-deployment)
I0320 08:49:00.011168       1 event_handler.go:161] Update event for cluster k8s-danlu-vm-dev-new
I0320 08:49:00.013811       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-k8s-danlu-vm-dev-new:10352) of cluster(k8s-danlu-vm-dev-new).
I0320 08:49:01.976317       1 scheduler.go:337] Don't need to schedule ResourceBinding(440-zxy-test-8-6214/nginx-1-deployment)
I0320 08:49:02.988261       1 scheduler.go:337] Don't need to schedule ResourceBinding(440-zxy-test-8-6214/nginx-1-deployment)
I0320 08:49:03.242296       1 event_handler.go:161] Update event for cluster qa-live
E0320 08:49:05.015077       1 cache.go:97] Failed to dial cluster(k8s-danlu-vm-dev-new): dial karmada-scheduler-estimator-k8s-danlu-vm-dev-new:10352 error: context deadline exceeded.
I0320 08:49:05.018972       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-qa-live:10352) of cluster(qa-live).
E0320 08:49:10.019342       1 cache.go:97] Failed to dial cluster(qa-live): dial karmada-scheduler-estimator-qa-live:10352 error: context deadline exceeded.
I0320 08:49:10.036735       1 event_handler.go:161] Update event for cluster k8s-danlu-vm-dev-new
I0320 08:49:10.039848       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-k8s-danlu-vm-dev-new:10352) of cluster(k8s-danlu-vm-dev-new).
I0320 08:49:13.292806       1 event_handler.go:161] Update event for cluster qa-live
E0320 08:49:15.040889       1 cache.go:97] Failed to dial cluster(k8s-danlu-vm-dev-new): dial karmada-scheduler-estimator-k8s-danlu-vm-dev-new:10352 error: context deadline exceeded.
I0320 08:49:15.040941       1 worker.go:127] Dropping resource "k8s-danlu-vm-dev-new" out of the queue: dial karmada-scheduler-estimator-k8s-danlu-vm-dev-new:10352 error: context deadline exceeded
I0320 08:49:15.044346       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-qa-live:10352) of cluster(qa-live).
E0320 08:49:20.044525       1 cache.go:97] Failed to dial cluster(qa-live): dial karmada-scheduler-estimator-qa-live:10352 error: context deadline exceeded.
I0320 08:49:20.047917       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-k8s-danlu-vm-dev-new:10352) of cluster(k8s-danlu-vm-dev-new).
E0320 08:49:25.048535       1 cache.go:97] Failed to dial cluster(k8s-danlu-vm-dev-new): dial karmada-scheduler-estimator-k8s-danlu-vm-dev-new:10352 error: context deadline exceeded.
I0320 08:49:25.052153       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-qa-live:10352) of cluster(qa-live).
E0320 08:49:30.053214       1 cache.go:97] Failed to dial cluster(qa-live): dial karmada-scheduler-estimator-qa-live:10352 error: context deadline exceeded.
I0320 08:49:30.053266       1 worker.go:127] Dropping resource "qa-live" out of the queue: dial karmada-scheduler-estimator-qa-live:10352 error: context deadline exceeded
I0320 08:49:30.056988       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-k8s-danlu-vm-dev-new:10352) of cluster(k8s-danlu-vm-dev-new).
E0320 08:49:35.057335       1 cache.go:97] Failed to dial cluster(k8s-danlu-vm-dev-new): dial karmada-scheduler-estimator-k8s-danlu-vm-dev-new:10352 error: context deadline exceeded.
I0320 08:49:35.071269       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-k8s-danlu-vm-dev-new:10352) of cluster(k8s-danlu-vm-dev-new).
E0320 08:49:40.072192       1 cache.go:97] Failed to dial cluster(k8s-danlu-vm-dev-new): dial karmada-scheduler-estimator-k8s-danlu-vm-dev-new:10352 error: context deadline exceeded.
I0320 08:49:40.096362       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-k8s-danlu-vm-dev-new:10352) of cluster(k8s-danlu-vm-dev-new).
E0320 08:49:45.096965       1 cache.go:97] Failed to dial cluster(k8s-danlu-vm-dev-new): dial karmada-scheduler-estimator-k8s-danlu-vm-dev-new:10352 error: context deadline exceeded.
I0320 08:49:45.154634       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-k8s-danlu-vm-dev-new:10352) of cluster(k8s-danlu-vm-dev-new).
E0320 08:49:50.155478       1 cache.go:97] Failed to dial cluster(k8s-danlu-vm-dev-new): dial karmada-scheduler-estimator-k8s-danlu-vm-dev-new:10352 error: context deadline exceeded.
I0320 08:49:50.240810       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-k8s-danlu-vm-dev-new:10352) of cluster(k8s-danlu-vm-dev-new).
E0320 08:49:55.241670       1 cache.go:97] Failed to dial cluster(k8s-danlu-vm-dev-new): dial karmada-scheduler-estimator-k8s-danlu-vm-dev-new:10352 error: context deadline exceeded.
I0320 08:49:55.405868       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-k8s-danlu-vm-dev-new:10352) of cluster(k8s-danlu-vm-dev-new).
E0320 08:50:00.406331       1 cache.go:97] Failed to dial cluster(k8s-danlu-vm-dev-new): dial karmada-scheduler-estimator-k8s-danlu-vm-dev-new:10352 error: context deadline exceeded.
I0320 08:50:00.730543       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-k8s-danlu-vm-dev-new:10352) of cluster(k8s-danlu-vm-dev-new).
E0320 08:50:05.731221       1 cache.go:97] Failed to dial cluster(k8s-danlu-vm-dev-new): dial karmada-scheduler-estimator-k8s-danlu-vm-dev-new:10352 error: context deadline exceeded.
I0320 08:50:06.378633       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-k8s-danlu-vm-dev-new:10352) of cluster(k8s-danlu-vm-dev-new).
E0320 08:50:11.379879       1 cache.go:97] Failed to dial cluster(k8s-danlu-vm-dev-new): dial karmada-scheduler-estimator-k8s-danlu-vm-dev-new:10352 error: context deadline exceeded.
I0320 08:50:12.663307       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-k8s-danlu-vm-dev-new:10352) of cluster(k8s-danlu-vm-dev-new).
E0320 08:50:17.664338       1 cache.go:97] Failed to dial cluster(k8s-danlu-vm-dev-new): dial karmada-scheduler-estimator-k8s-danlu-vm-dev-new:10352 error: context deadline exceeded.
I0320 08:50:20.228576       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-k8s-danlu-vm-dev-new:10352) of cluster(k8s-danlu-vm-dev-new).
I0320 08:50:23.598392       1 event_handler.go:161] Update event for cluster qa-live
E0320 08:50:25.229670       1 cache.go:97] Failed to dial cluster(k8s-danlu-vm-dev-new): dial karmada-scheduler-estimator-k8s-danlu-vm-dev-new:10352 error: context deadline exceeded.
I0320 08:50:25.232877       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-qa-live:10352) of cluster(qa-live).
E0320 08:50:30.233557       1 cache.go:97] Failed to dial cluster(qa-live): dial karmada-scheduler-estimator-qa-live:10352 error: context deadline exceeded.
I0320 08:50:30.251638       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-qa-live:10352) of cluster(qa-live).
E0320 08:50:35.252230       1 cache.go:97] Failed to dial cluster(qa-live): dial karmada-scheduler-estimator-qa-live:10352 error: context deadline exceeded.
I0320 08:50:35.256605       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-k8s-danlu-vm-dev-new:10352) of cluster(k8s-danlu-vm-dev-new).
E0320 08:50:40.257721       1 cache.go:97] Failed to dial cluster(k8s-danlu-vm-dev-new): dial karmada-scheduler-estimator-k8s-danlu-vm-dev-new:10352 error: context deadline exceeded.
I0320 08:50:40.261454       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-qa-live:10352) of cluster(qa-live).
E0320 08:50:45.262429       1 cache.go:97] Failed to dial cluster(qa-live): dial karmada-scheduler-estimator-qa-live:10352 error: context deadline exceeded.
I0320 08:50:45.285939       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-qa-live:10352) of cluster(qa-live).
E0320 08:50:50.286754       1 cache.go:97] Failed to dial cluster(qa-live): dial karmada-scheduler-estimator-qa-live:10352 error: context deadline exceeded.
I0320 08:50:50.333607       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-qa-live:10352) of cluster(qa-live).
E0320 08:50:55.334767       1 cache.go:97] Failed to dial cluster(qa-live): dial karmada-scheduler-estimator-qa-live:10352 error: context deadline exceeded.
I0320 08:50:55.337723       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-k8s-danlu-vm-dev-new:10352) of cluster(k8s-danlu-vm-dev-new).
E0320 08:51:00.338145       1 cache.go:97] Failed to dial cluster(k8s-danlu-vm-dev-new): dial karmada-scheduler-estimator-k8s-danlu-vm-dev-new:10352 error: context deadline exceeded.
I0320 08:51:00.341438       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-qa-live:10352) of cluster(qa-live).
I0320 08:51:03.727065       1 event_handler.go:161] Update event for cluster qa-live
E0320 08:51:05.342628       1 cache.go:97] Failed to dial cluster(qa-live): dial karmada-scheduler-estimator-qa-live:10352 error: context deadline exceeded.
I0320 08:51:05.345881       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-qa-live:10352) of cluster(qa-live).
E0320 08:51:10.346901       1 cache.go:97] Failed to dial cluster(qa-live): dial karmada-scheduler-estimator-qa-live:10352 error: context deadline exceeded.
I0320 08:51:10.350339       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-qa-live:10352) of cluster(qa-live).
E0320 08:51:15.351162       1 cache.go:97] Failed to dial cluster(qa-live): dial karmada-scheduler-estimator-qa-live:10352 error: context deadline exceeded.
I0320 08:51:15.354543       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-qa-live:10352) of cluster(qa-live).
E0320 08:51:20.355531       1 cache.go:97] Failed to dial cluster(qa-live): dial karmada-scheduler-estimator-qa-live:10352 error: context deadline exceeded.
I0320 08:51:20.358924       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-qa-live:10352) of cluster(qa-live).
E0320 08:51:25.359519       1 cache.go:97] Failed to dial cluster(qa-live): dial karmada-scheduler-estimator-qa-live:10352 error: context deadline exceeded.
I0320 08:51:25.362741       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-k8s-danlu-vm-dev-new:10352) of cluster(k8s-danlu-vm-dev-new).
E0320 08:51:30.363697       1 cache.go:97] Failed to dial cluster(k8s-danlu-vm-dev-new): dial karmada-scheduler-estimator-k8s-danlu-vm-dev-new:10352 error: context deadline exceeded.
I0320 08:51:30.367620       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-qa-live:10352) of cluster(qa-live).
E0320 08:51:35.368394       1 cache.go:97] Failed to dial cluster(qa-live): dial karmada-scheduler-estimator-qa-live:10352 error: context deadline exceeded.
I0320 08:51:40.496564       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-qa-live:10352) of cluster(qa-live).
E0320 08:51:45.497220       1 cache.go:97] Failed to dial cluster(qa-live): dial karmada-scheduler-estimator-qa-live:10352 error: context deadline exceeded.
I0320 08:51:55.741241       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-qa-live:10352) of cluster(qa-live).
E0320 08:52:00.741892       1 cache.go:97] Failed to dial cluster(qa-live): dial karmada-scheduler-estimator-qa-live:10352 error: context deadline exceeded.
I0320 08:52:11.327181       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-k8s-danlu-vm-dev-new:10352) of cluster(k8s-danlu-vm-dev-new).
I0320 08:52:13.933487       1 event_handler.go:161] Update event for cluster qa-live
E0320 08:52:16.327467       1 cache.go:97] Failed to dial cluster(k8s-danlu-vm-dev-new): dial karmada-scheduler-estimator-k8s-danlu-vm-dev-new:10352 error: context deadline exceeded.
I0320 08:52:16.330699       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-qa-live:10352) of cluster(qa-live).
E0320 08:52:21.331616       1 cache.go:97] Failed to dial cluster(qa-live): dial karmada-scheduler-estimator-qa-live:10352 error: context deadline exceeded.
I0320 08:52:21.335007       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-qa-live:10352) of cluster(qa-live).
E0320 08:52:26.335684       1 cache.go:97] Failed to dial cluster(qa-live): dial karmada-scheduler-estimator-qa-live:10352 error: context deadline exceeded.
I0320 08:53:02.296898       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-qa-live:10352) of cluster(qa-live).
E0320 08:53:07.297101       1 cache.go:97] Failed to dial cluster(qa-live): dial karmada-scheduler-estimator-qa-live:10352 error: context deadline exceeded.
I0320 08:53:07.297141       1 worker.go:127] Dropping resource "qa-live" out of the queue: dial karmada-scheduler-estimator-qa-live:10352 error: context deadline exceeded
I0320 08:53:38.251274       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-k8s-danlu-vm-dev-new:10352) of cluster(k8s-danlu-vm-dev-new).
E0320 08:53:43.251419       1 cache.go:97] Failed to dial cluster(k8s-danlu-vm-dev-new): dial karmada-scheduler-estimator-k8s-danlu-vm-dev-new:10352 error: context deadline exceeded.
I0320 08:53:43.251447       1 worker.go:127] Dropping resource "k8s-danlu-vm-dev-new" out of the queue: dial karmada-scheduler-estimator-k8s-danlu-vm-dev-new:10352 error: context deadline exceeded

nginx-1 yaml:

cat nginx-1.yamlapiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: nginx-1
    propagationpolicy.karmada.io/name: nginx-1-private-cloud
  name: nginx-1
spec:
  progressDeadlineSeconds: 600
  replicas: 6
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: nginx-1
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: nginx-1
    spec:
      containers:
      - args:
        - -c
        - sleep 99999
        command:
        - /bin/bash
        image: ubuntu:18.04
        imagePullPolicy: Always
        lifecycle:
          preStop:
            exec:
              command:
              - sleep
              - "20"
        name: nginx-1
        resources:
          limits:
            cpu: 500m
            memory: 500Mi
          requests:
            cpu: 500m
            memory: 500Mi
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /dev/shm
          name: empty-voldshm
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: custom-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
      volumes:
      - emptyDir:
          medium: Memory
        name: empty-voldshm

clusters:

get clusters
NAME                   VERSION    MODE   READY   AGE
k8s-danlu-vm-dev-new   v1.23.13   Pull   True    4d7h
qa-live                v1.18.3    Push   True    4d23h
RainbowMango commented 1 year ago

E0320 08:49:20.044525 1 cache.go:97] Failed to dial cluster(qa-live): dial karmada-scheduler-estimator-qa-live:10352 error: context deadline exceeded. E0320 08:52:16.327467 1 cache.go:97] Failed to dial cluster(k8s-danlu-vm-dev-new): dial karmada-scheduler-estimator-k8s-danlu-vm-dev-new:10352 error: context deadline exceeded.

These logs represent there is a connection problem between karmada-scheduler and karmada-scheduler-estimator . Are you using karmada-scheduler-estimator?

soulseen commented 1 year ago

and another question is that scaling down the deployment, should decrease the pods in the public cloud first, not only according to the cluster weight to propagate deployment?

soulseen commented 1 year ago

E0320 08:49:20.044525 1 cache.go:97] Failed to dial cluster(qa-live): dial karmada-scheduler-estimator-qa-live:10352 error: context deadline exceeded. E0320 08:52:16.327467 1 cache.go:97] Failed to dial cluster(k8s-danlu-vm-dev-new): dial karmada-scheduler-estimator-k8s-danlu-vm-dev-new:10352 error: context deadline exceeded.

These logs represent there is a connection problem between karmada-scheduler and karmada-scheduler-estimator . Are you using karmada-scheduler-estimator?

seems not. but I don't know why it will connect to karmada-scheduler-estimator. anyway, it propagates correctly finally.

$ k -n karmada-system get po
NAME                                           READY   STATUS    RESTARTS   AGE
etcd-0                                         1/1     Running   0          5d2h
karmada-aggregated-apiserver-8589dc995-72tv5   1/1     Running   1          5d2h
karmada-apiserver-7f5bbf879b-mqqfc             1/1     Running   0          5d2h
karmada-controller-manager-759c6f687c-cdzt5    1/1     Running   0          5d2h
karmada-scheduler-c7df64ccc-zpr7d              1/1     Running   0          5d2h
karmada-webhook-758fdcfc8d-jt2v9               1/1     Running   0          5d2h
kube-controller-manager-694d8dc5b9-wn5f8       1/1     Running   0          5d2h
RainbowMango commented 1 year ago

but I don't know why it will connect to karmada-scheduler-estimator

Perhaps the --enable-scheduler-estimator is specified with true for your karmada-scheduler:

      --enable-scheduler-estimator                                                                                                                                                                                               
                Enable calling cluster scheduler estimator for adjusting replicas.
RainbowMango commented 1 year ago

and another question is that scaling down the deployment, should decrease the pods in the public cloud first, not only according to the cluster weight to propagate deployment?

I don't think so. Otherwise, that will break the wight declaration.

soulseen commented 1 year ago

@RainbowMango I see, but this scenario is specific, is there any way or suggestion to implement such a strategy?

RainbowMango commented 1 year ago

I guess you can take a look at this proposal which aims to handle workload scaling scenarios.

soulseen commented 1 year ago

thanks @RainbowMango , I disabled enable-scheduler-estimator, and redeploy nginx-1 deployment, the result:

spec:
  clusters:
  - name: k8s-danlu-vm-dev-new
    replicas: 6
  - name: qa-live
    replicas: 0

and I confirmed the resources are sufficient, here is the karmada-scheduler log

$ k -n karmada-system logs -f karmada-scheduler-f85744cfc-g8xqz
I0321 12:47:37.242330       1 scheduler.go:109] karmada-scheduler version: version.Info{GitVersion:"v1.5.0", GitCommit:"51fbbb055d47331e83a6ca2e90b8a5312d6a7986", GitTreeState:"clean", BuildDate:"2023-02-28T12:05:13Z", GoVersion:"go1.19.5", Compiler:"gc", Platform:"linux/amd64"}
I0321 12:47:37.244631       1 registry.go:63] Enable Scheduler plugin "APIEnablement"
I0321 12:47:37.244665       1 registry.go:63] Enable Scheduler plugin "TaintToleration"
I0321 12:47:37.244683       1 registry.go:63] Enable Scheduler plugin "ClusterAffinity"
I0321 12:47:37.244695       1 registry.go:63] Enable Scheduler plugin "SpreadConstraint"
I0321 12:47:37.244706       1 registry.go:63] Enable Scheduler plugin "ClusterLocality"
I0321 12:47:37.245714       1 leaderelection.go:248] attempting to acquire leader lease karmada-system/karmada-scheduler...
I0321 12:47:37.273501       1 leaderelection.go:352] lock is held by karmada-scheduler-c7df64ccc-zpr7d_cf994ce4-d0c4-4aea-940a-e0bcd1191b51 and has not yet expired
I0321 12:47:37.273547       1 leaderelection.go:253] failed to acquire lease karmada-system/karmada-scheduler
I0321 12:47:40.571581       1 leaderelection.go:352] lock is held by karmada-scheduler-c7df64ccc-zpr7d_cf994ce4-d0c4-4aea-940a-e0bcd1191b51 and has not yet expired
I0321 12:47:40.571612       1 leaderelection.go:253] failed to acquire lease karmada-system/karmada-scheduler
I0321 12:47:43.534847       1 leaderelection.go:352] lock is held by karmada-scheduler-c7df64ccc-zpr7d_cf994ce4-d0c4-4aea-940a-e0bcd1191b51 and has not yet expired
I0321 12:47:43.534872       1 leaderelection.go:253] failed to acquire lease karmada-system/karmada-scheduler
I0321 12:47:45.611394       1 leaderelection.go:352] lock is held by karmada-scheduler-c7df64ccc-zpr7d_cf994ce4-d0c4-4aea-940a-e0bcd1191b51 and has not yet expired
I0321 12:47:45.611431       1 leaderelection.go:253] failed to acquire lease karmada-system/karmada-scheduler
I0321 12:47:49.698784       1 leaderelection.go:352] lock is held by karmada-scheduler-c7df64ccc-zpr7d_cf994ce4-d0c4-4aea-940a-e0bcd1191b51 and has not yet expired
I0321 12:47:49.698823       1 leaderelection.go:253] failed to acquire lease karmada-system/karmada-scheduler
I0321 12:47:52.066167       1 leaderelection.go:352] lock is held by karmada-scheduler-c7df64ccc-zpr7d_cf994ce4-d0c4-4aea-940a-e0bcd1191b51 and has not yet expired
I0321 12:47:52.066206       1 leaderelection.go:253] failed to acquire lease karmada-system/karmada-scheduler
I0321 12:47:56.178361       1 leaderelection.go:258] successfully acquired lease karmada-system/karmada-scheduler
I0321 12:47:56.178521       1 scheduler.go:247] Starting karmada scheduler
I0321 12:47:56.178647       1 reflector.go:221] Starting reflector *v1alpha2.ClusterResourceBinding (0s) from pkg/generated/informers/externalversions/factory.go:121
I0321 12:47:56.178664       1 reflector.go:257] Listing and watching *v1alpha2.ClusterResourceBinding from pkg/generated/informers/externalversions/factory.go:121
I0321 12:47:56.178647       1 reflector.go:221] Starting reflector *v1alpha2.ResourceBinding (0s) from pkg/generated/informers/externalversions/factory.go:121
I0321 12:47:56.178648       1 reflector.go:221] Starting reflector *v1alpha1.Cluster (0s) from pkg/generated/informers/externalversions/factory.go:121
I0321 12:47:56.178785       1 reflector.go:257] Listing and watching *v1alpha2.ResourceBinding from pkg/generated/informers/externalversions/factory.go:121
I0321 12:47:56.178819       1 reflector.go:257] Listing and watching *v1alpha1.Cluster from pkg/generated/informers/externalversions/factory.go:121
I0321 12:47:56.186319       1 event_handler.go:144] Add event for cluster qa-live
I0321 12:47:56.186372       1 event_handler.go:144] Add event for cluster k8s-danlu-vm-dev-new
I0321 12:47:56.278684       1 shared_informer.go:303] caches populated
I0321 12:47:56.278799       1 shared_informer.go:303] caches populated
I0321 12:47:56.278824       1 shared_informer.go:303] caches populated
I0321 12:47:56.279470       1 scheduler.go:337] Don't need to schedule ResourceBinding(440-zxy-test-8-6214/nginx-1-deployment)
I0321 12:47:56.279579       1 scheduler.go:337] Don't need to schedule ResourceBinding(default/nginx-deployment)
I0321 12:48:31.148353       1 event_handler.go:161] Update event for cluster qa-live
I0321 12:48:41.308490       1 event_handler.go:161] Update event for cluster qa-live
I0321 12:48:51.384961       1 event_handler.go:161] Update event for cluster qa-live
I0321 12:49:01.422802       1 event_handler.go:161] Update event for cluster qa-live
I0321 12:49:21.482837       1 event_handler.go:161] Update event for cluster qa-live
I0321 12:50:33.162170       1 scheduler.go:337] Don't need to schedule ResourceBinding(440-zxy-test-8-6214/nginx-1-deployment)
I0321 12:50:33.435179       1 scheduler.go:337] Don't need to schedule ResourceBinding(440-zxy-test-8-6214/nginx-1-deployment)
I0321 12:50:35.457056       1 event_handler.go:161] Update event for cluster k8s-danlu-vm-dev-new
I0321 12:50:41.831266       1 event_handler.go:161] Update event for cluster qa-live
I0321 12:50:45.484722       1 event_handler.go:161] Update event for cluster k8s-danlu-vm-dev-new
I0321 12:50:51.867113       1 event_handler.go:161] Update event for cluster qa-live
I0321 12:51:11.946681       1 event_handler.go:161] Update event for cluster qa-live
I0321 12:52:01.934493       1 scheduler.go:314] Start to schedule ResourceBinding(440-zxy-test-8-6214/nginx-1-deployment) as placement changed
I0321 12:52:01.934533       1 scheduler.go:441] "Begin scheduling resourceBinding with ClusterAffinities" resourceBinding="440-zxy-test-8-6214/nginx-1-deployment"
I0321 12:52:01.934609       1 scheduler.go:458] Schedule ResourceBinding(440-zxy-test-8-6214/nginx-1-deployment) with clusterAffiliates index(0)
I0321 12:52:01.934776       1 generic_scheduler.go:78] Feasible clusters found: [k8s-danlu-vm-dev-new qa-live]
I0321 12:52:01.934822       1 generic_scheduler.go:149] Plugin ClusterAffinity scores on 440-zxy-test-8-6214/nginx-1 => [{k8s-danlu-vm-dev-new 0} {qa-live 0}]
I0321 12:52:01.934840       1 generic_scheduler.go:149] Plugin ClusterLocality scores on 440-zxy-test-8-6214/nginx-1 => [{k8s-danlu-vm-dev-new 0} {qa-live 0}]
I0321 12:52:01.934857       1 generic_scheduler.go:84] Feasible clusters scores: [{k8s-danlu-vm-dev-new 0} {qa-live 0}]
I0321 12:52:01.934891       1 general.go:61] cluster k8s-danlu-vm-dev-new has max available replicas: 17 according to cluster resource models
I0321 12:52:01.934966       1 general.go:61] cluster qa-live has max available replicas: 134 according to cluster resource models
I0321 12:52:01.934977       1 util.go:74] Target cluster: [{k8s-danlu-vm-dev-new 17} {qa-live 134}]
I0321 12:52:01.934996       1 select_clusters.go:19] Select all clusters
I0321 12:52:01.935011       1 generic_scheduler.go:90] Selected clusters: [k8s-danlu-vm-dev-new qa-live]
I0321 12:52:01.935032       1 scheduler.go:492] ResourceBinding(440-zxy-test-8-6214/nginx-1-deployment) scheduled to clusters [{k8s-danlu-vm-dev-new 4} {qa-live 2}]
I0321 12:52:01.943674       1 scheduler.go:529] Patch schedule to ResourceBinding(440-zxy-test-8-6214/nginx-1-deployment) succeed
I0321 12:52:01.943695       1 scheduler.go:735] Begin to patch status with affinityName(private) to ResourceBinding(440-zxy-test-8-6214/nginx-1-deployment).
I0321 12:52:01.952507       1 scheduler.go:759] Patch schedule status to ResourceBinding(440-zxy-test-8-6214/nginx-1-deployment) succeed
I0321 12:52:01.952589       1 scheduler.go:497] "End scheduling resourceBinding with ClusterAffinities" resourceBinding="440-zxy-test-8-6214/nginx-1-deployment"
I0321 12:52:01.952601       1 scheduler.go:721] Begin to patch status condition to ResourceBinding(440-zxy-test-8-6214/nginx-1-deployment)
I0321 12:52:01.953197       1 event.go:294] "Event occurred" object="440-zxy-test-8-6214/nginx-1-deployment" fieldPath="" kind="ResourceBinding" apiVersion="work.karmada.io/v1alpha2" type="Normal" reason="ScheduleBindingSucceed" message="Binding has been scheduled"
I0321 12:52:01.953231       1 event.go:294] "Event occurred" object="440-zxy-test-8-6214/nginx-1" fieldPath="" kind="Deployment" apiVersion="apps/v1" type="Normal" reason="ScheduleBindingSucceed" message="Binding has been scheduled"
I0321 12:52:01.967064       1 scheduler.go:759] Patch schedule status to ResourceBinding(440-zxy-test-8-6214/nginx-1-deployment) succeed
I0321 12:52:01.967155       1 scheduler.go:337] Don't need to schedule ResourceBinding(440-zxy-test-8-6214/nginx-1-deployment)
I0321 12:52:02.004160       1 scheduler.go:337] Don't need to schedule ResourceBinding(440-zxy-test-8-6214/nginx-1-deployment)
I0321 12:52:02.074235       1 scheduler.go:337] Don't need to schedule ResourceBinding(440-zxy-test-8-6214/nginx-1-deployment)
I0321 12:52:02.162818       1 scheduler.go:337] Don't need to schedule ResourceBinding(440-zxy-test-8-6214/nginx-1-deployment)
I0321 12:52:02.203602       1 event_handler.go:161] Update event for cluster qa-live
I0321 12:52:02.229536       1 scheduler.go:337] Don't need to schedule ResourceBinding(440-zxy-test-8-6214/nginx-1-deployment)
I0321 12:52:05.605029       1 event_handler.go:161] Update event for cluster k8s-danlu-vm-dev-new
I0321 12:52:12.250563       1 event_handler.go:161] Update event for cluster qa-live
I0321 12:52:12.300516       1 event_handler.go:161] Update event for cluster qa-live
I0321 12:52:15.636331       1 event_handler.go:161] Update event for cluster k8s-danlu-vm-dev-new

propagation policy:

apiVersion: policy.karmada.io/v1alpha1
kind: PropagationPolicy
metadata:
  annotations:
  creationTimestamp: "2023-03-19T07:36:31Z"
  generation: 3
  name: nginx-1-private-cloud
  namespace: 440-zxy-test-8-6214
  resourceVersion: "910655"
  uid: 4353d81a-c0f4-4257-bd5a-6cb4d9a7e667
spec:
  placement:
    clusterAffinities:
    - affinityName: private
      clusterNames:
      - k8s-danlu-vm-dev-new
      - qa-live
    - affinityName: private-and-public
      clusterNames:
      - k8s-danlu-vm-dev-new
      - qa-live
    clusterTolerations:
    - effect: NoExecute
      key: cluster.karmada.io/not-ready
      operator: Exists
      tolerationSeconds: 300
    - effect: NoExecute
      key: cluster.karmada.io/unreachable
      operator: Exists
      tolerationSeconds: 300
    replicaScheduling:
      replicaDivisionPreference: Weighted
      replicaSchedulingType: Divided
      weightPreference:
        staticWeightList:
        - targetCluster:
            clusterNames:
            - k8s-danlu-vm-dev-new
          weight: 4
        - targetCluster:
            clusterNames:
            - qa-live
          weight: 2
  priority: 0
  resourceSelectors:
  - apiVersion: apps/v1
    kind: Deployment
    name: nginx-1
    namespace: 440-zxy-test-8-6214
  schedulerName: default-scheduler
RainbowMango commented 1 year ago

That's weird, I can see the assignment from the log which seems work as expected:

I0321 12:52:01.935032       1 scheduler.go:492] ResourceBinding(440-zxy-test-8-6214/nginx-1-deployment) scheduled to clusters [{k8s-danlu-vm-dev-new 4} {qa-live 2}]

Can this issue be reproduced on an environment built by hack/local-up-karmada.sh?

saowu commented 11 months ago

I also encountered a similar issue

root@wh-mcp-uat-host-master-217-41:/data/karmada# kmc get clusters
NAME                 VERSION   MODE   READY   AGE
mcp-bj-uat-member1   v1.25.9   Push   True    65m
mcp-wh-uat-member2   v1.25.9   Push   True    65m
root@wh-mcp-uat-host-master-217-41:/data/karmada# kmc get deploy
NAME    READY   UP-TO-DATE   AVAILABLE   AGE
nginx   0/2     0            0           59m
root@wh-mcp-uat-host-master-217-41:/data/karmada# kmc describe rb 
Name:         nginx-deployment
Namespace:    default
Labels:       propagationpolicy.karmada.io/name=example-policy
              propagationpolicy.karmada.io/namespace=default
Annotations:  <none>
API Version:  work.karmada.io/v1alpha2
Kind:         ResourceBinding
Metadata:
  Creation Timestamp:  2023-05-22T03:47:25Z
  Finalizers:
    karmada.io/binding-controller
  Generation:  1
  Managed Fields:
    API Version:  work.karmada.io/v1alpha2
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:finalizers:
          .:
          v:"karmada.io/binding-controller":
        f:labels:
          .:
          f:propagationpolicy.karmada.io/name:
          f:propagationpolicy.karmada.io/namespace:
        f:ownerReferences:
          .:
          k:{"uid":"24985b7b-9318-4b53-b750-fe51bb783b4d"}:
      f:spec:
        .:
        f:replicas:
        f:resource:
          .:
          f:apiVersion:
          f:kind:
          f:name:
          f:namespace:
          f:resourceVersion:
    Manager:      karmada-controller-manager
    Operation:    Update
    Time:         2023-05-22T03:47:25Z
    API Version:  work.karmada.io/v1alpha2
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        .:
        f:conditions:
    Manager:      karmada-controller-manager
    Operation:    Update
    Subresource:  status
    Time:         2023-05-22T03:47:25Z
  Owner References:
    API Version:           apps/v1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  Deployment
    Name:                  nginx
    UID:                   24985b7b-9318-4b53-b750-fe51bb783b4d
  Resource Version:        482827
  UID:                     aac25235-5994-4b82-a728-ac0c5905ce50
Spec:
  Replicas:  2
  Resource:
    API Version:       apps/v1
    Kind:              Deployment
    Name:              nginx
    Namespace:         default
    Resource Version:  482825
Status:
  Conditions:
    Last Transition Time:  2023-05-22T03:47:25Z
    Message:               Failed to apply all works, see status.aggregatedStatus for details
    Reason:                FullyAppliedFailed
    Status:                False
    Type:                  FullyApplied
Events:
  Type    Reason                  Age                From                Message
  ----    ------                  ----               ----                -------
  Normal  SyncWorkSucceed         59m (x2 over 59m)  binding-controller  Sync work of resourceBinding(default/nginx-deployment) successful.
  Normal  AggregateStatusSucceed  59m (x2 over 59m)  binding-controller  Update resourceBinding(default/nginx-deployment) with AggregatedStatus successfully.

karmada-scheduler-5b66ccfb64-fdvm6 logs

E0522 04:49:34.715066       1 cache.go:97] Failed to dial cluster(mcp-wh-uat-member2): dial karmada-scheduler-estimator-mcp-wh-uat-member2:10352 error: context deadline exceeded.
I0522 04:49:34.718663       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-mcp-bj-uat-member1:10352) of cluster(mcp-bj-uat-member1).
E0522 04:49:39.718983       1 cache.go:97] Failed to dial cluster(mcp-bj-uat-member1): dial karmada-scheduler-estimator-mcp-bj-uat-member1:10352 error: context deadline exceeded.
I0522 04:49:39.722157       1 cache.go:94] Start dialing estimator server(karmada-scheduler-estimator-mcp-wh-uat-member2:10352) of cluster(mcp-wh-uat-member2).
saowu commented 11 months ago 
apiVersion: policy.karmada.io/v1alpha1
kind: PropagationPolicy
metadata:
  name: nginx-propagation
spec:
  resourceSelectors:
    - apiVersion: apps/v1
      kind: Deployment
      name: nginx
  placement:
    clusterAffinity:
      clusterNames:
        - mcp-bj-uat-member1
        - mcp-uat-wh-member2
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - image: nginx
        name: nginx
saowu commented 11 months ago

image

RainbowMango commented 11 months ago

@saowu Please refer to https://github.com/karmada-io/karmada/issues/3284#issuecomment-1476166530 to disable the estimator if you don't need it by --enable-scheduler-estimator=false.

saowu commented 11 months ago

我需要它，但是我不知道怎么启动scheduler-estimator，我尝试使用下面的命令启动: karmadactl addons enable karmada-scheduler-estimator --cluster=mcp-wh-uat-member2 --member-kubeconfig /data/karmada/wh-config，但是出现了问题，日志如下：

I0522 08:15:12.829332       1 scheduler-estimator.go:84] karmada-scheduler-estimator version: version.Info{GitVersion:"v1.5.0", GitCommit:"51fbbb055d47331e83a6ca2e90b8a5312d6a7986", GitTreeState:"clean", BuildDate:"2023-02-28T12:05:12Z", GoVersion:"go1.19.5", Compiler:"gc", Platform:"linux/amd64"}
E0522 08:15:12.831690       1 run.go:74] "command failed" err="error building kubeconfig: invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable"
RainbowMango commented 11 months ago

@wuyingjun-lucky Could you please help with it?

saowu commented 11 months ago

@saowu如果您不需要估计器，请参考#3284（评论）--enable-scheduler-estimator=false禁用估计器。 我尝试--enable-scheduler-estimator=false，但是nginx依旧没有传播到成员集群

duanmengkk commented 11 months ago

我需要它，但是我不知道怎么启动scheduler-estimator，我尝试使用下面的命令启动: karmadactl addons enable karmada-scheduler-estimator --cluster=mcp-wh-uat-member2 --member-kubeconfig /data/karmada/wh-config，但是出现了问题，日志如下：

I0522 08:15:12.829332       1 scheduler-estimator.go:84] karmada-scheduler-estimator version: version.Info{GitVersion:"v1.5.0", GitCommit:"51fbbb055d47331e83a6ca2e90b8a5312d6a7986", GitTreeState:"clean", BuildDate:"2023-02-28T12:05:12Z", GoVersion:"go1.19.5", Compiler:"gc", Platform:"linux/amd64"}
E0522 08:15:12.831690       1 run.go:74] "command failed" err="error building kubeconfig: invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable"

Please try to use the --kubeconfig parameter and --karmada-kubeconfig parameter and execute the command again.

saowu commented 11 months ago 
karmadactl addons enable karmada-scheduler-estimator --cluster=mcp-bj-uat-member1 --member-kubeconfig /data/karmada/bj-config --karmada-scheduler-estimator-image=harbor.jiajiayue.com/abcdocker/karmada-scheduler-estimator:v1.5.0 --karmada-kubeconfig=/etc/karmada/karmada-apiserver.config --karmada-context=karmada-apiserver --kubeconfig=/root/.kube/config

No changes

image

duanmengkk commented 11 months ago
image

Estimator has been deployed

saowu commented 11 months ago

Repeated restart

I0523 01:35:20.875548       1 scheduler-estimator.go:84] karmada-scheduler-estimator version: version.Info{GitVersion:"v1.5.0", GitCommit:"51fbbb055d47331e83a6ca2e90b8a5312d6a7986", GitTreeState:"clean", BuildDate:"2023-02-28T12:05:12Z", GoVersion:"go1.19.5", Compiler:"gc", Platform:"linux/amd64"}
E0523 01:35:20.877582       1 run.go:74] "command failed" err="error building kubeconfig: invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable"
saowu commented 11 months ago

Found the problem because the member cluster context does not exist