search

apache / pinot

Apache Pinot - A realtime distributed OLAP datastore
https://pinot.apache.org/
Apache License 2.0
5.45k stars 1.28k forks source link

minimizeDataMovement rebalance should minimize partition movement across servers #14151

Open dang-stripe opened 2 weeks ago

dang-stripe commented 2 weeks ago

This is a continuation of this issue: https://github.com/apache/pinot/issues/11894

Background

When rebalancing upsert tables, we are noticing that nearly all segments are moving to a new host with minimizeDataMovement on. This can risk servers running out of disk space even when combined with the new low disk mode from https://github.com/apache/pinot/pull/12072.

Example

Our current table config looks like and is using pool-based instance assignment.

        "replicaGroupPartitionConfig": {
          "replicaGroupBased": true,
          "numInstances": 0,
          "numReplicaGroups": 2,
          "numInstancesPerReplicaGroup": 0,
          "numPartitions": 0,
          "numInstancesPerPartition": 0,
          "minimizeDataMovement": true
        },

In our test table, we have 6 servers in each pool. We added 1 server to each pool and used a script to compare ideal state before and after the rebalance. The script shows how many segments are added/removed from each server. As you can see here, there's a lot of movement.

Screenshot 2024-10-02 at 3 38 06 PM

We also tried switching the table to use explicit numPartitions and numInstancesPerPartition.

        "replicaGroupPartitionConfig": {
          "replicaGroupBased": true,
          "numInstances": 0,
          "numReplicaGroups": 2,
          "numInstancesPerReplicaGroup": 0,
          "numPartitions": 180,
          "numInstancesPerPartition": 1,
          "minimizeDataMovement": true
        },

When rebalancing to this configuration, no segments moved. When we tried to scale up again from 6 -> 7 servers, we saw the exact same segment movement.

During the scale up rebalance, we also used low disk mode. We saw disk go from 37% and peaked at 52%.

Screenshot 2024-09-26 at 9 34 31 AM

Desired Behavior

We ideally want minimizeDataMovement to operate the same as Kafka's StickyAssignor algorithm which 1) maintains a uniform distribution of partitions and 2) minimizes partition movements based on the previous assignment.

## Scale up

# Current partition assignment
Host 1 -> [1, 3, 5]
Host 2 -> [2, 4, 6]

# Current behavior
Host 1 -> [1, 4]
Host 2 -> [2, 5]
Host 3 -> [3, 6] 

# Desired behavior
Host 1 -> [1, 3]
Host 2 -> [2, 4]
Host 3 -> [5, 6]

cc @Jackie-Jiang @jadami10

Jackie-Jiang commented 2 weeks ago

The behavior without explicit partition assignment is expected because partitions are always assigned round-robin. When partition is explicitly assigned, we should try to minimize the movement. I'd expect 1/6 of the partitions being moved. @dang-stripe Can you give more details on the instance partitions before and after the reassignment when partition is explicitly assigned?

cc @jackjlli

dang-stripe commented 2 weeks ago

Instance partitions with numPartitions=0 and numInstancesPerPartition=0:

      "partitionToInstancesMap": {
        "0_0": [
          "testinstance-uswest2b-4",
          "testinstance-uswest2b-5",
          "testinstance-uswest2b-6",
          "testinstance-uswest2b-1",
          "testinstance-uswest2b-2",
          "testinstance-uswest2b-3",
          // 7 was added when rebalancing for the scale up
          "testinstance-uswest2b-7"
        ],
        "0_1": [
          "testinstance-uswest2a-4",
          "testinstance-uswest2a-5",
          "testinstance-uswest2a-6",
          "testinstance-uswest2a-1",
          "testinstance-uswest2a-2",
          "testinstance-uswest2a-3",
          // 7 was added when rebalancing for the scale up
          "testinstance-uswest2a-7"
        ]
      }

Instance partitions with numPartitions=180 and numInstancesPerPartition=1 with 6 hosts per pool

"partitionToInstancesMap": {
    "0_0": ["testinstance-uswest2b-4"],
    "0_1": ["testinstance-uswest2a-4"],
    "100_0": ["testinstance-uswest2b-2"],
    "100_1": ["testinstance-uswest2a-2"],
    "101_0": ["testinstance-uswest2b-3"],
    "101_1": ["testinstance-uswest2a-3"],
    "102_0": ["testinstance-uswest2b-4"],
    "102_1": ["testinstance-uswest2a-4"],
    "103_0": ["testinstance-uswest2b-5"],
    "103_1": ["testinstance-uswest2a-5"],
    "104_0": ["testinstance-uswest2b-6"],
    "104_1": ["testinstance-uswest2a-6"],
    "105_0": ["testinstance-uswest2b-1"],
    "105_1": ["testinstance-uswest2a-1"],
    "106_0": ["testinstance-uswest2b-2"],
    "106_1": ["testinstance-uswest2a-2"],
    "107_0": ["testinstance-uswest2b-3"],
    "107_1": ["testinstance-uswest2a-3"],
    "108_0": ["testinstance-uswest2b-4"],
    "108_1": ["testinstance-uswest2a-4"],
    "109_0": ["testinstance-uswest2b-5"],
    "109_1": ["testinstance-uswest2a-5"],
    "10_0": ["testinstance-uswest2b-2"],
    "10_1": ["testinstance-uswest2a-2"],
    "110_0": ["testinstance-uswest2b-6"],
    "110_1": ["testinstance-uswest2a-6"],
    "111_0": ["testinstance-uswest2b-1"],
    "111_1": ["testinstance-uswest2a-1"],
    "112_0": ["testinstance-uswest2b-2"],
    "112_1": ["testinstance-uswest2a-2"],
    "113_0": ["testinstance-uswest2b-3"],
    "113_1": ["testinstance-uswest2a-3"],
    "114_0": ["testinstance-uswest2b-4"],
    "114_1": ["testinstance-uswest2a-4"],
    "115_0": ["testinstance-uswest2b-5"],
    "115_1": ["testinstance-uswest2a-5"],
    "116_0": ["testinstance-uswest2b-6"],
    "116_1": ["testinstance-uswest2a-6"],
    "117_0": ["testinstance-uswest2b-1"],
    "117_1": ["testinstance-uswest2a-1"],
    "118_0": ["testinstance-uswest2b-2"],
    "118_1": ["testinstance-uswest2a-2"],
    "119_0": ["testinstance-uswest2b-3"],
    "119_1": ["testinstance-uswest2a-3"],
    "11_0": ["testinstance-uswest2b-3"],
    "11_1": ["testinstance-uswest2a-3"],
    "120_0": ["testinstance-uswest2b-4"],
    "120_1": ["testinstance-uswest2a-4"],
    "121_0": ["testinstance-uswest2b-5"],
    "121_1": ["testinstance-uswest2a-5"],
    "122_0": ["testinstance-uswest2b-6"],
    "122_1": ["testinstance-uswest2a-6"],
    "123_0": ["testinstance-uswest2b-1"],
    "123_1": ["testinstance-uswest2a-1"],
    "124_0": ["testinstance-uswest2b-2"],
    "124_1": ["testinstance-uswest2a-2"],
    "125_0": ["testinstance-uswest2b-3"],
    "125_1": ["testinstance-uswest2a-3"],
    "126_0": ["testinstance-uswest2b-4"],
    "126_1": ["testinstance-uswest2a-4"],
    "127_0": ["testinstance-uswest2b-5"],
    "127_1": ["testinstance-uswest2a-5"],
    "128_0": ["testinstance-uswest2b-6"],
    "128_1": ["testinstance-uswest2a-6"],
    "129_0": ["testinstance-uswest2b-1"],
    "129_1": ["testinstance-uswest2a-1"],
    "12_0": ["testinstance-uswest2b-4"],
    "12_1": ["testinstance-uswest2a-4"],
    "130_0": ["testinstance-uswest2b-2"],
    "130_1": ["testinstance-uswest2a-2"],
    "131_0": ["testinstance-uswest2b-3"],
    "131_1": ["testinstance-uswest2a-3"],
    "132_0": ["testinstance-uswest2b-4"],
    "132_1": ["testinstance-uswest2a-4"],
    "133_0": ["testinstance-uswest2b-5"],
    "133_1": ["testinstance-uswest2a-5"],
    "134_0": ["testinstance-uswest2b-6"],
    "134_1": ["testinstance-uswest2a-6"],
    "135_0": ["testinstance-uswest2b-1"],
    "135_1": ["testinstance-uswest2a-1"],
    "136_0": ["testinstance-uswest2b-2"],
    "136_1": ["testinstance-uswest2a-2"],
    "137_0": ["testinstance-uswest2b-3"],
    "137_1": ["testinstance-uswest2a-3"],
    "138_0": ["testinstance-uswest2b-4"],
    "138_1": ["testinstance-uswest2a-4"],
    "139_0": ["testinstance-uswest2b-5"],
    "139_1": ["testinstance-uswest2a-5"],
    "13_0": ["testinstance-uswest2b-5"],
    "13_1": ["testinstance-uswest2a-5"],
    "140_0": ["testinstance-uswest2b-6"],
    "140_1": ["testinstance-uswest2a-6"],
    "141_0": ["testinstance-uswest2b-1"],
    "141_1": ["testinstance-uswest2a-1"],
    "142_0": ["testinstance-uswest2b-2"],
    "142_1": ["testinstance-uswest2a-2"],
    "143_0": ["testinstance-uswest2b-3"],
    "143_1": ["testinstance-uswest2a-3"],
    "144_0": ["testinstance-uswest2b-4"],
    "144_1": ["testinstance-uswest2a-4"],
    "145_0": ["testinstance-uswest2b-5"],
    "145_1": ["testinstance-uswest2a-5"],
    "146_0": ["testinstance-uswest2b-6"],
    "146_1": ["testinstance-uswest2a-6"],
    "147_0": ["testinstance-uswest2b-1"],
    "147_1": ["testinstance-uswest2a-1"],
    "148_0": ["testinstance-uswest2b-2"],
    "148_1": ["testinstance-uswest2a-2"],
    "149_0": ["testinstance-uswest2b-3"],
    "149_1": ["testinstance-uswest2a-3"],
    "14_0": ["testinstance-uswest2b-6"],
    "14_1": ["testinstance-uswest2a-6"],
    "150_0": ["testinstance-uswest2b-4"],
    "150_1": ["testinstance-uswest2a-4"],
    "151_0": ["testinstance-uswest2b-5"],
    "151_1": ["testinstance-uswest2a-5"],
    "152_0": ["testinstance-uswest2b-6"],
    "152_1": ["testinstance-uswest2a-6"],
    "153_0": ["testinstance-uswest2b-1"],
    "153_1": ["testinstance-uswest2a-1"],
    "154_0": ["testinstance-uswest2b-2"],
    "154_1": ["testinstance-uswest2a-2"],
    "155_0": ["testinstance-uswest2b-3"],
    "155_1": ["testinstance-uswest2a-3"],
    "156_0": ["testinstance-uswest2b-4"],
    "156_1": ["testinstance-uswest2a-4"],
    "157_0": ["testinstance-uswest2b-5"],
    "157_1": ["testinstance-uswest2a-5"],
    "158_0": ["testinstance-uswest2b-6"],
    "158_1": ["testinstance-uswest2a-6"],
    "159_0": ["testinstance-uswest2b-1"],
    "159_1": ["testinstance-uswest2a-1"],
    "15_0": ["testinstance-uswest2b-1"],
    "15_1": ["testinstance-uswest2a-1"],
    "160_0": ["testinstance-uswest2b-2"],
    "160_1": ["testinstance-uswest2a-2"],
    "161_0": ["testinstance-uswest2b-3"],
    "161_1": ["testinstance-uswest2a-3"],
    "162_0": ["testinstance-uswest2b-4"],
    "162_1": ["testinstance-uswest2a-4"],
    "163_0": ["testinstance-uswest2b-5"],
    "163_1": ["testinstance-uswest2a-5"],
    "164_0": ["testinstance-uswest2b-6"],
    "164_1": ["testinstance-uswest2a-6"],
    "165_0": ["testinstance-uswest2b-1"],
    "165_1": ["testinstance-uswest2a-1"],
    "166_0": ["testinstance-uswest2b-2"],
    "166_1": ["testinstance-uswest2a-2"],
    "167_0": ["testinstance-uswest2b-3"],
    "167_1": ["testinstance-uswest2a-3"],
    "168_0": ["testinstance-uswest2b-4"],
    "168_1": ["testinstance-uswest2a-4"],
    "169_0": ["testinstance-uswest2b-5"],
    "169_1": ["testinstance-uswest2a-5"],
    "16_0": ["testinstance-uswest2b-2"],
    "16_1": ["testinstance-uswest2a-2"],
    "170_0": ["testinstance-uswest2b-6"],
    "170_1": ["testinstance-uswest2a-6"],
    "171_0": ["testinstance-uswest2b-1"],
    "171_1": ["testinstance-uswest2a-1"],
    "172_0": ["testinstance-uswest2b-2"],
    "172_1": ["testinstance-uswest2a-2"],
    "173_0": ["testinstance-uswest2b-3"],
    "173_1": ["testinstance-uswest2a-3"],
    "174_0": ["testinstance-uswest2b-4"],
    "174_1": ["testinstance-uswest2a-4"],
    "175_0": ["testinstance-uswest2b-5"],
    "175_1": ["testinstance-uswest2a-5"],
    "176_0": ["testinstance-uswest2b-6"],
    "176_1": ["testinstance-uswest2a-6"],
    "177_0": ["testinstance-uswest2b-1"],
    "177_1": ["testinstance-uswest2a-1"],
    "178_0": ["testinstance-uswest2b-2"],
    "178_1": ["testinstance-uswest2a-2"],
    "179_0": ["testinstance-uswest2b-3"],
    "179_1": ["testinstance-uswest2a-3"],
    "17_0": ["testinstance-uswest2b-3"],
    "17_1": ["testinstance-uswest2a-3"],
    "18_0": ["testinstance-uswest2b-4"],
    "18_1": ["testinstance-uswest2a-4"],
    "19_0": ["testinstance-uswest2b-5"],
    "19_1": ["testinstance-uswest2a-5"],
    "1_0": ["testinstance-uswest2b-5"],
    "1_1": ["testinstance-uswest2a-5"],
    "20_0": ["testinstance-uswest2b-6"],
    "20_1": ["testinstance-uswest2a-6"],
    "21_0": ["testinstance-uswest2b-1"],
    "21_1": ["testinstance-uswest2a-1"],
    "22_0": ["testinstance-uswest2b-2"],
    "22_1": ["testinstance-uswest2a-2"],
    "23_0": ["testinstance-uswest2b-3"],
    "23_1": ["testinstance-uswest2a-3"],
    "24_0": ["testinstance-uswest2b-4"],
    "24_1": ["testinstance-uswest2a-4"],
    "25_0": ["testinstance-uswest2b-5"],
    "25_1": ["testinstance-uswest2a-5"],
    "26_0": ["testinstance-uswest2b-6"],
    "26_1": ["testinstance-uswest2a-6"],
    "27_0": ["testinstance-uswest2b-1"],
    "27_1": ["testinstance-uswest2a-1"],
    "28_0": ["testinstance-uswest2b-2"],
    "28_1": ["testinstance-uswest2a-2"],
    "29_0": ["testinstance-uswest2b-3"],
    "29_1": ["testinstance-uswest2a-3"],
    "2_0": ["testinstance-uswest2b-6"],
    "2_1": ["testinstance-uswest2a-6"],
    "30_0": ["testinstance-uswest2b-4"],
    "30_1": ["testinstance-uswest2a-4"],
    "31_0": ["testinstance-uswest2b-5"],
    "31_1": ["testinstance-uswest2a-5"],
    "32_0": ["testinstance-uswest2b-6"],
    "32_1": ["testinstance-uswest2a-6"],
    "33_0": ["testinstance-uswest2b-1"],
    "33_1": ["testinstance-uswest2a-1"],
    "34_0": ["testinstance-uswest2b-2"],
    "34_1": ["testinstance-uswest2a-2"],
    "35_0": ["testinstance-uswest2b-3"],
    "35_1": ["testinstance-uswest2a-3"],
    "36_0": ["testinstance-uswest2b-4"],
    "36_1": ["testinstance-uswest2a-4"],
    "37_0": ["testinstance-uswest2b-5"],
    "37_1": ["testinstance-uswest2a-5"],
    "38_0": ["testinstance-uswest2b-6"],
    "38_1": ["testinstance-uswest2a-6"],
    "39_0": ["testinstance-uswest2b-1"],
    "39_1": ["testinstance-uswest2a-1"],
    "3_0": ["testinstance-uswest2b-1"],
    "3_1": ["testinstance-uswest2a-1"],
    "40_0": ["testinstance-uswest2b-2"],
    "40_1": ["testinstance-uswest2a-2"],
    "41_0": ["testinstance-uswest2b-3"],
    "41_1": ["testinstance-uswest2a-3"],
    "42_0": ["testinstance-uswest2b-4"],
    "42_1": ["testinstance-uswest2a-4"],
    "43_0": ["testinstance-uswest2b-5"],
    "43_1": ["testinstance-uswest2a-5"],
    "44_0": ["testinstance-uswest2b-6"],
    "44_1": ["testinstance-uswest2a-6"],
    "45_0": ["testinstance-uswest2b-1"],
    "45_1": ["testinstance-uswest2a-1"],
    "46_0": ["testinstance-uswest2b-2"],
    "46_1": ["testinstance-uswest2a-2"],
    "47_0": ["testinstance-uswest2b-3"],
    "47_1": ["testinstance-uswest2a-3"],
    "48_0": ["testinstance-uswest2b-4"],
    "48_1": ["testinstance-uswest2a-4"],
    "49_0": ["testinstance-uswest2b-5"],
    "49_1": ["testinstance-uswest2a-5"],
    "4_0": ["testinstance-uswest2b-2"],
    "4_1": ["testinstance-uswest2a-2"],
    "50_0": ["testinstance-uswest2b-6"],
    "50_1": ["testinstance-uswest2a-6"],
    "51_0": ["testinstance-uswest2b-1"],
    "51_1": ["testinstance-uswest2a-1"],
    "52_0": ["testinstance-uswest2b-2"],
    "52_1": ["testinstance-uswest2a-2"],
    "53_0": ["testinstance-uswest2b-3"],
    "53_1": ["testinstance-uswest2a-3"],
    "54_0": ["testinstance-uswest2b-4"],
    "54_1": ["testinstance-uswest2a-4"],
    "55_0": ["testinstance-uswest2b-5"],
    "55_1": ["testinstance-uswest2a-5"],
    "56_0": ["testinstance-uswest2b-6"],
    "56_1": ["testinstance-uswest2a-6"],
    "57_0": ["testinstance-uswest2b-1"],
    "57_1": ["testinstance-uswest2a-1"],
    "58_0": ["testinstance-uswest2b-2"],
    "58_1": ["testinstance-uswest2a-2"],
    "59_0": ["testinstance-uswest2b-3"],
    "59_1": ["testinstance-uswest2a-3"],
    "5_0": ["testinstance-uswest2b-3"],
    "5_1": ["testinstance-uswest2a-3"],
    "60_0": ["testinstance-uswest2b-4"],
    "60_1": ["testinstance-uswest2a-4"],
    "61_0": ["testinstance-uswest2b-5"],
    "61_1": ["testinstance-uswest2a-5"],
    "62_0": ["testinstance-uswest2b-6"],
    "62_1": ["testinstance-uswest2a-6"],
    "63_0": ["testinstance-uswest2b-1"],
    "63_1": ["testinstance-uswest2a-1"],
    "64_0": ["testinstance-uswest2b-2"],
    "64_1": ["testinstance-uswest2a-2"],
    "65_0": ["testinstance-uswest2b-3"],
    "65_1": ["testinstance-uswest2a-3"],
    "66_0": ["testinstance-uswest2b-4"],
    "66_1": ["testinstance-uswest2a-4"],
    "67_0": ["testinstance-uswest2b-5"],
    "67_1": ["testinstance-uswest2a-5"],
    "68_0": ["testinstance-uswest2b-6"],
    "68_1": ["testinstance-uswest2a-6"],
    "69_0": ["testinstance-uswest2b-1"],
    "69_1": ["testinstance-uswest2a-1"],
    "6_0": ["testinstance-uswest2b-4"],
    "6_1": ["testinstance-uswest2a-4"],
    "70_0": ["testinstance-uswest2b-2"],
    "70_1": ["testinstance-uswest2a-2"],
    "71_0": ["testinstance-uswest2b-3"],
    "71_1": ["testinstance-uswest2a-3"],
    "72_0": ["testinstance-uswest2b-4"],
    "72_1": ["testinstance-uswest2a-4"],
    "73_0": ["testinstance-uswest2b-5"],
    "73_1": ["testinstance-uswest2a-5"],
    "74_0": ["testinstance-uswest2b-6"],
    "74_1": ["testinstance-uswest2a-6"],
    "75_0": ["testinstance-uswest2b-1"],
    "75_1": ["testinstance-uswest2a-1"],
    "76_0": ["testinstance-uswest2b-2"],
    "76_1": ["testinstance-uswest2a-2"],
    "77_0": ["testinstance-uswest2b-3"],
    "77_1": ["testinstance-uswest2a-3"],
    "78_0": ["testinstance-uswest2b-4"],
    "78_1": ["testinstance-uswest2a-4"],
    "79_0": ["testinstance-uswest2b-5"],
    "79_1": ["testinstance-uswest2a-5"],
    "7_0": ["testinstance-uswest2b-5"],
    "7_1": ["testinstance-uswest2a-5"],
    "80_0": ["testinstance-uswest2b-6"],
    "80_1": ["testinstance-uswest2a-6"],
    "81_0": ["testinstance-uswest2b-1"],
    "81_1": ["testinstance-uswest2a-1"],
    "82_0": ["testinstance-uswest2b-2"],
    "82_1": ["testinstance-uswest2a-2"],
    "83_0": ["testinstance-uswest2b-3"],
    "83_1": ["testinstance-uswest2a-3"],
    "84_0": ["testinstance-uswest2b-4"],
    "84_1": ["testinstance-uswest2a-4"],
    "85_0": ["testinstance-uswest2b-5"],
    "85_1": ["testinstance-uswest2a-5"],
    "86_0": ["testinstance-uswest2b-6"],
    "86_1": ["testinstance-uswest2a-6"],
    "87_0": ["testinstance-uswest2b-1"],
    "87_1": ["testinstance-uswest2a-1"],
    "88_0": ["testinstance-uswest2b-2"],
    "88_1": ["testinstance-uswest2a-2"],
    "89_0": ["testinstance-uswest2b-3"],
    "89_1": ["testinstance-uswest2a-3"],
    "8_0": ["testinstance-uswest2b-6"],
    "8_1": ["testinstance-uswest2a-6"],
    "90_0": ["testinstance-uswest2b-4"],
    "90_1": ["testinstance-uswest2a-4"],
    "91_0": ["testinstance-uswest2b-5"],
    "91_1": ["testinstance-uswest2a-5"],
    "92_0": ["testinstance-uswest2b-6"],
    "92_1": ["testinstance-uswest2a-6"],
    "93_0": ["testinstance-uswest2b-1"],
    "93_1": ["testinstance-uswest2a-1"],
    "94_0": ["testinstance-uswest2b-2"],
    "94_1": ["testinstance-uswest2a-2"],
    "95_0": ["testinstance-uswest2b-3"],
    "95_1": ["testinstance-uswest2a-3"],
    "96_0": ["testinstance-uswest2b-4"],
    "96_1": ["testinstance-uswest2a-4"],
    "97_0": ["testinstance-uswest2b-5"],
    "97_1": ["testinstance-uswest2a-5"],
    "98_0": ["testinstance-uswest2b-6"],
    "98_1": ["testinstance-uswest2a-6"],
    "99_0": ["testinstance-uswest2b-1"],
    "99_1": ["testinstance-uswest2a-1"],
    "9_0": ["testinstance-uswest2b-1"],
    "9_1": ["testinstance-uswest2a-1"
    ]
  }
}

Instance partitions with numPartitions=180 and numInstancesPerPartition=1 with 7 hosts per pool. I diff'ed this with the last instance partition map and counted 300 total partition-replica changes.

"partitionToInstancesMap": {
        "0_0": ["testinstance-uswest2b-4"],
        "0_1": ["testinstance-uswest2a-4"],
        "100_0": ["testinstance-uswest2b-5"],
        "100_1": ["testinstance-uswest2a-5"],
        "101_0": ["testinstance-uswest2b-6"],
        "101_1": ["testinstance-uswest2a-6"],
        "102_0": ["testinstance-uswest2b-1"],
        "102_1": ["testinstance-uswest2a-1"],
        "103_0": ["testinstance-uswest2b-2"],
        "103_1": ["testinstance-uswest2a-2"],
        "104_0": ["testinstance-uswest2b-7"],
        "104_1": ["testinstance-uswest2a-7"],
        "105_0": ["testinstance-uswest2b-3"],
        "105_1": ["testinstance-uswest2a-3"],
        "106_0": ["testinstance-uswest2b-4"],
        "106_1": ["testinstance-uswest2a-4"],
        "107_0": ["testinstance-uswest2b-5"],
        "107_1": ["testinstance-uswest2a-5"],
        "108_0": ["testinstance-uswest2b-6"],
        "108_1": ["testinstance-uswest2a-6"],
        "109_0": ["testinstance-uswest2b-1"],
        "109_1": ["testinstance-uswest2a-1"],
        "10_0": ["testinstance-uswest2b-6"],
        "10_1": ["testinstance-uswest2a-6"],
        "110_0": ["testinstance-uswest2b-2"],
        "110_1": ["testinstance-uswest2a-2"],
        "111_0": ["testinstance-uswest2b-7"],
        "111_1": ["testinstance-uswest2a-7"],
        "112_0": ["testinstance-uswest2b-3"],
        "112_1": ["testinstance-uswest2a-3"],
        "113_0": ["testinstance-uswest2b-4"],
        "113_1": ["testinstance-uswest2a-4"],
        "114_0": ["testinstance-uswest2b-5"],
        "114_1": ["testinstance-uswest2a-5"],
        "115_0": ["testinstance-uswest2b-6"],
        "115_1": ["testinstance-uswest2a-6"],
        "116_0": ["testinstance-uswest2b-1"],
        "116_1": ["testinstance-uswest2a-1"],
        "117_0": ["testinstance-uswest2b-2"],
        "117_1": ["testinstance-uswest2a-2"],
        "118_0": ["testinstance-uswest2b-7"],
        "118_1": ["testinstance-uswest2a-7"],
        "119_0": ["testinstance-uswest2b-3"],
        "119_1": ["testinstance-uswest2a-3"],
        "11_0": ["testinstance-uswest2b-1"],
        "11_1": ["testinstance-uswest2a-1"],
        "120_0": ["testinstance-uswest2b-4"],
        "120_1": ["testinstance-uswest2a-4"],
        "121_0": ["testinstance-uswest2b-5"],
        "121_1": ["testinstance-uswest2a-5"],
        "122_0": ["testinstance-uswest2b-6"],
        "122_1": ["testinstance-uswest2a-6"],
        "123_0": ["testinstance-uswest2b-1"],
        "123_1": ["testinstance-uswest2a-1"],
        "124_0": ["testinstance-uswest2b-2"],
        "124_1": ["testinstance-uswest2a-2"],
        "125_0": ["testinstance-uswest2b-7"],
        "125_1": ["testinstance-uswest2a-7"],
        "126_0": ["testinstance-uswest2b-3"],
        "126_1": ["testinstance-uswest2a-3"],
        "127_0": ["testinstance-uswest2b-4"],
        "127_1": ["testinstance-uswest2a-4"],
        "128_0": ["testinstance-uswest2b-5"],
        "128_1": ["testinstance-uswest2a-5"],
        "129_0": ["testinstance-uswest2b-6"],
        "129_1": ["testinstance-uswest2a-6"],
        "12_0": ["testinstance-uswest2b-2"],
        "12_1": ["testinstance-uswest2a-2"],
        "130_0": ["testinstance-uswest2b-1"],
        "130_1": ["testinstance-uswest2a-1"],
        "131_0": ["testinstance-uswest2b-2"],
        "131_1": ["testinstance-uswest2a-2"],
        "132_0": ["testinstance-uswest2b-7"],
        "132_1": ["testinstance-uswest2a-7"],
        "133_0": ["testinstance-uswest2b-3"],
        "133_1": ["testinstance-uswest2a-3"],
        "134_0": ["testinstance-uswest2b-4"],
        "134_1": ["testinstance-uswest2a-4"],
        "135_0": ["testinstance-uswest2b-5"],
        "135_1": ["testinstance-uswest2a-5"],
        "136_0": ["testinstance-uswest2b-6"],
        "136_1": ["testinstance-uswest2a-6"],
        "137_0": ["testinstance-uswest2b-1"],
        "137_1": ["testinstance-uswest2a-1"],
        "138_0": ["testinstance-uswest2b-2"],
        "138_1": ["testinstance-uswest2a-2"],
        "139_0": ["testinstance-uswest2b-7"],
        "139_1": ["testinstance-uswest2a-7"],
        "13_0": ["testinstance-uswest2b-7"],
        "13_1": ["testinstance-uswest2a-7"],
        "140_0": ["testinstance-uswest2b-3"],
        "140_1": ["testinstance-uswest2a-3"],
        "141_0": ["testinstance-uswest2b-4"],
        "141_1": ["testinstance-uswest2a-4"],
        "142_0": ["testinstance-uswest2b-5"],
        "142_1": ["testinstance-uswest2a-5"],
        "143_0": ["testinstance-uswest2b-6"],
        "143_1": ["testinstance-uswest2a-6"],
        "144_0": ["testinstance-uswest2b-1"],
        "144_1": ["testinstance-uswest2a-1"],
        "145_0": ["testinstance-uswest2b-2"],
        "145_1": ["testinstance-uswest2a-2"],
        "146_0": ["testinstance-uswest2b-7"],
        "146_1": ["testinstance-uswest2a-7"],
        "147_0": ["testinstance-uswest2b-3"],
        "147_1": ["testinstance-uswest2a-3"],
        "148_0": ["testinstance-uswest2b-4"],
        "148_1": ["testinstance-uswest2a-4"],
        "149_0": ["testinstance-uswest2b-5"],
        "149_1": ["testinstance-uswest2a-5"],
        "14_0": ["testinstance-uswest2b-3"],
        "14_1": ["testinstance-uswest2a-3"],
        "150_0": ["testinstance-uswest2b-6"],
        "150_1": ["testinstance-uswest2a-6"],
        "151_0": ["testinstance-uswest2b-1"],
        "151_1": ["testinstance-uswest2a-1"],
        "152_0": ["testinstance-uswest2b-2"],
        "152_1": ["testinstance-uswest2a-2"],
        "153_0": ["testinstance-uswest2b-7"],
        "153_1": ["testinstance-uswest2a-7"],
        "154_0": ["testinstance-uswest2b-3"],
        "154_1": ["testinstance-uswest2a-3"],
        "155_0": ["testinstance-uswest2b-4"],
        "155_1": ["testinstance-uswest2a-4"],
        "156_0": ["testinstance-uswest2b-5"],
        "156_1": ["testinstance-uswest2a-5"],
        "157_0": ["testinstance-uswest2b-6"],
        "157_1": ["testinstance-uswest2a-6"],
        "158_0": ["testinstance-uswest2b-1"],
        "158_1": ["testinstance-uswest2a-1"],
        "159_0": ["testinstance-uswest2b-2"],
        "159_1": ["testinstance-uswest2a-2"],
        "15_0": ["testinstance-uswest2b-4"],
        "15_1": ["testinstance-uswest2a-4"],
        "160_0": ["testinstance-uswest2b-7"],
        "160_1": ["testinstance-uswest2a-7"],
        "161_0": ["testinstance-uswest2b-3"],
        "161_1": ["testinstance-uswest2a-3"],
        "162_0": ["testinstance-uswest2b-4"],
        "162_1": ["testinstance-uswest2a-4"],
        "163_0": ["testinstance-uswest2b-5"],
        "163_1": ["testinstance-uswest2a-5"],
        "164_0": ["testinstance-uswest2b-6"],
        "164_1": ["testinstance-uswest2a-6"],
        "165_0": ["testinstance-uswest2b-1"],
        "165_1": ["testinstance-uswest2a-1"],
        "166_0": ["testinstance-uswest2b-2"],
        "166_1": ["testinstance-uswest2a-2"],
        "167_0": ["testinstance-uswest2b-7"],
        "167_1": ["testinstance-uswest2a-7"],
        "168_0": ["testinstance-uswest2b-3"],
        "168_1": ["testinstance-uswest2a-3"],
        "169_0": ["testinstance-uswest2b-4"],
        "169_1": ["testinstance-uswest2a-4"],
        "16_0": ["testinstance-uswest2b-5"],
        "16_1": ["testinstance-uswest2a-5"],
        "170_0": ["testinstance-uswest2b-5"],
        "170_1": ["testinstance-uswest2a-5"],
        "171_0": ["testinstance-uswest2b-6"],
        "171_1": ["testinstance-uswest2a-6"],
        "172_0": ["testinstance-uswest2b-1"],
        "172_1": ["testinstance-uswest2a-1"],
        "173_0": ["testinstance-uswest2b-2"],
        "173_1": ["testinstance-uswest2a-2"],
        "174_0": ["testinstance-uswest2b-7"],
        "174_1": ["testinstance-uswest2a-7"],
        "175_0": ["testinstance-uswest2b-3"],
        "175_1": ["testinstance-uswest2a-3"],
        "176_0": ["testinstance-uswest2b-4"],
        "176_1": ["testinstance-uswest2a-4"],
        "177_0": ["testinstance-uswest2b-5"],
        "177_1": ["testinstance-uswest2a-5"],
        "178_0": ["testinstance-uswest2b-6"],
        "178_1": ["testinstance-uswest2a-6"],
        "179_0": ["testinstance-uswest2b-1"],
        "179_1": ["testinstance-uswest2a-1"],
        "17_0": ["testinstance-uswest2b-6"],
        "17_1": ["testinstance-uswest2a-6"],
        "18_0": ["testinstance-uswest2b-1"],
        "18_1": ["testinstance-uswest2a-1"],
        "19_0": ["testinstance-uswest2b-2"],
        "19_1": ["testinstance-uswest2a-2"],
        "1_0": ["testinstance-uswest2b-5"],
        "1_1": ["testinstance-uswest2a-5"],
        "20_0": ["testinstance-uswest2b-7"],
        "20_1": ["testinstance-uswest2a-7"],
        "21_0": ["testinstance-uswest2b-3"],
        "21_1": ["testinstance-uswest2a-3"],
        "22_0": ["testinstance-uswest2b-4"],
        "22_1": ["testinstance-uswest2a-4"],
        "23_0": ["testinstance-uswest2b-5"],
        "23_1": ["testinstance-uswest2a-5"],
        "24_0": ["testinstance-uswest2b-6"],
        "24_1": ["testinstance-uswest2a-6"],
        "25_0": ["testinstance-uswest2b-1"],
        "25_1": ["testinstance-uswest2a-1"],
        "26_0": ["testinstance-uswest2b-2"],
        "26_1": ["testinstance-uswest2a-2"],
        "27_0": ["testinstance-uswest2b-7"],
        "27_1": ["testinstance-uswest2a-7"],
        "28_0": ["testinstance-uswest2b-3"],
        "28_1": ["testinstance-uswest2a-3"],
        "29_0": ["testinstance-uswest2b-4"],
        "29_1": ["testinstance-uswest2a-4"],
        "2_0": ["testinstance-uswest2b-6"],
        "2_1": ["testinstance-uswest2a-6"],
        "30_0": ["testinstance-uswest2b-5"],
        "30_1": ["testinstance-uswest2a-5"],
        "31_0": ["testinstance-uswest2b-6"],
        "31_1": ["testinstance-uswest2a-6"],
        "32_0": ["testinstance-uswest2b-1"],
        "32_1": ["testinstance-uswest2a-1"],
        "33_0": ["testinstance-uswest2b-2"],
        "33_1": ["testinstance-uswest2a-2"],
        "34_0": ["testinstance-uswest2b-7"],
        "34_1": ["testinstance-uswest2a-7"],
        "35_0": ["testinstance-uswest2b-3"],
        "35_1": ["testinstance-uswest2a-3"],
        "36_0": ["testinstance-uswest2b-4"],
        "36_1": ["testinstance-uswest2a-4"],
        "37_0": ["testinstance-uswest2b-5"],
        "37_1": ["testinstance-uswest2a-5"],
        "38_0": ["testinstance-uswest2b-6"],
        "38_1": ["testinstance-uswest2a-6"],
        "39_0": ["testinstance-uswest2b-1"],
        "39_1": ["testinstance-uswest2a-1"],
        "3_0": ["testinstance-uswest2b-1"],
        "3_1": ["testinstance-uswest2a-1"],
        "40_0": ["testinstance-uswest2b-2"],
        "40_1": ["testinstance-uswest2a-2"],
        "41_0": ["testinstance-uswest2b-7"],
        "41_1": ["testinstance-uswest2a-7"],
        "42_0": ["testinstance-uswest2b-3"],
        "42_1": ["testinstance-uswest2a-3"],
        "43_0": ["testinstance-uswest2b-4"],
        "43_1": ["testinstance-uswest2a-4"],
        "44_0": ["testinstance-uswest2b-5"],
        "44_1": ["testinstance-uswest2a-5"],
        "45_0": ["testinstance-uswest2b-6"],
        "45_1": ["testinstance-uswest2a-6"],
        "46_0": ["testinstance-uswest2b-1"],
        "46_1": ["testinstance-uswest2a-1"],
        "47_0": ["testinstance-uswest2b-2"],
        "47_1": ["testinstance-uswest2a-2"],
        "48_0": ["testinstance-uswest2b-7"],
        "48_1": ["testinstance-uswest2a-7"],
        "49_0": ["testinstance-uswest2b-3"],
        "49_1": ["testinstance-uswest2a-3"],
        "4_0": ["testinstance-uswest2b-2"],
        "4_1": ["testinstance-uswest2a-2"],
        "50_0": ["testinstance-uswest2b-4"],
        "50_1": ["testinstance-uswest2a-4"],
        "51_0": ["testinstance-uswest2b-5"],
        "51_1": ["testinstance-uswest2a-5"],
        "52_0": ["testinstance-uswest2b-6"],
        "52_1": ["testinstance-uswest2a-6"],
        "53_0": ["testinstance-uswest2b-1"],
        "53_1": ["testinstance-uswest2a-1"],
        "54_0": ["testinstance-uswest2b-2"],
        "54_1": ["testinstance-uswest2a-2"],
        "55_0": ["testinstance-uswest2b-7"],
        "55_1": ["testinstance-uswest2a-7"],
        "56_0": ["testinstance-uswest2b-3"],
        "56_1": ["testinstance-uswest2a-3"],
        "57_0": ["testinstance-uswest2b-4"],
        "57_1": ["testinstance-uswest2a-4"],
        "58_0": ["testinstance-uswest2b-5"],
        "58_1": ["testinstance-uswest2a-5"],
        "59_0": ["testinstance-uswest2b-6"],
        "59_1": ["testinstance-uswest2a-6"],
        "5_0": ["testinstance-uswest2b-3"],
        "5_1": ["testinstance-uswest2a-3"],
        "60_0": ["testinstance-uswest2b-1"],
        "60_1": ["testinstance-uswest2a-1"],
        "61_0": ["testinstance-uswest2b-2"],
        "61_1": ["testinstance-uswest2a-2"],
        "62_0": ["testinstance-uswest2b-7"],
        "62_1": ["testinstance-uswest2a-7"],
        "63_0": ["testinstance-uswest2b-3"],
        "63_1": ["testinstance-uswest2a-3"],
        "64_0": ["testinstance-uswest2b-4"],
        "64_1": ["testinstance-uswest2a-4"],
        "65_0": ["testinstance-uswest2b-5"],
        "65_1": ["testinstance-uswest2a-5"],
        "66_0": ["testinstance-uswest2b-6"],
        "66_1": ["testinstance-uswest2a-6"],
        "67_0": ["testinstance-uswest2b-1"],
        "67_1": ["testinstance-uswest2a-1"],
        "68_0": ["testinstance-uswest2b-2"],
        "68_1": ["testinstance-uswest2a-2"],
        "69_0": ["testinstance-uswest2b-7"],
        "69_1": ["testinstance-uswest2a-7"],
        "6_0": ["testinstance-uswest2b-7"],
        "6_1": ["testinstance-uswest2a-7"],
        "70_0": ["testinstance-uswest2b-3"],
        "70_1": ["testinstance-uswest2a-3"],
        "71_0": ["testinstance-uswest2b-4"],
        "71_1": ["testinstance-uswest2a-4"],
        "72_0": ["testinstance-uswest2b-5"],
        "72_1": ["testinstance-uswest2a-5"],
        "73_0": ["testinstance-uswest2b-6"],
        "73_1": ["testinstance-uswest2a-6"],
        "74_0": ["testinstance-uswest2b-1"],
        "74_1": ["testinstance-uswest2a-1"],
        "75_0": ["testinstance-uswest2b-2"],
        "75_1": ["testinstance-uswest2a-2"],
        "76_0": ["testinstance-uswest2b-7"],
        "76_1": ["testinstance-uswest2a-7"],
        "77_0": ["testinstance-uswest2b-3"],
        "77_1": ["testinstance-uswest2a-3"],
        "78_0": ["testinstance-uswest2b-4"],
        "78_1": ["testinstance-uswest2a-4"],
        "79_0": ["testinstance-uswest2b-5"],
        "79_1": ["testinstance-uswest2a-5"],
        "7_0": ["testinstance-uswest2b-3"],
        "7_1": ["testinstance-uswest2a-3"],
        "80_0": ["testinstance-uswest2b-6"],
        "80_1": ["testinstance-uswest2a-6"],
        "81_0": ["testinstance-uswest2b-1"],
        "81_1": ["testinstance-uswest2a-1"],
        "82_0": ["testinstance-uswest2b-2"],
        "82_1": ["testinstance-uswest2a-2"],
        "83_0": ["testinstance-uswest2b-7"],
        "83_1": ["testinstance-uswest2a-7"],
        "84_0": ["testinstance-uswest2b-3"],
        "84_1": ["testinstance-uswest2a-3"],
        "85_0": ["testinstance-uswest2b-4"],
        "85_1": ["testinstance-uswest2a-4"],
        "86_0": ["testinstance-uswest2b-5"],
        "86_1": ["testinstance-uswest2a-5"],
        "87_0": ["testinstance-uswest2b-6"],
        "87_1": ["testinstance-uswest2a-6"],
        "88_0": ["testinstance-uswest2b-1"],
        "88_1": ["testinstance-uswest2a-1"],
        "89_0": ["testinstance-uswest2b-2"],
        "89_1": ["testinstance-uswest2a-2"],
        "8_0": ["testinstance-uswest2b-4"],
        "8_1": ["testinstance-uswest2a-4"],
        "90_0": ["testinstance-uswest2b-7"],
        "90_1": ["testinstance-uswest2a-7"],
        "91_0": ["testinstance-uswest2b-3"],
        "91_1": ["testinstance-uswest2a-3"],
        "92_0": ["testinstance-uswest2b-4"],
        "92_1": ["testinstance-uswest2a-4"],
        "93_0": ["testinstance-uswest2b-5"],
        "93_1": ["testinstance-uswest2a-5"],
        "94_0": ["testinstance-uswest2b-6"],
        "94_1": ["testinstance-uswest2a-6"],
        "95_0": ["testinstance-uswest2b-1"],
        "95_1": ["testinstance-uswest2a-1"],
        "96_0": ["testinstance-uswest2b-2"],
        "96_1": ["testinstance-uswest2a-2"],
        "97_0": ["testinstance-uswest2b-7"],
        "97_1": ["testinstance-uswest2a-7"],
        "98_0": ["testinstance-uswest2b-3"],
        "98_1": ["testinstance-uswest2a-3"],
        "99_0": ["testinstance-uswest2b-4"],
        "99_1": ["testinstance-uswest2a-4"],
        "9_0": ["testinstance-uswest2b-5"],
        "9_1": ["testinstance-uswest2a-5"]
      }
    }
Jackie-Jiang commented 2 weeks ago

Ideally minimize movement should use a greedy algorithm to preserve existing instance as much as possible. I can see it is not using round robin, but also not preserving instances effectively. @jackjlli could this be a bug in the algorithm? Or this scenario is not handled?