Upgrades fail with instances not available for some asg

[kianjones4]

Is this a BUG REPORT or FEATURE REQUEST?: BUG REPORT

What happened: upgrade failed for kafka cluster with zookeeper saying Instances not available, and rollup objects stuck in error state

What you expected to happen: Instances in the zk-nodes and kafka-nodes asgs to be upgraded appropriately.

How to reproduce it (as minimally and precisely as possible): Create asg with name zk-nodes.kafka-test.cluster.k8s.local or kafka-nodes.kafka-test.cluster.k8s.local in aws, then try to submit a rollup object with spec.AsgName: zk-nodes.kafka-test.cluster.k8s.local or kafka-nodes.kafka-test.cluster.k8s.local and check the logs of the rollup controller pod

Anything else we need to know?: This doesn't seem to be a problem with all asg names. My other rollups foo-bar1 and iks-system upgraded successfully

Environment: AWS

• rolling-upgrade-controller version 0.5
• Kubernetes version : 1.13.10

  $ kubectl version -o yaml

Other debugging information (if applicable):

• RollingUpgrade status: error

  
  $ kubectl describe rollingupgrade <rollingupgrade-name>
  Name:         rollup-kafka-nodes-1.21.0-snapshot-kafka-nodes.kjones.cluster.k8s.local-20190912212658
  Namespace:    kube-system
  Labels:       controller-tools.k8s.io=1.0
  Annotations:  janitor/ttl: 1d
            kubectl.kubernetes.io/last-applied-configuration:
              {"apiVersion":"asgtools.intuit.k8s.io/v1beta1","kind":"Rollup","metadata":{"annotations":{},"labels":{"controller-tools.k8s.io":"1.0"},"na...
  API Version:  asgtools.intuit.k8s.io/v1beta1
  Kind:         Rollup
  Metadata:
  Creation Timestamp:  2019-09-12T21:42:48Z
  Generation:          6
  Resource Version:    1942816
  Self Link:           /apis/asgtools.intuit.k8s.io/v1beta1/namespaces/kube-system/rollups/rollup-kafka-nodes-1.21.0-snapshot-kafka-nodes.kjones.cluster.k8s.local-20190912212658
  UID:                 4342b09b-d5a6-11e9-95b0-06c5ebd983cc
  Spec:
  Asg Name:                  kafka-nodes.kjones.cluster.k8s.local
  Node Interval Seconds:     300
  Post Drain Delay Seconds:  90
  Post Drain Script:         echo "Hello, postDrain!"
  
  Post Drain Wait Script:  echo "Hello, postDrainWait!"
  
  Post Terminate Script:  echo "Hello, postTerminate!"
  
  Region:  us-west-2
  Strategy:
  Drain Timeout:    600
  Max Unavailable:  1
  Type:             randomUpdate
  Status:
  Current Status:         error
  End Time:               2019-09-12T21:56:04Z
  Nodes Processed:        2
  Start Time:             2019-09-12T21:42:48Z
  Total Nodes:            2
  Total Processing Time:  13m16.320780069s
  Events:                   <none>

- controller logs:

$ kubectl logs 2019/09/26 13:20:38 error: Instances are not available for update occurred for rollup-kafka-nodes-1.21.0-snapshot-kafka-nodes.kafka-test.cluster.k8s.local-20190926194335 2019/09/26 13:20:38 error: Instances are not available for update occurred for rollup-kafka-nodes-1.21.0-snapshot-kafka-nodes.kafka-test.cluster.k8s.local-20190926194335 2019/09/26 13:20:38 Deleted the entries of ASG kafka-nodes.kafka-test.cluster.k8s.local in the cluster store for rollup-kafka-nodes-1.21.0-snapshot-kafka-nodes.kafka-test.cluster.k8s.local-20190926194335 2019/09/26 13:20:38 Marked object rollup-kafka-nodes-1.21.0-snapshot-kafka-nodes.kafka-test.cluster.k8s.local-20190926194335 as error 2019/09/26 13:20:38 Deleted rollup-kafka-nodes-1.21.0-snapshot-kafka-nodes.kafka-test.cluster.k8s.local-20190926194335 from admission map 0xc0001fd740

[shrinandj]

Note that the bug itself does not have anything to do with Kafka. If a cluster has multiple instance-groups (ASGs) where name of one is a substring of other/s, upgrade-manager can get confused and cause rolling-upgrade of instances in different IGs.