external_service_test.go does not wait for service to disappear

[porridge]

The test for disabling the service expects the service to be gone right after the deploy plan finishes.

This is incorrect. In many cases the service resource is deleted but still present due to a finalizer. Example:

- apiVersion: v1
  kind: Service
  metadata:
    annotations:
      kudo.dev/last-applied-configuration: |
   [...]
      kudo.dev/last-plan-execution-uid: 9285b6a6-55ec-49a6-a6cb-02c867646b45
      kudo.dev/phase: nodes
      kudo.dev/plan: deploy
      kudo.dev/step: pre-node
    creationTimestamp: "2020-06-04T17:00:10Z"
    deletionGracePeriodSeconds: 0
    deletionTimestamp: "2020-06-04T17:01:38Z"
    finalizers:
    - service.kubernetes.io/load-balancer-cleanup
    labels:
      heritage: kudo
      kudo.dev/instance: cassandra-instance
      kudo.dev/operator: cassandra
    name: cassandra-instance-svc-external
    namespace: ext-service-test-namespace
    ownerReferences:
    - apiVersion: kudo.dev/v1beta1
      blockOwnerDeletion: true
      controller: true
      kind: Instance
      name: cassandra-instance
      uid: 7751f32b-67bb-428f-a5ad-0fe0317509f1
    resourceVersion: "19404"
    selfLink: /api/v1/namespaces/ext-service-test-namespace/services/cassandra-instance-svc-external
    uid: cfe3ec6f-754d-4772-b877-e1485f8e862c

[porridge]

I have two ideas how to fix this:

1. poll for the service to disappear,
2. rather than assert an error fetching the service, assert that either it's gone or deletion timestamp is set

I'd prefer (1) even though it's slower, because (2) would happily pass even if we manage to break the service in such way that it's impossible to remove ever.

cc @nfnt who added this logic for ideas

[nfnt]

Polling is what we use in similar test setups in KUDO. It's also what I recommend here because it guarantees that the service is gone.