deploy-sourcegraph: restricted integration test fails with Kubernetes 1.16+

[bobheadxi]

https://github.com/sourcegraph/deploy-sourcegraph/pull/1067 builds (e.g. https://buildkite.com/sourcegraph/deploy-sourcegraph/builds/4557) and other deploy-sourcegraph builds were failing for a variety of reasons:

ERROR: (gcloud.container.clusters.delete) Some requests did not succeed:
 - args: ["Operation [<Operation\n clusterConditions: [<StatusCondition\n message: 'Failed to delete cluster'>]\n detail: 'Failed to delete cluster'\n endTime: '2020-10-08T16:03:27.65647411Z'\n name: 'operation-1602172835577-6227ef6b'\n nodepoolConditions: []\n operationType: OperationTypeValueValuesEnum(DELETE_CLUSTER, 2)\n selfLink: 'https://container.googleapis.com/v1/projects/232313352803/zones/us-central1-a/operations/operation-1602172835577-6227ef6b'\n startTime: '2020-10-08T16:00:35.577431094Z'\n status: StatusValueValuesEnum(DONE, 3)\n statusMessage: 'Failed to delete cluster'\n targetLink: 'https://container.googleapis.com/v1/projects/232313352803/zones/us-central1-a/clusters/ds-test-restricted-202943b6'\n zone: 'us-central1-a'>] finished with error: Failed to delete cluster"]
   exit_code: 1

Then, a few days later:

+ kubectl create role -n ns-sourcegraph nonroot:unprivileged --verb=use --resource=podsecuritypolicy --resource-name=nonroot-policy
error: can not perform 'use' on 'podsecuritypolicies' in group 'policy'

Since the latter error (which occurs early in the test) replaced the former, I didn't investigate further into the first error, but the second error was traced down to a version change in the test (due to versions not being pinned) to Kubernetes 1.16+, which I was unable to resolve, possibly related to:

• https://kubernetes.io/blog/2019/09/18/kubernetes-1-16-release-announcement/#custom-resources-reach-general-availability:

  As you upgrade to the GA API, you’ll notice that several of the previously optional guard rails have become required and/or default behavior. Things like structural schemas, pruning unknown fields, validation, and protecting the *.k8s.io group are important for ensuring the longevity of your APIs and are now much harder to accidentally miss. Defaulting is another important part of API evolution and that support will be on by default for CRD.v1. The combination of these, along with CRD conversion mechanisms are enough to build stable APIs that evolve over time, the same way that native Kubernetes resources have changed without breaking backward-compatibility.

• https://kubernetes.io/blog/2019/07/18/api-deprecations-in-1-16/

A workaround to unblock the 3.21 release was introduced in https://github.com/sourcegraph/deploy-sourcegraph/pull/1068 by simply pinning the kubernetes version to the version used in the last passing build, 1.15.12-gke.20

This is probably not a long-term solution since we will likely need to upgrade eventually, so follow-up items include:

• [x] check current documentation to ensure we don't recommend the steps that fail in the restricted test => this is currently only mentioned outside of test.sh files in one place, this handbook page: https://about.sourcegraph.com/handbook/engineering/distribution/k8s_admin_custom_policy - cc @uwedeportivo , seems like you created this in https://github.com/sourcegraph/about/pull/534?
• [ ] update the restricted test to work with 1.16+

Stretch goals might include running the restricted test across multiple versions the same way the "fresh" test does via Pulumi as well

[bobheadxi]

@uwedeportivo assigning you for now since you might have some context given it seems you wrote the handbook page on it (https://github.com/sourcegraph/about/pull/534), but let me know if that's not the case!

[daxmc99]

Fixed here https://github.com/sourcegraph/deploy-sourcegraph/commit/11f118af7708d342f73825ae7cf8dfa005e95614#diff-15e0ca283574f31c001f74562a52a0108305e3df7e86c9181a21b8401b6a9273R53-R54

But we should evaluate a CI solution that doesn't require spinning up a GKE cluster to test. Our CI is a bit brittle on deploy-sourcegraph right now.