[Umbrella Issue] Build GKE Cluster for running bots, utilities

[dims]

• We need one "alpha" cluster that is time-boxed (30 days?)
• Same cluster will be used for both jobs
• New Google Group for admins : k8s-infra-cluster-admins
• Group will have "GKE Admin" access (described in [1])
• Google Group seeded with
  • dims
  • justinsb
• Job owners will be temporarily given "GKE Developer" access
• Once the Job is running, Job owners will be switched over to "GKE Viewer" access
• Jobs must have images, yamls, scripts in public repositories.
• Job owners must document how to run the jobs
• Job logs must be made public (somehow)
• We will store info about each job in kubernetes/k8s.io repository for easy discovery
• Eventually, we should have enough scripts to start a new cluster with all the jobs that we need

[1] https://cloud.google.com/iam/docs/understanding-roles#kubernetes_engine_roles

[dims]

cc @sttts @justinsb @thockin

[justinsb]

I think this is a great list, balancing starting on the journey with where we want to go.

If I could express an end-state philosophy, I think it is this:

• Some group of people have to seed the system, and act as backstop in case of problems, but those credentials/permissions should should not be used day-to-day.
• We want all provisioning of the cluster(s) to be automated and driven by a public git repo.
• We want all the jobs to be provisioned automatically, driven by a git repo.
• We want job developers to be effective (e.g. logs, see pods and events?), but we otherwise want least privilege.
• Ideally we would want to run both high-trust (sign artifacts) and low-trust (run CI) jobs on the same cluster. This might not be possible / practical / a good idea, but we would prefer not to have one-cluster-per-job.

I think we should proceed with the 30 day cluster and iterate towards this goal. My only concern is that it's a little tricky if I'm in all three roles (using myself as example) of cluster admin, project admin, developer: I want to verify that the experience as a developer is reasonable, but that's difficult if I happen to have cluster-admin rights. I really like your suggestion of using roles though - I think we could give the core group IAM admin rights, and then they could add and remove their own privileges as needed. (So we would "sudo" by granting ourselves GKE Admin, and then remove that role when we're done) Unless anyone knows a better way?

[thockin]

The way I tested was by having a distinct gmail account that I granted and removed privs from.

On Thu, Nov 8, 2018 at 7:34 AM Justin Santa Barbara < notifications@github.com> wrote:

I think this is a great list, balancing starting on the journey with where we want to go.

If I could express an end-state philosophy, I think it is this:

• Some group of people have to seed the system, and act as backstop in case of problems, but those credentials/permissions should should not be used day-to-day.
• We want all provisioning of the cluster(s) to be automated and driven by a public git repo.
• We want all the jobs to be provisioned automatically, driven by a git repo.
• We want job developers to be effective (e.g. logs, see pods and events?), but we otherwise want least privilege.
• Ideally we would want to run both high-trust (sign artifacts) and low-trust (run CI) jobs on the same cluster. This might not be possible / practical / a good idea, but we would prefer not to have one-cluster-per-job.

I think we should proceed with the 30 day cluster and iterate towards this goal. My only concern is that it's a little tricky if I'm in all three roles (using myself as example) of cluster admin, project admin, developer: I want to verify that the experience as a developer is reasonable, but that's difficult if I happen to have cluster-admin rights. I really like your suggestion of using roles though - I think we could give the core group IAM admin rights, and then they could add and remove their own privileges as needed. (So we would "sudo" by granting ourselves GKE Admin, and then remove that role when we're done) Unless anyone knows a better way?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/kubernetes/k8s.io/issues/152#issuecomment-437038608, or mute the thread https://github.com/notifications/unsubscribe-auth/AFVgVC0ZUK7cUVaBf9zlnBhQJGUkxdQXks5utE8IgaJpZM4YUvF4 .

[dims]

@thockin i can turn what we have here into a PR (README in cluster/ directory in this repo).

Is this enough for us to get started with a 30 day cluster?

[thockin]

Yes. The AI today was to create the ggroup, which we called out in this issue already - does that group exist?

We should endeavor to do better than owner/viewer split - a ggroup per app or something, plus overall admins group.

[dims]

Tim, k8s-infra-cluster-admins is the name of the google group. @justinsb has created that group and sent invites.

/assign @thockin

[thockin]

The group has cluster admin access

[dims]

i have been able to access the cluster and run the publishing-bot

[dims]

/assign @thockin /unassign

[spiffxp]

Iterating here at the moment; tldr moving to use terraform instead of bash https://github.com/kubernetes/k8s.io/issues/243#issuecomment-510118579

[spiffxp]

@thockin would like to take one last look at terraform and then we move forward

[spiffxp]

burn down cluster, rebuild using terraform https://github.com/kubernetes/k8s.io/issues/243

enumerate followup tasks like monitoring here

[thockin]

Update: we have a cluster "aaa", that comes from terraform. We have not moved everything over yet, so I'll propose that the goalpost for this bug be:

1) EOL the "development2" cluster (publisher bot) 2) Move all of the things in the google "utilicluster" into "aaa"

• cert-manager
• gcsweb
• k8s-io
• kube-perf?
• velodrome?

For this context, "move" means "with minimal but non-zero monitoring and docs".

At that point, this mission is done and new missions can be formulated.

[thockin]

I have opened issues for each specific item. "aaa" is live and gcsweb has moved.