audit followup: k8s-infra GCP projects shouldn't have random services enabled

spiffxp commented 3 years ago

This is a followup to https://github.com/kubernetes/k8s.io/issues/1675 which was focused just on k8s-staging-* GCP projects

While I haven't found docs for it, I am told that over-privileged users (e.g. organization admins) who use the UI are capable of auto-enabling services as they click from project-to-project or service-to-service. I think I often run into this when I'm using a service in my personal project, and then switch to a kubernetes.io project.

https://github.com/kubernetes/k8s.io/pull/1859 adds an ensure_only_services "${project}" "${services[@]}" function to auto-disable services that were directly specified, or direct dependencies of what was specified. I wasn't comfortable enabling it by default for everything, and I didn't want to rewrite ensure-prod-storage.sh just yet.

This issue is intended to cover:

determining what set of services we think should be enabled by default for a given class/type of project
determining how to add in special-case services for a given project (e.g. prod-only, test-only)
pruning services until we're sure we have only what we want enabled
enabling some mechanism to enforce continually, e.g.
- remove obnoxiously long env var gate, do it every time ensure_project is run by any human running a script
- setup a custom script that only prunes services, run a prowjob that does this
- etc.

spiffxp commented 3 years ago

/area infra/auditing

spiffxp commented 3 years ago

As part of https://github.com/kubernetes/k8s.io/pull/1890 I'm looking at k8s-conform's services. This project should be storage-only, so definitely going to disable compute

k8s-conform:
  intent:
    - secretmanager.googleapis.com
    - storage-api.googleapis.com
    - storage-component.googleapis.com
  enabled:
    - compute.googleapis.com
    - oslogin.googleapis.com
    - secretmanager.googleapis.com
    - storage-api.googleapis.com
    - storage-component.googleapis.com
  expected:
    - secretmanager.googleapis.com
    - storage-api.googleapis.com
    - storage-component.googleapis.com
  to_enable: []
  to_disable:
    - compute.googleapis.com
    - oslogin.googleapis.com

spiffxp commented 3 years ago

https://github.com/kubernetes/k8s.io/pull/1890#issuecomment-816326889 removed extraneous services for k8s-conform

spiffxp commented 3 years ago

I was looking into whether terraform could make this any easier... and it does not. They used to have a google_project_services resource, but ended up deprecating and deleting it, ref: https://github.com/hashicorp/terraform-provider-google/pull/4587

I'm iterating on an attempt to make ensure_only_services work a little better over in https://github.com/kubernetes/k8s.io/pull/2016 to close out https://github.com/kubernetes/k8s.io/issues/1963

spiffxp commented 3 years ago

/kind cleanup /priority important-longterm /assign I'll self assign since I'm noodling. But I could use some help, especially suggestions or prior art.

Ultimately I think the future here is a golang tool or something akin to crossplane rather than the ludicrous bash/jq I have going on

ameukam commented 3 years ago

Ultimately I think the future here is a golang tool or something akin to crossplane rather than the ludicrous bash/jq I have going on

I started to take a look at CRDs provided by config-connector : https://cloud.google.com/config-connector/docs/concepts/resources. Planned to do a parallel to #1885.

spiffxp commented 3 years ago

From a freshly provisioned project (ref: https://github.com/kubernetes/k8s.io/pull/2195#issuecomment-861823341)

The existing bash/jq is not walking deps enough. None of the services should be unexpected after this first call.

    plan to enable/disable the following services
    to_enable:
      - compute.googleapis.com
      - containerregistry.googleapis.com
      - logging.googleapis.com
      - monitoring.googleapis.com
      - storage-component.googleapis.com
    to_disable: []
    Operation "operations/acf.p2-783965148085-e3561dac-7684-4ec5-8a4e-087848353038" finished successfully.
    Operation "operations/acf.p2-783965148085-73b68eda-a5fc-45e5-a827-90df93bfa3ef" finished successfully.
    Operation "operations/acf.p2-783965148085-69a4fc7c-d39b-44da-9978-eb05e6bd4abc" finished successfully.
    Operation "operations/acf.p2-783965148085-17922383-9f5a-4e41-b070-fb6d703dc45f" finished successfully.
    Operation "operations/acf.p2-783965148085-de4f0675-e9d1-4b66-8fd6-9ecf662d56cd" finished successfully.
    "INFO: dry-run mode, would run:" gcloud services disable --force oslogin.googleapis.com --project=k8s-infra-e2e-scale-5k-project
    "INFO: dry-run mode, would run:" gcloud services disable --force pubsub.googleapis.com --project=k8s-infra-e2e-scale-5k-project
    "INFO: dry-run mode, would run:" gcloud services disable --force storage-api.googleapis.com --project=k8s-infra-e2e-scale-5k-project
    WARN: ensure_only_services: after enable/disable cycle, still projects to enable/disable: k8s-infra-e2e-scale-5k-project
    intent:
      - compute.googleapis.com
      - containerregistry.googleapis.com
      - logging.googleapis.com
      - monitoring.googleapis.com
      - storage-component.googleapis.com
    enabled:
      - compute.googleapis.com
      - containerregistry.googleapis.com
      - logging.googleapis.com
      - monitoring.googleapis.com
      - oslogin.googleapis.com
      - pubsub.googleapis.com
      - storage-api.googleapis.com
      - storage-component.googleapis.com
    expected:
      - compute.googleapis.com
      - containerregistry.googleapis.com
      - logging.googleapis.com
      - monitoring.googleapis.com
      - storage-component.googleapis.com
    to_enable: []
    to_disable:
      - oslogin.googleapis.com
      - pubsub.googleapis.com
      - storage-api.googleapis.com

I have a much gnarler set of jq lurking somewhere, I've been afraid to PR it since maybe it takes us too far in an unmaintainable direction. But I'm pretty sure it handles this correctly. I'll see if I can dig it up.

spiffxp commented 3 years ago

So I don't know when this happened, but it appears that gcloud services list no longer includes dependency information.

That's gonna put a real damper on automating an audit of this.

spiffxp commented 3 years ago

/milestone v1.23 I would like to at least get a picture of how random the enabled services are

ameukam commented 2 years ago

/milestone v1.24

k8s-triage-robot commented 2 years ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue or PR as fresh with /remove-lifecycle stale
Mark this issue or PR as rotten with /lifecycle rotten
Close this issue or PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

ameukam commented 2 years ago

/remove-lifecycle stale /lifecycle frozen /milestone clear

kubernetes / k8s.io

audit followup: k8s-infra GCP projects shouldn't have random services enabled #1887