search

GoogleCloudPlatform / gcr-cleaner

Delete untagged image refs in Google Container Registry or Artifact Registry
Apache License 2.0
805 stars 112 forks source link

not cleaning repos #116

Closed ipsingh88 closed 2 years ago

ipsingh88 commented 2 years ago

TL;DR

The cleaner stopped cleaning any repos

Expected behavior

The cleaner should clean all repos in container registry recursively but doesn't do

Observed behavior

The cleaner is not cleaning any images, the repos url I passed is not being used and seems like it is only looking into us.gcr.io rather than actual repository

Debug log output

{
    "insertId": "6361291c0009570a95bb9983",
    "jsonPayload": {
      "version": "gcr-cleaner 0.10.0 (103e593b1bf59309841ff4a483cf33f3474acf28, linux/amd64)",
      "message": "server is starting"
    },
    "resource": {
      "type": "cloud_run_revision",
      "labels": {
        "configuration_name": "gcr-cleaner",
        "revision_name": "gcr-cleaner-00006-koj",
        "location": "us-central1",
        "service_name": "gcr-cleaner",
        "project_id": "postgresql-302517"
      }
    },
    "timestamp": "2022-11-01T14:11:40Z",
    "severity": "DEBUG",
    "labels": {
      "instanceId": "001c9ea28b651e2bd95a8371783002be3f5b653cb644b05a926ffb38bdd6ad1c3d6e87217c2a0caa3cf46c02381c3cdf6ec2c984e2ed802720a2ed709335e03f23a3"
    },
    "logName": "projects/postgresql-302517/logs/run.googleapis.com%2Fstderr",
    "receiveTimestamp": "2022-11-01T14:11:40.618015956Z"
  },
  {
    "insertId": "6361291c0009590266f3c205",
    "jsonPayload": {
      "port": "8080",
      "message": "server is listening"
    },
    "resource": {
      "type": "cloud_run_revision",
      "labels": {
        "revision_name": "gcr-cleaner-00006-koj",
        "location": "us-central1",
        "configuration_name": "gcr-cleaner",
        "service_name": "gcr-cleaner",
        "project_id": "postgresql-302517"
      }
    },
    "timestamp": "2022-11-01T14:11:40Z",
    "severity": "INFO",
    "labels": {
      "instanceId": "001c9ea28b651e2bd95a8371783002be3f5b653cb644b05a926ffb38bdd6ad1c3d6e87217c2a0caa3cf46c02381c3cdf6ec2c984e2ed802720a2ed709335e03f23a3"
    },
    "logName": "projects/postgresql-302517/logs/run.googleapis.com%2Fstderr",
    "receiveTimestamp": "2022-11-01T14:11:40.949687272Z"
  },
  {
    "insertId": "6361291c000b022de90eb269",
    "jsonPayload": {
      "message": "starting clean request",
      "version": "gcr-cleaner 0.10.0 (103e593b1bf59309841ff4a483cf33f3474acf28, linux/amd64)"
    },
    "resource": {
      "type": "cloud_run_revision",
      "labels": {
        "service_name": "gcr-cleaner",
        "location": "us-central1",
        "project_id": "postgresql-302517",
        "revision_name": "gcr-cleaner-00006-koj",
        "configuration_name": "gcr-cleaner"
      }
    },
    "timestamp": "2022-11-01T14:11:40Z",
    "severity": "DEBUG",
    "labels": {
      "instanceId": "001c9ea28b651e2bd95a8371783002be3f5b653cb644b05a926ffb38bdd6ad1c3d6e87217c2a0caa3cf46c02381c3cdf6ec2c984e2ed802720a2ed709335e03f23a3"
    },
    "logName": "projects/postgresql-302517/logs/run.googleapis.com%2Fstderr",
    "receiveTimestamp": "2022-11-01T14:11:40.949687272Z"
  },
  {
    "textPayload": "gathering child repositories recursively",
    "insertId": "6361291c000b03a5fabaeb95",
    "resource": {
      "type": "cloud_run_revision",
      "labels": {
        "service_name": "gcr-cleaner",
        "project_id": "postgresql-302517",
        "revision_name": "gcr-cleaner-00006-koj",
        "location": "us-central1",
        "configuration_name": "gcr-cleaner"
      }
    },
    "timestamp": "2022-11-01T14:11:40Z",
    "severity": "DEBUG",
    "labels": {
      "instanceId": "001c9ea28b651e2bd95a8371783002be3f5b653cb644b05a926ffb38bdd6ad1c3d6e87217c2a0caa3cf46c02381c3cdf6ec2c984e2ed802720a2ed709335e03f23a3"
    },
    "logName": "projects/postgresql-302517/logs/run.googleapis.com%2Fstderr",
    "receiveTimestamp": "2022-11-01T14:11:40.949687272Z"
  },
  {
    "insertId": "6361291c000b03b895773b1d",
    "jsonPayload": {
      "message": "finding all child repositories",
      "roots": [
        "us.gcr.io/postgresql-302517/dbt"
      ]
    },
    "resource": {
      "type": "cloud_run_revision",
      "labels": {
        "revision_name": "gcr-cleaner-00006-koj",
        "service_name": "gcr-cleaner",
        "location": "us-central1",
        "project_id": "postgresql-302517",
        "configuration_name": "gcr-cleaner"
      }
    },
    "timestamp": "2022-11-01T14:11:40Z",
    "severity": "DEBUG",
    "labels": {
      "instanceId": "001c9ea28b651e2bd95a8371783002be3f5b653cb644b05a926ffb38bdd6ad1c3d6e87217c2a0caa3cf46c02381c3cdf6ec2c984e2ed802720a2ed709335e03f23a3"
    },
    "logName": "projects/postgresql-302517/logs/run.googleapis.com%2Fstderr",
    "receiveTimestamp": "2022-11-01T14:11:40.949687272Z"
  },
  {
    "insertId": "6361291c000b03c594389721",
    "jsonPayload": {
      "message": "listing child repositories for registry",
      "registry": "us.gcr.io"
    },
    "resource": {
      "type": "cloud_run_revision",
      "labels": {
        "service_name": "gcr-cleaner",
        "configuration_name": "gcr-cleaner",
        "location": "us-central1",
        "revision_name": "gcr-cleaner-00006-koj",
        "project_id": "postgresql-302517"
      }
    },
    "timestamp": "2022-11-01T14:11:40Z",
    "severity": "DEBUG",
    "labels": {
      "instanceId": "001c9ea28b651e2bd95a8371783002be3f5b653cb644b05a926ffb38bdd6ad1c3d6e87217c2a0caa3cf46c02381c3cdf6ec2c984e2ed802720a2ed709335e03f23a3"
    },
    "logName": "projects/postgresql-302517/logs/run.googleapis.com%2Fstderr",
    "receiveTimestamp": "2022-11-01T14:11:40.949687272Z"
  },
  {
    "insertId": "6361291c000dec8cd1299592",
    "jsonPayload": {
      "registry": "us.gcr.io",
      "message": "found child repositories for registry",
      "repos": null
    },
    "resource": {
      "type": "cloud_run_revision",
      "labels": {
        "configuration_name": "gcr-cleaner",
        "revision_name": "gcr-cleaner-00006-koj",
        "service_name": "gcr-cleaner",
        "project_id": "postgresql-302517",
        "location": "us-central1"
      }
    },
    "timestamp": "2022-11-01T14:11:40Z",
    "severity": "DEBUG",
    "labels": {
      "instanceId": "001c9ea28b651e2bd95a8371783002be3f5b653cb644b05a926ffb38bdd6ad1c3d6e87217c2a0caa3cf46c02381c3cdf6ec2c984e2ed802720a2ed709335e03f23a3"
    },
    "logName": "projects/postgresql-302517/logs/run.googleapis.com%2Fstderr",
    "receiveTimestamp": "2022-11-01T14:11:40.949687272Z"
  },
  {
    "insertId": "6361291c000deca3afcc2f1f",
    "jsonPayload": {
      "message": "recursively listed child repositories",
      "in": [
        "us.gcr.io/postgresql-302517/dbt"
      ],
      "out": []
    },
    "resource": {
      "type": "cloud_run_revision",
      "labels": {
        "service_name": "gcr-cleaner",
        "revision_name": "gcr-cleaner-00006-koj",
        "configuration_name": "gcr-cleaner",
        "location": "us-central1",
        "project_id": "postgresql-302517"
      }
    },
    "timestamp": "2022-11-01T14:11:40Z",
    "severity": "DEBUG",
    "labels": {
      "instanceId": "001c9ea28b651e2bd95a8371783002be3f5b653cb644b05a926ffb38bdd6ad1c3d6e87217c2a0caa3cf46c02381c3cdf6ec2c984e2ed802720a2ed709335e03f23a3"
    },
    "logName": "projects/postgresql-302517/logs/run.googleapis.com%2Fstderr",
    "receiveTimestamp": "2022-11-01T14:11:40.949687272Z"
  },
  {
    "insertId": "6361291c000decaddbffda55",
    "jsonPayload": {
      "message": "deleting refs",
      "repos": [],
      "since": "2022-11-01T14:11:40.721676233Z"
    },
    "resource": {
      "type": "cloud_run_revision",
      "labels": {
        "project_id": "postgresql-302517",
        "service_name": "gcr-cleaner",
        "configuration_name": "gcr-cleaner",
        "location": "us-central1",
        "revision_name": "gcr-cleaner-00006-koj"
      }
    },
    "timestamp": "2022-11-01T14:11:40Z",
    "severity": "INFO",
    "labels": {
      "instanceId": "001c9ea28b651e2bd95a8371783002be3f5b653cb644b05a926ffb38bdd6ad1c3d6e87217c2a0caa3cf46c02381c3cdf6ec2c984e2ed802720a2ed709335e03f23a3"
    },
    "logName": "projects/postgresql-302517/logs/run.googleapis.com%2Fstderr",
    "receiveTimestamp": "2022-11-01T14:11:40.949687272Z"
  },
  {
    "insertId": "6361291c000decb5bfad6cd9",
    "jsonPayload": {
      "refs": {},
      "message": "deleted refs"
    },
    "resource": {
      "type": "cloud_run_revision",
      "labels": {
        "project_id": "postgresql-302517",
        "configuration_name": "gcr-cleaner",
        "service_name": "gcr-cleaner",
        "revision_name": "gcr-cleaner-00006-koj",
        "location": "us-central1"
      }
    },
    "timestamp": "2022-11-01T14:11:40Z",
    "severity": "INFO",
    "labels": {
      "instanceId": "001c9ea28b651e2bd95a8371783002be3f5b653cb644b05a926ffb38bdd6ad1c3d6e87217c2a0caa3cf46c02381c3cdf6ec2c984e2ed802720a2ed709335e03f23a3"
    },
    "logName": "projects/postgresql-302517/logs/run.googleapis.com%2Fstderr",
    "receiveTimestamp": "2022-11-01T14:11:40.949687272Z"
  },
  {
    "insertId": "6361291c000e1f71db3a43f6",
    "httpRequest": {
      "requestMethod": "POST",
      "requestUrl": "https://gcr-cleaner-mapvv4hxhq-uc.a.run.app/http",
      "requestSize": "1280",
      "status": 200,
      "responseSize": "749",
      "userAgent": "Google-Cloud-Scheduler",
      "remoteIp": "107.178.194.108",
      "serverIp": "216.239.36.53",
      "latency": "0.513933574s",
      "protocol": "HTTP/1.1"
    },
    "resource": {
      "type": "cloud_run_revision",
      "labels": {
        "revision_name": "gcr-cleaner-00006-koj",
        "project_id": "postgresql-302517",
        "service_name": "gcr-cleaner",
        "configuration_name": "gcr-cleaner",
        "location": "us-central1"
      }
    },
    "timestamp": "2022-11-01T14:11:40.925553Z",
    "severity": "INFO",
    "labels": {
      "instanceId": "001c9ea28b651e2bd95a8371783002be3f5b653cb644b05a926ffb38bdd6ad1c3d6e87217c2a0caa3cf46c02381c3cdf6ec2c984e2ed802720a2ed709335e03f23a3"
    },
    "logName": "projects/postgresql-302517/logs/run.googleapis.com%2Frequests",
    "trace": "projects/postgresql-302517/traces/f2cd2778ca6342173ddfe24fdf0e8be3",
    "receiveTimestamp": "2022-11-01T14:11:40.926907093Z",
    "spanId": "1613091932504172794",
    "traceSampled": true
  }
]

Additional information

No response

sethvargo commented 2 years ago

Hi @ipsingh88

Looking through the logs:

{
  "version": "gcr-cleaner 0.10.0 (103e593b1bf59309841ff4a483cf33f3474acf28, linux/amd64)",
  "message": "server is starting"
}
{
  "port": "8080",
  "message": "server is listening"
}
{
  "message": "starting clean request",
  "version": "gcr-cleaner 0.10.0 (103e593b1bf59309841ff4a483cf33f3474acf28, linux/amd64)"
}
{
  "message": "finding all child repositories",
  "roots": [
    "us.gcr.io/postgresql-302517/dbt"
  ]
}
{
  "message": "listing child repositories for registry",
  "registry": "us.gcr.io"
}
{
  "registry": "us.gcr.io",
  "message": "found child repositories for registry",
  "repos": null
}
{
  "message": "recursively listed child repositories",
  "in": [
    "us.gcr.io/postgresql-302517/dbt"
  ],
  "out": []
}
{
  "message": "deleting refs",
  "repos": [],
  "since": "2022-11-01T14:11:40.721676233Z"
}
{
  "refs": {},
  "message": "deleted refs"
}

It looks like gcr-cleaner is looking for children of us.gcr.io/postgresql-302517/dbt.

  1. What arguments did you invoke the clean request with?
  2. Does that repo exist?
  3. Does it have children?
  4. Does this reproduce using gcr-cleaner-cli?
ipsingh88 commented 2 years ago

The arguments

Hi @ipsingh88

Looking through the logs:

{
  "version": "gcr-cleaner 0.10.0 (103e593b1bf59309841ff4a483cf33f3474acf28, linux/amd64)",
  "message": "server is starting"
}
{
  "port": "8080",
  "message": "server is listening"
}
{
  "message": "starting clean request",
  "version": "gcr-cleaner 0.10.0 (103e593b1bf59309841ff4a483cf33f3474acf28, linux/amd64)"
}
{
  "message": "finding all child repositories",
  "roots": [
    "us.gcr.io/postgresql-302517/dbt"
  ]
}
{
  "message": "listing child repositories for registry",
  "registry": "us.gcr.io"
}
{
  "registry": "us.gcr.io",
  "message": "found child repositories for registry",
  "repos": null
}
{
  "message": "recursively listed child repositories",
  "in": [
    "us.gcr.io/postgresql-302517/dbt"
  ],
  "out": []
}
{
  "message": "deleting refs",
  "repos": [],
  "since": "2022-11-01T14:11:40.721676233Z"
}
{
  "refs": {},
  "message": "deleted refs"
}

It looks like gcr-cleaner is looking for children of us.gcr.io/postgresql-302517/dbt.

  1. What arguments did you invoke the clean request with?
  2. Does that repo exist?
  3. Does it have children?
  4. Does this reproduce using gcr-cleaner-cli?

  1. The arguments I passed are: { "repos": [ "us.gcr.io/postgresql-302517/dbt" ], "recursive": true }

  2. yes the repo exists

3 yes it has children

  1. I could not find any documentation to run gcr-cleaner-cli , can you point me or guide me on that
ipsingh88 commented 2 years ago

All the images exists in following paths: us.gcr.io/postgresql-302517/dbt/master/boosterdbt/ and us.gcr.io/postgresql-302517/dbt/branch_builds//boosterdbt/

sethvargo commented 2 years ago

Hi @ipsingh88

You can download the gcr-cleaner-cli from the releases page.

You should be able to run something like:

GCRCLEANER_LOG=debug gcr-cleaner-cli -repo us.gcr.io/postgresql-302517/dbt -recursive

Could you also post a screenshot of what you see in the Google Cloud Console for this repository?

vicenteguedes commented 2 years ago

Hi @ipsingh88, not sure it is the same issue as I was having right now, but my problem was that my images were tagged, and gcr-cleaner does not delete those by default. So I had to set the flag -tag-filter-any like so:

docker run -v "${HOME}/.config/gcloud:/.config/gcloud" -it us-docker.pkg.dev/gcr-cleaner/gcr-cleaner/gcr-cleaner-cli -grace 720h -repo gcr.io/[MY-PROJECT]/[MY-REPO] -tag-filter-any "^((\d|\w){8})|(\d+\.\d+\.\d+)$"

ipsingh88 commented 2 years ago

Running the docker command locally works and deleted all the required images but when scheduled via cloudn Run it is not working

sethvargo commented 2 years ago

Hi @ipsingh88 - that could be:

  1. Using an old container image on Cloud Run. Try deleting the service and re-deploying with the latest image. Enable debug logging.
  2. The Cloud Run service's service account does not have the proper permissions.
  3. The payload sent to the service is malformed.
ipsingh88 commented 2 years ago

Thanks @sethvargo it seems we need Storage Admin role added to service account