search

GoogleCloudPlatform / gcr-cleaner

Delete untagged image refs in Google Container Registry or Artifact Registry
Apache License 2.0
805 stars 112 forks source link

Not deleting image versions in artifact registry #74

Closed dsnodgrass45 closed 2 years ago

dsnodgrass45 commented 2 years ago

I have setup gcr-cleaner in Cloud Run using us-docker.pkg.dev/gcr-cleaner/gcr-cleaner/gcr-cleaner. The Cloud Scheduler job is passing the following payload: 

{"dry_run":false,"keep":1,"recursive":true,"repos":"us-central1-docker.pkg.dev/my-project/my-repo"}

When executed I see the following info/debug logs:

{
  "insertId": "6244a38d000d46f4fc3176b4",
  "jsonPayload": {
    "repos": [
      "us-central1-docker.pkg.dev/my-project/my-repo",
      "us-central1-docker.pkg.dev/my-project/my-repo/my-image"
    ],
    "since": "2022-03-30T18:38:05.663332949Z",
    "message": "deleting refs"
  },
  "resource": {
    "type": "cloud_run_revision",
    "labels": {
      "project_id": "my-project",
      "location": "us-central1",
      "configuration_name": "gcr-cleaner",
      "revision_name": "gcr-cleaner-5gwql",
      "service_name": "gcr-cleaner"
    }
  },
  "timestamp": "2022-03-30T18:38:05Z",
  "severity": "INFO",
  "labels": {
    "instanceId": "00bf4bf02d8c102af0b010b37017868833bc2c7cbb65d488b4575d79e0465be497d6206d279f9acaa7f8451534764f72b401fd3ada22dd770632deee5644"
  },
  "logName": "projects/my-project/logs/run.googleapis.com%2Fstdout",
  "receiveTimestamp": "2022-03-30T18:38:05.874229456Z"
}
{
  "insertId": "6244a38d000d4721a31029b8",
  "jsonPayload": {
    "message": "deleting refs for repo",
    "repo": "us-central1-docker.pkg.dev/my-project/my-repo"
  },
  "resource": {
    "type": "cloud_run_revision",
    "labels": {
      "project_id": "my-project",
      "location": "us-central1",
      "service_name": "gcr-cleaner",
      "revision_name": "gcr-cleaner-5gwql",
      "configuration_name": "gcr-cleaner"
    }
  },
  "timestamp": "2022-03-30T18:38:05Z",
  "severity": "DEBUG",
  "labels": {
    "instanceId": "00bf4bf02d8c102af0b010b37017868833bc2c7cbb65d488b4575d79e0465be497d6206d279f9acaa7f8451534764f72b401fd3ada22dd770632deee5644"
  },
  "logName": "projects/my-project/logs/run.googleapis.com%2Fstdout",
  "receiveTimestamp": "2022-03-30T18:38:05.874229456Z"
}
{
  "insertId": "6244a38d000f36260cc14f75",
  "jsonPayload": {
    "message": "deleting refs for repo",
    "repo": "us-central1-docker.pkg.dev/my-project/my-repo/my-image"
  },
  "resource": {
    "type": "cloud_run_revision",
    "labels": {
      "revision_name": "gcr-cleaner-5gwql",
      "project_id": "my-proj",
      "configuration_name": "gcr-cleaner",
      "location": "us-central1",
      "service_name": "gcr-cleaner"
    }
  },
  "timestamp": "2022-03-30T18:38:05Z",
  "severity": "DEBUG",
  "labels": {
    "instanceId": "00bf4bf02d8c102af0b010b37017868833bc2c7cbb65d488b4575d79e0465be497d6206d279f9acaa7f8451534764f72b401fd3ada22dd770632deee5644"
  },
  "logName": "projects/my-project/logs/run.googleapis.com%2Fstdout",
  "receiveTimestamp": "2022-03-30T18:38:06.207204185Z"
}
{
  "insertId": "1t7yjlafjlvxp9",
  "jsonPayload": {
    "url": "https://gcr-cleaner-ip4job3fda-uc.a.run.app/http",
    "targetType": "HTTP",
    "@type": "type.googleapis.com/google.cloud.scheduler.logging.AttemptFinished",
    "jobName": "projects/my-project/locations/us-central1/jobs/gcr-job"
  },
  "httpRequest": {
    "status": 200
  },
  "resource": {
    "type": "cloud_scheduler_job",
    "labels": {
      "job_id": "gcr-job",
      "location": "us-central1",
      "project_id": "my-project"
    }
  },
  "timestamp": "2022-03-30T18:38:06.127451233Z",
  "severity": "INFO",
  "logName": "projects/my-project/logs/cloudscheduler.googleapis.com%2Fexecutions",
  "receiveTimestamp": "2022-03-30T18:38:06.127451233Z"
}

I have three versions in us-central1-docker.pkg.dev/my-project/my-repo/my-image. One is tagged latest and the other two are untagged. Even with the argument to keep:1 and the logs showing 200 OK none of the images are being pruned. Any help would be appreciated.

goaliejordan commented 2 years ago

I am facing the same issue. I tested locally on the gcr-cleaner-cli and the default of cleaning all un-tagged images is not working. To add more to this though, when I put a filter on such as --tag-filter-any "^remove.+$" the cleaner can successfully remove all of those images with the remove tag.

sethvargo commented 2 years ago

Hi @dsnodgrass45 - can you please share a screenshot of your Artifact Registry layout? If you could switch to using the CLI with debug mode, we should see more output.

dsnodgrass45 commented 2 years ago
Image_Level Repo_Level
dsnodgrass45 commented 2 years ago

@sethvargo I have added images of the repo/image space. Please note that the logs were changed to not show the actual project/repo names. The images show the real ones but wanted you to know the actual logs match the real names.

sethvargo commented 2 years ago

What is the output of using the CLI:

gcr-cleaner -repo us-central1-docker.pkg.dev/snod-prod/automate-pipeline -dry-run -keep 1 -recursive
dsnodgrass45 commented 2 years ago 
> docker run -it us-docker.pkg.dev/gcr-cleaner/gcr-cleaner/gcr-cleaner-cli
Unable to find image 'us-docker.pkg.dev/gcr-cleaner/gcr-cleaner/gcr-cleaner-cli:latest' locally
latest: Pulling from gcr-cleaner/gcr-cleaner/gcr-cleaner-cli
01a0d6d9dcc9: Pull complete
53ebec517518: Pull complete
Digest: sha256:cb94d3cd9c3c52b9db008805079da33e3c5e02740de230879a783e09754b7093
Status: Downloaded newer image for us-docker.pkg.dev/gcr-cleaner/gcr-cleaner/gcr-cleaner-cli:latest
missing -repo
sethvargo commented 2 years ago

Hi @dsnodgrass45 could you please run the command:

gcr-cleaner -repo us-central1-docker.pkg.dev/snod-prod/automate-pipeline -dry-run -keep 1 -recursive

and paste the result?

dsnodgrass45 commented 2 years ago

@sethvargo I'm having the above issue of "missing -repo" when trying to run gcr-cleaner-cli locally.

sethvargo commented 2 years ago

If you're trying to use it via Docker:

docker run -it us-docker.pkg.dev/gcr-cleaner/gcr-cleaner/gcr-cleaner-cli -- -repo us-central1-docker.pkg.dev/snod-prod/automate-pipeline -dry-run -keep 1 -recursive
dsnodgrass45 commented 2 years ago

Here is my output from running it locally. Sorry for the delay.

> ./gcr-cleaner-cli -repo us-central1-docker.pkg.dev/snod-prod/automate-pipeline -dry-run -keep 1 -recursive
WARNING: Running in dry-run mode - nothing will actually be cleaned!

Deleting refs older than 2022-04-03T21:21:17Z on 2 repo(s)...

us-central1-docker.pkg.dev/snod-prod/automate-pipeline
  ✗ no refs were deleted

us-central1-docker.pkg.dev/snod-prod/automate-pipeline/http-doom
  ✗ no refs were deleted
sethvargo commented 2 years ago

Hi @dsnodgrass45

Thank you for the reply. Does the currently authenticated user have permission to list on those repos? What is the output of:

gcloud artifacts packages list --location us-central1 --repository automate-pipeline --project snod-prod
dsnodgrass45 commented 2 years ago

The user does have permissions for list.

> gcloud artifacts packages list --location us-central1 --repository automate-pipeline --project snod-prod
Listing items under project snod-prod, location us-central1, repository automate-pipeline.

PACKAGE    CREATE_TIME          UPDATE_TIME
http-doom  2022-03-28T14:34:13  2022-04-02T18:00:02
dsnodgrass45 commented 2 years ago

@sethvargo the gcr-cleaner service account has roles/artifactretistry.repoAdmin which includes both .writer and .reader (allows listing) of the repository.

sethvargo commented 2 years ago

Hi @dsnodgrass45 - thank you for your patience here. I just pushed up v0.7.2 which includes a lot more logging information that should point us towards any bugs. Could you please download v0.7.2 and run:

GCRCLEANER_LOG=debug ./gcr-cleaner-cli -repo us-central1-docker.pkg.dev/snod-prod/automate-pipeline -dry-run -keep 1 -recursive
dsnodgrass45 commented 2 years ago

@sethvargo got the new version going locally. Here is the desired output.

> GCRCLEANER_LOG=debug ./gcr-cleaner-cli -repo us-central1-docker.pkg.dev/snod-prod/automate-pipeline -dry-run -keep 1 -recursive
{"message":"using default token resolution for authentication","severity":"DEBUG","time":"2022-04-06T21:04:17Z"}
{"message":"gathering child repositories recursively","severity":"DEBUG","time":"2022-04-06T21:04:17Z"}
WARNING: Running in dry-run mode - nothing will actually be cleaned!

Deleting refs older than 2022-04-06T21:04:17Z on 2 repo(s)...

us-central1-docker.pkg.dev/snod-prod/automate-pipeline
{"message":"computed repo","repo":"us-central1-docker.pkg.dev/snod-prod/automate-pipeline","severity":"DEBUG","time":"2022-04-06T21:04:18Z"}
  ✗ no refs were deleted

us-central1-docker.pkg.dev/snod-prod/automate-pipeline/http-doom
{"message":"computed repo","repo":"us-central1-docker.pkg.dev/snod-prod/automate-pipeline/http-doom","severity":"DEBUG","time":"2022-04-06T21:04:19Z"}
{"digest":"sha256:cc34637cef290a07e0944bc272c1ce450ced2075d7903a438298af012bfd27a1","message":"processing manifest","repo":"us-central1-docker.pkg.dev/snod-prod/automate-pipeline/http-doom","severity":"DEBUG","tags":[],"time":"2022-04-06T21:04:19Z","uploaded":"2022-03-28T14:34:13-05:00"}
{"digest":"sha256:cc34637cef290a07e0944bc272c1ce450ced2075d7903a438298af012bfd27a1","message":"should delete","reason":"no tags","repo":"us-central1-docker.pkg.dev/snod-prod/automate-pipeline/http-doom","severity":"DEBUG","time":"2022-04-06T21:04:19Z"}
{"digest":"sha256:cc34637cef290a07e0944bc272c1ce450ced2075d7903a438298af012bfd27a1","keep":1,"keep_count":0,"message":"skipping deletion because of keep count","repo":"us-central1-docker.pkg.dev/snod-prod/automate-pipeline/http-doom","severity":"DEBUG","time":"2022-04-06T21:04:19Z"}
{"digest":"sha256:1d620b118edf50a371a78f73af2fa23b38f592f73b9324f7bfda1ba3c3d7ec15","message":"processing manifest","repo":"us-central1-docker.pkg.dev/snod-prod/automate-pipeline/http-doom","severity":"DEBUG","tags":["latest"],"time":"2022-04-06T21:04:19Z","uploaded":"2022-03-30T12:30:07-05:00"}
{"digest":"sha256:1d620b118edf50a371a78f73af2fa23b38f592f73b9324f7bfda1ba3c3d7ec15","message":"should not delete","reason":"no filter matches","repo":"us-central1-docker.pkg.dev/snod-prod/automate-pipeline/http-doom","severity":"DEBUG","time":"2022-04-06T21:04:19Z"}
  ✗ no refs were deleted
sethvargo commented 2 years ago

Hi @dsnodgrass45

Thank you for the log output. Does the b9b44... image still exist from your screenshot above?

Here's what I'm seeing:

Therefore, no references are deleted.

dsnodgrass45 commented 2 years ago

@sethvargo so running it locally is cleaning up the artifact registry repo as designed. I'm redeploying v0.7.2 and will try with cloud scheduler/run again and see what the results are.

dsnodgrass45 commented 2 years ago

@sethvargo my initial testing with v0.7.3 is working now! Greatly appreciate your work on this and your patience to get this working.

sethvargo commented 2 years ago

Thanks!