search

snok / container-retention-policy

GitHub action for pruning old GHCR container image versions.
MIT License
178 stars 29 forks source link

V3 Failed to fetch packages with temporal token #96

Open ricardo-s-ferreira-alb opened 1 month ago

ricardo-s-ferreira-alb commented 1 month ago

when using a list of images (separeted by comma or space) I get an error. if the configuration has only one image name it runs sucessully.

Here a sample of falling:

        uses:  snok/container-retention-policy@v3.0.0
        with:
          account: xxxx
          image-names: "xxxx/aaaa,xxxx/bbbb"
          image-tags: "!*.*.*-r*"
          cut-off: 1d
          keep-n-most-recent: 3
          token: ${{ secrets.GITHUB_TOKEN }}
          dry-run: false
          rust-log: debug

2024-08-06T18:21:19.085046Z DEBUG container_retention_policy: Logging initialized
2024-08-06T18:21:19.085528Z DEBUG parse input: container_retention_policy::cli::models: Recognized token as temporal token
2024-08-06T18:21:19.085781Z DEBUG parse input: container_retention_policy::client::builder: Constructing base urls
2024-08-06T18:21:19.085818Z DEBUG parse input: container_retention_policy::client::builder: Constructing HTTP headers
2024-08-06T18:21:19.085845Z DEBUG parse input: container_retention_policy::client::builder: Creating rate-limited services
2024-08-06T18:21:19.086147Z DEBUG fetch rate limit: container_retention_policy::client::client: Retrieving Github API rate limit
2024-08-06T18:21:19.086253Z DEBUG fetch rate limit: reqwest::connect: starting new connection: https://api.github.com/    
2024-08-06T18:21:19.086456Z DEBUG hyper_util::client::legacy::connect::dns: resolving host="api.github.com"
2024-08-06T18:21:19.106598Z DEBUG fetch rate limit: hyper_util::client::legacy::connect::http: connecting to 140.82.121.6:443
2024-08-06T18:21:19.[15](https://github.com/xxxxx/sigo/actions/runs/xxx/job/xxx#step:4:16)3514Z DEBUG fetch rate limit: hyper_util::client::legacy::connect::http: connected to 140.82.121.6:443
2024-08-06T18:21:19.417357Z DEBUG fetch rate limit: hyper_util::client::legacy::pool: pooling idle connection for ("https", api.github.com)
2024-08-06T18:21:19.417477Z DEBUG fetch rate limit: container_retention_policy::client::client: There are 5000 requests remaining in the rate limit
2024-08-06T18:21:19.496825Z DEBUG select packages: container_retention_policy::client::client: Fetching package from https://api.github.com/orgs/xxxxx/packages/container/xx%2Fxxxx
2024-08-06T18:21:19.496984Z DEBUG select packages: reqwest::connect: starting new connection: https://api.github.com/    
2024-08-06T18:21:19.497115Z DEBUG hyper_util::client::legacy::connect::dns: resolving host="api.github.com"
2024-08-06T18:21:19.509532Z DEBUG select packages: hyper_util::client::legacy::connect::http: connecting to 140.82.121.6:443
2024-08-06T18:21:19.56[16](https://github.com/xxxxx/sigo/actions/runs/1xxxx/job/xxx#step:4:17)42Z DEBUG select packages: hyper_util::client::legacy::connect::http: connected to 140.82.121.6:443
2024-08-06T18:21:19.869385Z DEBUG select packages: hyper_util::client::legacy::pool: pooling idle connection for ("https", api.github.com)
thread 'main' panicked at src/client/client.rs:50:[18](https://github.com/xxxx/sigo/actions/runs/xx/job/xxxx#step:4:19):
Failed to fetch packages: missing field `id` at line 1 column 150
Location:
    src/client/client.rs:294:12
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
sondrelg commented 1 month ago

Thanks for the report! Are you able to run this, in a workflow?

- run: |
    curl -L \
        -H "Accept: application/vnd.github+json" \
        -H "Authorization: Bearer ${{ secrets.GITHUB_TOKEN }}" \
        -H "X-GitHub-Api-Version: 2022-11-28" \
        https://api.github.com/orgs/{{ account }}/packages/container/{{ image1 }}
- run: |
    curl -L \
        -H "Accept: application/vnd.github+json" \
        -H "Authorization: Bearer ${{ secrets.GITHUB_TOKEN }}" \
        -H "X-GitHub-Api-Version: 2022-11-28" \
        https://api.github.com/orgs/{{ account }}/packages/container/{{ image2 }}

(replace /orgs/{{ account }} with /{{ username }} if it's a personal account)

For each of your images?

You should get a response like this:

{
  "id": 6611266,
  "name": "container-retention-policy",
  "package_type": "container",
  "owner": {
    "login": "snok",
    "id": 64945977,
    "node_id": "MDEyOk9yZ2FuaXphdGlvbjY0OTQ1OTc3",
    "avatar_url": "https://avatars.githubusercontent.com/u/64945977?v=4",
    "gravatar_id": "",
    "url": "https://api.github.com/users/snok",
    "html_url": "https://github.com/snok",
    "followers_url": "https://api.github.com/users/snok/followers",
    "following_url": "https://api.github.com/users/snok/following{/other_user}",
    "gists_url": "https://api.github.com/users/snok/gists{/gist_id}",
    "starred_url": "https://api.github.com/users/snok/starred{/owner}{/repo}",
    "subscriptions_url": "https://api.github.com/users/snok/subscriptions",
    "organizations_url": "https://api.github.com/users/snok/orgs",
    "repos_url": "https://api.github.com/users/snok/repos",
    "events_url": "https://api.github.com/users/snok/events{/privacy}",
    "received_events_url": "https://api.github.com/users/snok/received_events",
    "type": "Organization",
    "site_admin": false
  },
  "version_count": 20,
  "visibility": "public",
  "url": "https://api.github.com/orgs/snok/packages/container/container-retention-policy",
  "created_at": "2024-05-27T20:20:13Z",
  "updated_at": "2024-08-07T08:27:09Z",
  "repository": {
    "id": 395465327,
    "node_id": "MDEwOlJlcG9zaXRvcnkzOTU0NjUzMjc=",
    "name": "container-retention-policy",
    "full_name": "snok/container-retention-policy",
    "private": false,
    "owner": {
      "login": "snok",
      "id": 64945977,
      "node_id": "MDEyOk9yZ2FuaXphdGlvbjY0OTQ1OTc3",
      "avatar_url": "https://avatars.githubusercontent.com/u/64945977?v=4",
      "gravatar_id": "",
      "url": "https://api.github.com/users/snok",
      "html_url": "https://github.com/snok",
      "followers_url": "https://api.github.com/users/snok/followers",
      "following_url": "https://api.github.com/users/snok/following{/other_user}",
      "gists_url": "https://api.github.com/users/snok/gists{/gist_id}",
      "starred_url": "https://api.github.com/users/snok/starred{/owner}{/repo}",
      "subscriptions_url": "https://api.github.com/users/snok/subscriptions",
      "organizations_url": "https://api.github.com/users/snok/orgs",
      "repos_url": "https://api.github.com/users/snok/repos",
      "events_url": "https://api.github.com/users/snok/events{/privacy}",
      "received_events_url": "https://api.github.com/users/snok/received_events",
      "type": "Organization",
      "site_admin": false
    },
    "html_url": "https://github.com/snok/container-retention-policy",
    "description": "GitHub action for pruning old GHCR container image versions.",
    "fork": false,
    "url": "https://api.github.com/repos/snok/container-retention-policy",
    "forks_url": "https://api.github.com/repos/snok/container-retention-policy/forks",
    "keys_url": "https://api.github.com/repos/snok/container-retention-policy/keys{/key_id}",
    "collaborators_url": "https://api.github.com/repos/snok/container-retention-policy/collaborators{/collaborator}",
    "teams_url": "https://api.github.com/repos/snok/container-retention-policy/teams",
    "hooks_url": "https://api.github.com/repos/snok/container-retention-policy/hooks",
    "issue_events_url": "https://api.github.com/repos/snok/container-retention-policy/issues/events{/number}",
    "events_url": "https://api.github.com/repos/snok/container-retention-policy/events",
    "assignees_url": "https://api.github.com/repos/snok/container-retention-policy/assignees{/user}",
    "branches_url": "https://api.github.com/repos/snok/container-retention-policy/branches{/branch}",
    "tags_url": "https://api.github.com/repos/snok/container-retention-policy/tags",
    "blobs_url": "https://api.github.com/repos/snok/container-retention-policy/git/blobs{/sha}",
    "git_tags_url": "https://api.github.com/repos/snok/container-retention-policy/git/tags{/sha}",
    "git_refs_url": "https://api.github.com/repos/snok/container-retention-policy/git/refs{/sha}",
    "trees_url": "https://api.github.com/repos/snok/container-retention-policy/git/trees{/sha}",
    "statuses_url": "https://api.github.com/repos/snok/container-retention-policy/statuses/{sha}",
    "languages_url": "https://api.github.com/repos/snok/container-retention-policy/languages",
    "stargazers_url": "https://api.github.com/repos/snok/container-retention-policy/stargazers",
    "contributors_url": "https://api.github.com/repos/snok/container-retention-policy/contributors",
    "subscribers_url": "https://api.github.com/repos/snok/container-retention-policy/subscribers",
    "subscription_url": "https://api.github.com/repos/snok/container-retention-policy/subscription",
    "commits_url": "https://api.github.com/repos/snok/container-retention-policy/commits{/sha}",
    "git_commits_url": "https://api.github.com/repos/snok/container-retention-policy/git/commits{/sha}",
    "comments_url": "https://api.github.com/repos/snok/container-retention-policy/comments{/number}",
    "issue_comment_url": "https://api.github.com/repos/snok/container-retention-policy/issues/comments{/number}",
    "contents_url": "https://api.github.com/repos/snok/container-retention-policy/contents/{+path}",
    "compare_url": "https://api.github.com/repos/snok/container-retention-policy/compare/{base}...{head}",
    "merges_url": "https://api.github.com/repos/snok/container-retention-policy/merges",
    "archive_url": "https://api.github.com/repos/snok/container-retention-policy/{archive_format}{/ref}",
    "downloads_url": "https://api.github.com/repos/snok/container-retention-policy/downloads",
    "issues_url": "https://api.github.com/repos/snok/container-retention-policy/issues{/number}",
    "pulls_url": "https://api.github.com/repos/snok/container-retention-policy/pulls{/number}",
    "milestones_url": "https://api.github.com/repos/snok/container-retention-policy/milestones{/number}",
    "notifications_url": "https://api.github.com/repos/snok/container-retention-policy/notifications{?since,all,participating}",
    "labels_url": "https://api.github.com/repos/snok/container-retention-policy/labels{/name}",
    "releases_url": "https://api.github.com/repos/snok/container-retention-policy/releases{/id}",
    "deployments_url": "https://api.github.com/repos/snok/container-retention-policy/deployments"
  },
  "html_url": "https://github.com/orgs/snok/packages/container/package/container-retention-policy"
}

Which should be parsable by our Package representation:

#[derive(Debug, Clone, Deserialize)]
pub struct Package {
    pub id: u32,
    pub name: String,
    pub created_at: DateTime<Utc>,
    pub updated_at: Option<DateTime<Utc>>,
}

The error message you get, seems to indicate that you're not able to fetch the data properly, but let's make sure 🙂

csc-felipe commented 1 month ago

This helped me!

I had the same error message. Using the curl command above I got a 404, which made me notice that there was an error in the image name, it had the repository/image format.

cswilliams commented 1 month ago

hmm, I'm also struggling to get this to work. I've created a new github app with these permissions: 

 Read access to code and metadata
 Read and write access to packages

I've installed it in my org and given it access to all repos. I also checked that my packages have "admin" role on the repo running the workflow. Here's the workflow I'm using:

      - name: Generate a token
        id: generate-token
        uses: actions/create-github-app-token@31c86eb3b33c9b601a1f60f98dcbfd1d70f379b4 # v1.10.3
        with:
          app-id: ${{ vars.PKG_CLEANUP_APP_ID }}
          private-key: ${{ secrets.PKG_CLEANUP_APP_PRIVATE_KEY }}
      - name: Cleanup
        uses: snok/container-retention-policy@4f22ef80902ad409ed55a99dc5133cc1250a0d03 # v3.0.0
        with:
          image-names: core_base
          token: ${{ steps.generate-token.outputs.token }}
          cut-off: 4w 
          timestamp-to-use: updated_at
          account: ${{ github.repository_owner }}
          tag-selection: untagged
          dry-run: true

and I get: 

thread 'main' panicked at src/client/client.rs:50:18:
Failed to fetch packages: missing field `id` at line 1 column [15](https://github.com/my-org/my-repo/actions/runs/10326702647/job/28590674861#step:5:15)0
Location:
    src/client/client.rs:294:12
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

I tried your troubleshooting step above putting this between the generate-token and the container-retention-policy: 

      - run: |
          curl -L \
            -H "Accept: application/vnd.github+json" \
            -H "Authorization: Bearer ${{ steps.generate-token.outputs.token }}" \
            -H "X-GitHub-Api-Version: 2022-11-28" \
            https://api.github.com/orgs/my-org/packages/container/core_base

and I get:

{
  "message": "Package not found.",
  "documentation_url": "https://docs.github.com/rest/packages/packages#get-a-package-for-an-organization",
  "status": "404"
}

However, if I make this same curl request from my local machine using a PAT with package:read I get a full response showing the package information and don't get a 404. Any ideas what I'm doing wrong? I'm guessing it must be some permissions issue with the app, but I've already granted it RW packages, RO metadata. I also tried adding read only to contents to see if that would help, but no luck. Thanks!

cswilliams commented 1 month ago

I was able to get it working after swapping to the temporal token instead of the github app installation access token. I tried a bunch of different things (recreated and reinstalled the app several times, tried enabling every permission, tried setting repo and permissions explicitly when generating the access token, but no matter what I tried, I would always get a 404 when trying to list packages. I wonder if it's some issue on the github side. Their api docs seem to suggest that an installation access token should work, but who knows. At any rate, the temporal token works for me and is less setup than the app based token anyway. I probably should have just started with that all along!

sondrelg commented 1 month ago

How strange. The workflow and permissions seem correct, according to how I've set it up during testing. The image names do need to be percent encoded, if you've got, e.g., a / in the image name. Underscores should be OK, I think 🤔

sscheib commented 3 weeks ago

I seem to hit the same issue, but with both the temporal GH Token and a personal token.

While debugging this, I found that the following URLs worked:

Whereas the suggested URL https://api.github.com/sscheib/packages/container/jekyll-build yields a 404.

I have no idea what's causing this or whether GitHub changed the API in the meanwhile.

// EDIT

Totally my fault. I misunderstood the account variable. I put in my username 😆

sondrelg commented 3 weeks ago

I seem to hit the same issue, but with both the temporal GH Token and a personal token.

While debugging this, I found that the following URLs worked:

* https://api.github.com/users/sscheib/packages/container/jekyll-build

* https://api.github.com/user/packages/container/jekyll-build

Whereas the suggested URL https://api.github.com/sscheib/packages/container/jekyll-build yields a 404.

I have no idea what's causing this or whether GitHub changed the API in the meanwhile.

// EDIT

Totally my fault. I misunderstood the account variable. I put in my username 😆

Hehe yeah for potential readers, specifying anything other than user for the username implicitly means you're telling the action you're an organization. I can definitely see how, while it makes for a compressed interface for the action, this might become a source of confusion 😅