search

kyma-project / test-infra

Test infrastructure for the Kyma project.
https://status.build.kyma-project.io/
Apache License 2.0
38 stars 181 forks source link

Implement Image Syncer Job as GitHub Action #11384

Open akiioto opened 2 months ago

akiioto commented 2 months ago

Description

Migrate the image-syncer job from Prow to GitHub Actions by creating a reusable workflow. This migration is necessary to address the issue of exceeding DockerHub's 100 requests per hour limit and to decentralize the maintenance of image synchronization. Each team will manage their own external-images.yaml file. Detailed instructions will be provided to ensure a smooth transition. The reusable workflow will run on GitHub Actions, leveraging different IP addresses to mitigate rate limit issues.

The reusable workflow enables team to use their own secret to access Docker Hub. If no secret is provided, the anonymous download will be used. The credentials to the registry have to be restricted to the reusable workflow and not be shared with the teams.

Reasons

  1. Decentralized Maintenance: Allowing each team to manage their own configuration file and credentials reduces the central team's burden and increases autonomy for teams.
  2. Scalability: GitHub Actions runners use different IP addresses, which can help distribute the load and avoid rate limits.
  3. Efficiency: Teams can approve their own repository changes, streamlining the process and reducing review times for the central team.

Postponed, as probably dockerhub will not be an issue with GitHub as a runner. Required to do performance test and decide Rate Limit Issue: The current Prow-based solution frequently hits DockerHub's 100 requests per hour limit, causing failures.

Acceptance Criteria

Attachments

KacperMalachowski commented 1 month ago

The're is a agreement between Docker and GIthub: https://github.com/actions/runner-images/issues/1445#issuecomment-713861495

Agreement is already made: https://github.com/actions/runner-images/issues/1445#issuecomment-729675520

dekiel commented 3 weeks ago

We've made a decision to define a common location for .yml file with definition of images to sync. The users will not have a way to define their own location. Having files in the same location in every repository using it, will let us collect information for these files or react on changes in the list of images to sync. Implementing such change in the future would require all teams align their tooling and file to common location.

dekiel commented 3 weeks ago

An amd64Only flag in image-syncer does not prevent syncing image index or image manifest for different architecture. At present a flag just enable syncing image manifest if a source image reference points to it. @Sawthis would like to remove this flag. This will be planned in other issue as this is not a blocker for migration.