search

carvel-dev / vendir

Easy way to vendor portions of git repos, github releases, helm charts, docker image contents, etc. declaratively
https://carvel.dev/vendir
Apache License 2.0
269 stars 47 forks source link

Sync Cache that relies on release versions #274

Closed fritzduchardt closed 9 months ago

fritzduchardt commented 1 year ago

Describe the problem/challenge you have We have developed a deployment software that leverages vendir to pull Helm Charts, OCI Images and Git Repos for hundreds of applications. Every time the software runs, it executes vendir sync for each application to pick up changes in case the corresponding vendir.yaml was modified. This slows down the process tremendously and is mostly unnecessary, since we usually only modify one vendir.yaml at the time leaving the other untouched.

Describe the solution you'd like The issue could be solved by an optional synchronization mode that only synchronizes, if the last vendored files are not in line with the release versions set in the corresponding vendir.yaml.

Possibly this feature could be using the lock file as a record of the versions last synced and a re-sync only, if the tags in the lock file diverge from the refs in vendir.yaml.

Also, on could ensure that caching only takes effect for refs that follow semantic version or other standardized version formats.

Naturally, caching only works, if releases are stable, which is the case with our applications. If they were not, we would see this as an issue with the release process rather than vendir.

Vote on this request

This is an invitation to the community to vote on issues, to help us prioritize our backlog. Use the "smiley face" up to the right of this comment to vote.

👍 "I would like to see this addressed as soon as possible" 👎 "There are other more important things to focus on right now"

We are also happy to receive and review Pull Requests if you want to help working on this issue.

100mik commented 1 year ago

Caching does need to be explicitly enabled and would only benefit OCI Images using shasum references today. The thoughts around this are documented here: https://carvel.dev/vendir/docs/v0.34.x/sync/#caching

100mik commented 1 year ago

I think a fundamental rule that we are trying to follow is that we want to cache resources if they are referred to in an immutable manner.

kumaritanushree commented 1 year ago

Trying to understand the problem/your use case better:

You have a software deployed which leverages vendir to pull contents (in your use case it is pulling helm charts, OCI images and git repos) for hundreds of application.

Every time the software runs, it executes vendir sync for each application to pick up changes in case the corresponding vendir.yaml was modified.

Does your application have logic to check if vendir.yaml is modified and then only run vendir sync for that? Because as per my understanding vendir does not check either vendir.yaml has modified or not. It just read the file and do its job.

This slows down the process tremendously and is mostly unnecessary, since we usually only modify one vendir.yaml at the time leaving the other untouched.

What I can understand is, this is getting slow down as it is trying to sync contents for all of your applications but your expectation is that it should try to sync only for modified vendir.yaml.

@fritzduchardt can you please help me either I understood your use case and problem correctly or not?

github-actions[bot] commented 11 months ago

This issue is being marked as stale due to a long period of inactivity and will be closed in 5 days if there is no response.

github-actions[bot] commented 9 months ago

This issue is being marked as stale due to a long period of inactivity and will be closed in 5 days if there is no response.

Zebradil commented 9 months ago

For the record: this issue was superseded by #278 and is being worked on in #279.