concourse / github-release-resource

a resource for github releases
Apache License 2.0
57 stars 58 forks source link

Feature: Invalidate caches when a put is done to a version #118

Open stanislav-zaprudskiy opened 2 years ago

stanislav-zaprudskiy commented 2 years ago

The idea behind the following pipeline is that each subsequent job updates the body of the same pre-release until the final job does its update and makes that pre-release to become a normal release:

pipeline.yaml ```yaml resources: - name: github-pre-release type: github-release source: owner: foo repository: bar access_token: ((token)) release: false pre_release: true - name: github-release type: github-release source: owner: foo repository: bar access_token: ((token)) release: true pre_release: false jobs: - name: job1 serial: true plan: - get: github-pre-release trigger: true - task: update-release-body config: platform: linux image_resource: type: docker-image source: repository: alpine inputs: - name: github-pre-release outputs: - name: github-pre-release run: path: /bin/sh args: - -c - | echo "this is an update from job1" >> github-pre-release/body - put: github-pre-release inputs: - github-pre-release params: name: github-pre-release/tag tag: github-pre-release/tag commitish: github-pre-release/commit_sha body: github-pre-release/body - name: job2 serial: true plan: - get: github-pre-release trigger: true passed: - job1 - task: update-release-body config: platform: linux image_resource: type: docker-image source: repository: alpine inputs: - name: github-pre-release outputs: - name: github-pre-release run: path: /bin/sh args: - -c - | echo "this is an update from job2" >> github-pre-release/body - put: github-pre-release inputs: - github-pre-release params: name: github-pre-release/tag tag: github-pre-release/tag commitish: github-pre-release/commit_sha body: github-pre-release/body - name: job3 serial: true plan: - get: github-pre-release trigger: true passed: - job2 - task: update-release-body config: platform: linux image_resource: type: docker-image source: repository: alpine inputs: - name: github-pre-release outputs: - name: github-pre-release run: path: /bin/sh args: - -c - | echo "this is an update from job3" >> github-pre-release/body - put: github-pre-release inputs: - github-pre-release params: name: github-pre-release/tag tag: github-pre-release/tag commitish: github-pre-release/commit_sha body: github-pre-release/body - name: job4 serial: true plan: - get: github-pre-release trigger: true passed: - job3 - task: update-release-body config: platform: linux image_resource: type: docker-image source: repository: alpine inputs: - name: github-pre-release outputs: - name: github-pre-release run: path: /bin/sh args: - -c - | echo "this is a final update from job4" >> github-pre-release/body - put: github-release inputs: - github-release params: name: github-pre-release/tag tag: github-pre-release/tag commitish: github-pre-release/commit_sha body: github-pre-release/body ```

However, it doesn't work as expected. The subsequent jobs don't get the updated body of the release most of the time (depending on cache accessibility/worker used, afaics), which results into a body like this:

this is an update from job2
this is a final update from job4

(in various combinations, but never a complete one from all 4 jobs).

I believe it matches the explanation from https://github.com/concourse/github-release-resource/issues/61#issuecomment-360503706:

This is working as expected of the caching semantics. When a resource version changes in-place, there's no way for Concourse to know that that same version is technically different now. I don't think it's an issue specific to this resource, but you could open a general issue to invalidate caches when a put is done to a version; that may be a route we can explore.

And is also similar to https://github.com/concourse/github-release-resource/issues/69.

Would storing more release metadata in Concourse solve the problem? E.g. not only the id, tag and timestamp, but also body, commit_sha, etc. Or perhaps some extra timestamp/hash value of the result of the last update of a resource by Concourse's put.

Concourse version: 7.6.0, resource version: bundled with Concourse (v1.6.4).

stanislav-zaprudskiy commented 2 years ago

I also tried adding more passed dependencies for the subsequent jobs, but that doesn't make any difference

diff ```diff @@ -89,12 +89,13 @@ - name: job3 serial: true plan: - get: github-pre-release trigger: true passed: + - job1 - job2 - task: update-release-body config: platform: linux image_resource: @@ -124,12 +125,14 @@ - name: job4 serial: true plan: - get: github-pre-release trigger: true passed: + - job1 + - job2 - job3 - task: update-release-body config: platform: linux image_resource: ```