Adding files to an existing release

[ebabani]

Hey,

We are trying to use the pivnet resource for adding product files to our release. Currently if the release we're trying to add files to exists, then the operation fails.

Are there any plans on allowing adding files to an existing release, instead of always having to create a new pivnet release? Or is the idea in this case to upload directly to the s3 bucket?

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this. You can view the current status of your issue at: https://www.pivotaltracker.com/story/show/114984113.

[robdimsdale]

@ebabani this was a recent change, introduced in https://github.com/pivotal-cf-experimental/pivnet-resource/commit/bd211266fe2040c40a2348b0102d00cb9bbd0460, and is by design.

Previous to this change, trying to modify an existing release would actually create a new release. This is undesirable and not obvious behavior. The current behavior is at least safer.

I would very much like to get the pivnet resource to a place where the put is idempotent. There's a few stories that already exist around this:

• pivnet resource out should update release if it already exists
• pivnet resource out should optionally associate pre-existing files to the newly-created resource
• pivnet resource out should update existing files

My main concern is that the version we provide to concourse, the product version, does not represent the entire state of that release of the product. For example, you add files to an existing product or change its description without changing the version. This would cause concourse not to view the version as different, and as a result concourse build plans may not execute as expected (e.g. trigger: true would not actually trigger a build)

Any mechanism to allow for modification of existing product releases should take this into account.

Do you have any thoughts on this subject? Does the above explanation make sense?

[ebabani]

Hey @robdimsdale

Yeah, the above explanation makes sense and I agree that the current behaviour is safer. You can't end up overwriting an already released version accidentally.

You're right about the version number not being enough to determine when a resource has been modified. It would be nice if pivnet provided a timestamp of when a release or its associated files were last modified but doesn't look like anything exists in the api.

However I do like the current implementation of triggering only on version changes, even though technically it's not correct. On our pipeline it lets us do things like update our tile version and create a new tile whenever a new stemcell version is made public. And I think that when a release on pivnet is made public you shouldn't have any more changes in its containing files.

[robdimsdale]

@ebabani I agree that it's a nice workflow for product authors - incrementally adding files/metadata to the product until it's ready, then flipping the switch and making it public. I believe we could make this work now without modifying the version, but I see a couple of problems with consumers of the product when it is in this unreleased state:

• Users in a separate pipeline - there's essentially nothing we can do about this. They will not receive updates.
• Users in the same pipeline - this might work as intended; I'm not sure. It depends on how concourse caching (the so-called baggageclaim) handles this. I'll ask the concourse team.

I think the right way to solve this would be to have last_modified field or something similar on all product releases, and combine that with the product version to indicate that the product has been updated. We can certainly talk to PivNet about adding this. This would be my preferred way to proceed, but it depends on the ability and willingness of the PivNet team to add this. It bypasses the question of when you should consider a product 'final' and refrain from updating it further.

I'll talk to both the concouse and PivNet teams and follow up in this issue.

[robdimsdale]

@ebabani an update from the concourse team: it is not possible to update an existing release without also changing the version. The version must encapsulate the entire state of the product. This is a contract that resources must implement, and will manifest itself in unexpected behavior across the system if it violated. One example currently is caching; due to volumes being cached on a get, and the metadata for the get not changing (the resource name and version are unchanged) the cached volume from before the put will be used - essentially the updates will be invisible to concourse.

This is at least consistent with the experience users would get across pipelines, even if it does prevent us from exploring this option further.

So, I'm going to engage with the PivNet team (/cc @jchesterpivotal @timlombardo) to see if they have a solution to this versioning issue - is there any way to uniquely identify the entire state of the product (including state of associated product files)? The best way to do this would be to have a hash/checksum of the state of the product, or another option could be to add a last_updated field to each product release.

I'll keep this issue open and update with our findings.

Thanks, Rob

[robdimsdale]

@ebabani after talking with the PivNet team, they already provide a (weak) ETag in the HTTP headers section (see https://en.wikipedia.org/wiki/HTTP_ETag) which is sufficient to guarantee uniqueness of the release contents and its files.

So, there's some stories prioritized to switch to this versioning, which will pave the way for the stories I linked above in an earlier comment (essentially unlocking updates via out by returning a new product version to concourse).

• pivnet resource check should return a product version that changes when release contents change
• pivnet resource in should return the product version with the same extra identifying information it is provided with
• pivnet resource out should create release with the extra identifying information that changes on metadata/files changes

I'll close this issue now, but please feel free to open another one if you have more questions or thoughts.

Thanks, Rob

[robdimsdale]

@ebabani with the release of v0.10.0 the version returned to concourse represents the entire state of the release. This paves the way for modifying the release and hence the stories linked to earlier in this conversation to supporting updating existing releases via out have been prioritized in the backlog.