search

pivotal-cf / pivnet-resource

Concourse Resource to interact with the Tanzu Network API V2 interface.
Apache License 2.0
29 stars 37 forks source link

Newly created resources can't discover a product's previous versions #78

Closed jpluscplusm closed 6 years ago

jpluscplusm commented 6 years ago

Once a new concourse pipeline has been created and the initial, latest, version of a pivnet product/resource discovered, I can't find a way to make the resource discover an older version of the product.

I don't believe this is the same problem described in issue #69. There, the problem appears to be the rerelease of an existing product, i.e. 1.2.3#TS1 can't be used after 1.2.3#TS2 is discovered.

However, here, I'm specifically looking for a way to tell a pipeline to discover the MySQL tile's 1.10.2#2017-08-22T00:11:47.847Z version when 1.10.3#2017-09-14T17:59:51.442Z is the latest release, and the pipeline is newly minted, only ever having seen 1.10.3#

Using fly/concourse v3.3.3, these are invocations of fly check-resource, using a product version which I know exists, that I've found don't work:

$ fly -t ci cr -r pivnet/mysql --from='product_version:1.10.2#2017-08-22T00:11:47.847Z'
$ fly -t ci cr -r pivnet/mysql --from='product_version:1.10.2#2017-08-22T00:11:47.847Z'
$ fly -t ci cr -r pivnet/mysql --from='product_version:1.10.2\#2017-08-22T00:11:47.847Z'
$ fly -t ci cr -r pivnet/mysql --from='product_version:1.10.2\#2017-08-22T00\:11\:47.847Z'
$ fly -t ci cr -r pivnet/mysql --from='product_version:1.10.2\#2017-08-22T00\:11\:47\.847Z'
$ fly -t ci cr -r pivnet/mysql --from='product_version:1\.10\.2\#2017-08-22T00\:11\:47\.847Z'
$ fly -t ci cr -r pivnet/mysql --from='product_version:"1.10.2#2017-08-22T00:11:47.847Z"'
$ fly -t ci cr -r pivnet/mysql --from='product_version:"1.10.2\#2017-08-22T00:11:47.847Z"'
$ fly -t ci cr -r pivnet/mysql --from='product_version:"1.10.2\\#2017-08-22T00:11:47.847Z"'
$ fly -t ci cr -r pivnet/mysql --from='product_version:"1.10.2\#2017-08-22T00\:11\:47\.847Z"'
$ fly -t ci cr -r pivnet/mysql --from='product_version:"1.10.2"'
$ fly -t ci cr -r pivnet/mysql --from='product_version:"1.10.2*"'
$ fly -t ci cr -r pivnet/mysql --from='product_version:"1.10.2.*"'
$ fly -t ci cr -r pivnet/mysql --from='product_version:1.10.2.*'
$ fly -t ci cr -r pivnet/mysql --from='product_version:1.10.2*'
$ fly -t ci cr -r pivnet/mysql --from='product_version:1.10.2'

After these, the page https://ci.example.com/teams/main/pipelines/pivnet/resources/mysql shows only one version: the 1.10.3 version which was discovered the moment I unpaused the pipeline after creation.

The check-resource operation against the same pipeline works fine with a git resource. Older versions are discovered and are consumable. The pivnet resource seems to do something different and non-obvious.

For clarity, I understand the product_version param of the resource's source configuration can be used to constrain the versions discovered. I could specify product_version: 1.10.2. However, I don't want to restrict future versions consumed by this pipeline to just that patch version of the resource - I need 1.10.2 and later versions to be discovered. I'm doing version pinning in jobs' get steps, FWIW.

Any pointers to what I'm doing wrong, or if this just isn't possible and/or is a bug?

cf-gitbot commented 6 years ago

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/151172347

The labels on this github issue will be updated when the story is started.

rkoster commented 6 years ago

Run in the same issue, got it working with:

VERSION="$(curl -s https://network.pivotal.io/api/v2/products/buildpacks/releases/356 | jq -r '"\(.version)#\(.software_files_updated_at)"')"
fly -t bucc check-resource -r custom-java-buildpack/buildpack-3x --from="product_version:${VERSION}"
rkoster commented 6 years ago

In this case VERSION is: Java Buildpack 3.0#2016-12-16T16:25:27.896Z

rkoster commented 6 years ago

figured it out by reading tests: https://github.com/pivotal-cf/pivnet-resource/blob/master/acceptance/check_test.go#L17

jchesterpivotal commented 6 years ago

Hi @jpluscplusm -- we actually got an independent report from another person via another channel. I (shamefully) suspected everyone and everything but pivnet-resource, but after some further tinkering we are now of the view that it's a pivnet-resource bug.

Our hunch is that this function is misbehaving, but we'll want to write test cases and whatnot. We've opened a bug on our Tracker, we'll keep you posted.

jpluscplusm commented 6 years ago

Thanks, @jchesterpivotal. I'm no longer working with a team using this resource, so can't validate any fixes or answer any questions on usage - but I do think it's genuinely a bug and not user error ;-)

theozaurus commented 6 years ago

I am also experiencing this issue. Here's the steps I've used recently to replicate it:

When setting the pipeline:

resource_types: 
- name: pivnet 
type: docker-image 
source: 
repository: {{pivnet_resource_with_proxy_image_repository}} 
username: {{docker_registry_username}} 
password: {{docker_registry_password}} 
ca_certs: 
- domain: {{docker_registry_ip}} 
cert: {{trusted_certificates}}

resources: 
- name: stemcells-3263-pivnet 
type: pivnet 
source: 
api_token: {{pivnet_api_token}} 
product_slug: stemcells 
product_version: 3263\.*

jobs: 
- name: stub 
plan: 
- get: stemcells-3263-pivnet

Using concourse: 3.5.0, and the pivnet-resource: 0.31.3

The stemcells-3263-pivnet resource shows only the latest version as expected:

If we check pivnet for what versions are available we get:

$ pivnet releases -p stemcells | grep 3263 
| 5881 | 3263.28 | Bump Ubuntu stemcells for | 2017-07-01T01:27:02.040Z | 
| 5659 | 3263.27 | Bump Ubuntu stemcells | 2017-06-01T01:49:47.324Z | 
| 5543 | 3263.26 | Periodic Ubuntu stemcells | 2017-06-01T01:50:28.059Z | 
| 5203 | 3263.24 | Bump Ubuntu stemcells for | 2017-04-26T00:41:49.534Z | 
| 4898 | 3263.23 | Bump Ubuntu stemcells for | 2017-04-06T02:14:51.665Z | 
| 4806 | 3263.22 | Bump Ubuntu stemcells for | 2017-03-31T02:04:28.972Z | 
| 4583 | 3263.21 | Bumps Ubuntu stemcells for | 2017-03-13T21:25:59.518Z | 
| 4419 | 3263.20 | Bump Ubuntu stemcells for | 2017-03-13T21:27:12.259Z | 
| 3844 | 3263.17 | Bump Ubuntu stemcells for | 2017-03-13T21:28:18.636Z | 
| 3641 | 3263.15 | Bump Ubuntu stemcells to pin | 2017-01-05T23:57:13.856Z | 
| 3443 | 3263.14 | Bump Ubuntu stemcells to | 2016-12-16T16:25:27.797Z | 
| 3409 | 3263.13 | Bump Ubuntu stemcells for | 2016-12-16T16:25:28.399Z | 
| 3243 | 3263.12 | Bump Ubuntu stemcells for | 2016-12-16T16:25:29.109Z | 
| 2673 | 3263.8 | Bump Ubuntu stemcells for | 2016-12-16T16:25:30.148Z | 
| 2509 | 3263.7 | Bump Ubuntu stemcell for | 2016-12-16T16:25:31.231Z | 
| 2317 | 3263.3 | Bump Ubuntu stemcell for | 2016-12-16T16:25:33.108Z | 
| 2288 | 3263.2 | Bump Ubuntu stemcell for | 2016-12-16T16:25:35.367Z |

If I check resource using from with the version 3263.26 like so:

fly -t cci-vsphere check-resource --resource experiment/stemcells-3263-pivnet --from product_version:3263.26

I expect to see the versions:

However I only see the version:

If I check resource using from with the version 3263.26 like so:

fly -t cci-vsphere check-resource --resource experiment/stemcells-3263-pivnet --from product_version:3263.26#2017-06-01T01:50:28.059Z

I expect to see the versions:

However I only see the version:

If I check resource using from with the version 3263.3#2016-12-16T16:25:33.108Z like so:

fly -t cci-vsphere check-resource --resource experiment/stemcells-3263-pivnet --from product_version:3263.3#2016-12-16T16:25:33.108Z

I expect to see the versions:

However I see:

If I check resource using from with the version 3263.2#2016-12-16T16:25:35.367Z like so:

fly -t cci-vsphere check-resource --resource experiment/stemcells-3263-pivnet --from product_version:3263.2#2016-12-16T16:25:35.367Z

I expect to see the versions:

However I see:

In summary:

Given available resources:

| 5881 | 3263.28 | Bump Ubuntu stemcells for | 2017-07-01T01:27:02.040Z | | 5659 | 3263.27 | Bump Ubuntu stemcells | 2017-06-01T01:49:47.324Z | | 5543 | 3263.26 | Periodic Ubuntu stemcells | 2017-06-01T01:50:28.059Z | | 5203 | 3263.24 | Bump Ubuntu stemcells for | 2017-04-26T00:41:49.534Z | | 4898 | 3263.23 | Bump Ubuntu stemcells for | 2017-04-06T02:14:51.665Z | | 4806 | 3263.22 | Bump Ubuntu stemcells for | 2017-03-31T02:04:28.972Z | | 4583 | 3263.21 | Bumps Ubuntu stemcells for | 2017-03-13T21:25:59.518Z | | 4419 | 3263.20 | Bump Ubuntu stemcells for | 2017-03-13T21:27:12.259Z | | 3844 | 3263.17 | Bump Ubuntu stemcells for | 2017-03-13T21:28:18.636Z | | 3641 | 3263.15 | Bump Ubuntu stemcells to pin | 2017-01-05T23:57:13.856Z | | 3443 | 3263.14 | Bump Ubuntu stemcells to | 2016-12-16T16:25:27.797Z | | 3409 | 3263.13 | Bump Ubuntu stemcells for | 2016-12-16T16:25:28.399Z | | 3243 | 3263.12 | Bump Ubuntu stemcells for | 2016-12-16T16:25:29.109Z | | 2673 | 3263.8 | Bump Ubuntu stemcells for | 2016-12-16T16:25:30.148Z | | 2509 | 3263.7 | Bump Ubuntu stemcell for | 2016-12-16T16:25:31.231Z | | 2317 | 3263.3 | Bump Ubuntu stemcell for | 2016-12-16T16:25:33.108Z | | 2288 | 3263.2 | Bump Ubuntu stemcell for | 2016-12-16T16:25:35.367Z |

fly -t cci-vsphere check-resource --resource experiment/stemcells-3263-pivnet --from product_version:3263.26#2017-06-01T01:50:28.059Z

Fails to find anything.

fly -t cci-vsphere check-resource --resource experiment/stemcells-3263-pivnet --from product_version:3263.3#2016-12-16T16:25:33.108Z

Finds everything newer than 3263.3, but not 3263.3.

fly -t cci-vsphere check-resource --resource experiment/stemcells-3263-pivnet --from product_version:3263.2#2016-12-16T16:25:35.367Z

Finds everything newer than 3263.2, but not 3263.2.

It would be great if it worked like other resources where something like:

fly -t cci-vsphere check-resource --resource experiment/stemcells-3263-pivnet --from product_version:0.0

Would return all old resources.

jchesterpivotal commented 6 years ago

Hi @jpluscplusm @theozaurus @rkoster, thanks for your patience. We believe we've identified the problem.

Pivnet-resource determines new versions in versions.Since. This code in itself is OK. The problem is that it assumes Pivnet's API responds with ordered results. For some products this is true, for each of your reports, it's not true.

For example, ERT is almost completely ordered:

# Elastic Runtime
$ http https://pivnet-acceptance.cfapps.io/api/v2/products/elastic-runtime/releases Authorization:"Token [redacted]" | jq '.releases | .[] | .version'
"1.12.7"
"1.12.6"
"1.12.5"
"1.12.4"
"1.12.3"
"1.12.2"
"1.12.1"
"1.12.0"
[snip]

Whereas MySQL, Buildpacks and Stemcells are not returned in order, with varying degrees of disorder:

# MySQL
$ http https://pivnet-acceptance.cfapps.io/api/v2/products/p-mysql/releases Authorization:"Token [redacted]" | jq '.releases | .[] | .version'
"1.10.6"
"1.9.13"
"1.10.5"
"1.10.4"
"1.9.12"
"1.9.11"
"1.9.10"
"1.8.11"

# Stemcells
$ http https://pivnet-acceptance.cfapps.io/api/v2/products/stemcells/releases Authorization:"Token [redacted]" | jq '.releases | .[] | .version'
"3445.16"
"3421.31"
"3363.41"
"3312.45"
"3363.37"
"3312.38"
"3421.26"
"3445.11"
"3421.24"
"3363.35"
"3445.7"
"3421.20"
"3312.34"
"3363.31"
[snip]

Our current thinking is to sort responses on the server. Right now it is possible use sort_by: semver to perform a version sort on pivnet-resource itself, but this will not work for all cases. Many products have non-semver versions in their history, which I believe is why the sort_by: semver is optional.

This approach means we do not need to update pivnet-resource itself, so all users immediately see the repair. The tradeoff is additional load on our database (though presumably not much, these are relatively small sets).

I'll loop back once we've made the change.

jchesterpivotal commented 6 years ago

Hi all;

False alarm on my part. The story is more complex than we thought on Tuesday.

Ordering

We think these cases (Stemcells, MySQL, Buildpacks) are fixed by using sort_by: semver, assuming correct version strings are provided -- Concourse does not appear to have a mechanism for a resource to signal "not found".

My comment above is correct in saying that pivnet-resource relies on the order coming back from the server. But it's wrong in saying thtat the order is undetermined. In fact, Pivnet assigns a sort_order value to every release created. This is intended to allow Product Admins to reorder releases. But if they choose not to manually adjust the ordering, the ordering is based purely on the order in which releases were created.

If you look at the example we gave of Stemcells above:

$ http https://pivnet-acceptance.cfapps.io/api/v2/products/stemcells/releases Authorization:"Token [redacted]" | jq '.releases | .[] | .version'
"3445.16"
"3421.31"
"3363.41"
"3312.45"
"3363.37"
"3312.38"
"3421.26"
"3445.11"
"3421.24"
"3363.35"
"3445.7"
"3421.20"
"3312.34"
"3363.31"
[snip]

You will discover that while the versions are not sequential, they are ordered for each major. This comes about because the BOSH team makes multiple simultaneous releases and the sort_order value is autoassigned.

If I rearrange the version output manually:

"3445.16"
              "3421.31"
                            "3363.41"
                                          "3312.45"
                            "3363.37"
                                          "3312.38"
              "3421.26"
"3445.11"
              "3421.24"
                            "3363.35"
"3445.7"
              "3421.20"
                                          "3312.34"
                            "3363.31"

Now you can see that each major version, read vertically, is ordered.

If you apply the sort_by: semver configuration, pivnet-resource will locally sort by version, causing majors to become grouped. When that's done, we observe that for each of Stemcells, MySQL and Buildpacks, using the sort fixes the observed problem.

Off-by-one

It does not fix the off-by-one problem. Pivnet-resource is incorrectly "swallowing" the version provided. So given 1.1, 1.2 and 1.3, a request to check-resource --from product_version:1.1#sometimestamp would currently only give 1.2 and 1.3.

This leaves us at variance with the Concourse documentation for resource implementers, which says that check scripts:

must print the array of new versions, in chronological order, to stdout, including the requested version if it's still valid ... If the given version is already the latest, an array with that version as the sole entry should be listed.

Our code was instead discarding the requested / given version in its response to Concourse, causing the "off-by-one" behaviour observed by @theozaurus.

We've repaired pivnet-resource to correctly include the requested version, to prevent the off-by-one error. It should be working as of v0.31.5.

Documentation

We've also updated our README to add a "Common Gotchas" section, to describe this problem better for future users.

jpluscplusm commented 6 years ago

@jchesterpivotal Great detective work :-)

Reading through https://github.com/pivotal-cf/pivnet-resource#some-common-gotchas, it's not immediately obvious to me how to fix the situation that's a combination of:

Am I missing something obvious, like "sort_by: semver drills into the version value until it finds something semver-like"?

[ As mentioned above, I'm not currently using the pivnet-resource so don't have any skin in the game here! ]

theozaurus commented 6 years ago

Thanks @jchesterpivotal , thanks for sorting the off be one issue. In terms of the semver ordering side of things here's what I've discovered for each product slug:

Semver

4 dot

Mainly semver

v in front

Only end of availability

End of availability and Semver

All over the place

The bulk of them are semver, some are 4 dots. These seem easy to handle.

The mainly semver ones have issues like release candidates or betas not being in the form of 1.8.0-rc.2 or 1.0.0-beta, or having strings to try and identify some variant. The release candidate ones could easily be changed to fit a standard form. The other ones look like they need a new home. Some of the mainly semver versions have what looks like descriptions Session State Caching, PWS Entitlement, GemFire for PCF, these need to be assigned versions

There are a few with v in front of the version number, the remedy here would be to remove the v.

Quite a few are using the version field to signal End of Availability. This should be removed from the version field.

The hardest ones are things like buildpacks which groups Ruby buildpacks with Java buildpacks. These seem like different things and should not share the same slug.

I appreciate this has broader ramifications, but with Pivnet not enforcing semver versions causes a lot of downstream complexity and with some changes all resources could fit semver.