Semver mode can wrongfully report tag `latest` as the newest version in race scenario

[odormond]

We recently encountered an issue where the version of a resource type used by concourse was not the most recent one despite the fact that check correctly reported the most recent version available.

Here is the definition of the resource:

resource_types:
  - name: cogito
    type: registry-image
    check_every: 1h
    source:
      repository: pix4d/cogito

As you can see, source.tag is not set and we expect semver versioning of the resource type.

Looking at the resource_config_versions table in the DB, here is what I saw:

 check_order |                                                version                                                 
-------------+--------------------------------------------------------------------------------------------------------
          13 | {"tag": "latest", "digest": "sha256:9ef19fc7b58192be3de6dd9c400f72d9f6bc4d8f2bab5ff642366c25812789f8"}
          12 | {"tag": "0.6.2", "digest": "sha256:e59670b6b6bf4b7e6dd2c92d7af771719cd63b898329569a6d0a93119dd45543"}
          10 | {"tag": "0.6.1", "digest": "sha256:9ef19fc7b58192be3de6dd9c400f72d9f6bc4d8f2bab5ff642366c25812789f8"}
           8 | {"tag": "0.6.0", "digest": "sha256:cef4adfe17e9ee5f7c16a4094e435d88857f663f080507dcb6b61e7390b52852"}
...

As you can see:

• tag latest has the highest check_order and so is considered the most recent version
• the digest of tag latest is the same as the one of tag 0.6.1
• tag 0.6.2 exists and is indeed more recent then 0.6.1

Forcing a check prints out 0.6.2 as expected.

After looking at the code, the problem can be explained as a race condition between tagging the repository images and checking them. Here is the scenario:

• Start with a docker repository containing a single image with digest hash1 and tags latest and 0.6.1.
• Configure a resource type for it and run the check
  1. all the tags are retrieved from the docker registry
  2. tag latest is processed and variable latestTag set and its digest stored in tagDigests["latest"]
  3. then tag 0.6.1 is processed and digestVersions["hash1"] is set to "0.6.1"
  4. the digestVersions map is turned into the list tagVersions which is then sorted
  5. the check response is built from tagVersions
  6. as latestTag is set, its digest (hash1) is looked up in digestVersions and found and so existsAsSemver is true and so response is left unchanged
  7. the response, which is [{"tag": "0.6.1", "digest": "hash1"}] is returned
• Push a new docker image to the registry with tag 0.6.2 only. Now it contains two images with hashes hash1 and hash2. hash2 has tag 0.6.2 and hash1 has the tags 0.6.1 and latest.
• Run the check
  1. all the tags are retrieved from the docker registry
  2. tags is shuffled to put 0.6.1 first because from is set to {"tag": "0.6.1", "digest": "hash1"} by concourse following the previous check
  3. tag 0.6.1 is processed, digestVersions["hash1"] is set to "0.6.1" and cursorVer is set to "0.6.1"
  4. tag latest is processed and variable latestTag set and its digest stored in tagDigests["latest"]
  5. tag 0.6.2 is processed and digestVersions["hash2"] is set to "0.6.2" because it is greater than or equal to cursorVer
  6. the digestVersions map is turned into the list tagVersions which is then sorted
  7. the check response is built from tagVersions
  8. as latestTag is set, its digest (hash1) is looked up in digestVersions and found and so existsAsSemver is true and so response is left unchanged
  9. the response, which is [{"tag": "0.6.1", "digest": "hash1"}, {"tag": "0.6.2", "digest": "hash2"}] is returned
• So far, so good. Now, run the check again
  1. all the tags are retrieved from the docker registry
  2. tags is shuffled to put 0.6.2 first because from is set to {"tag": "0.6.2", "digest": "hash2"} by concourse following the previous check
  3. tag 0.6.2 is processed, digestVersions["hash2"] is set to "0.6.2" and cursorVer is set to "0.6.2"
  4. tag latest is processed and variable latestTag set and its digest stored in tagDigests["latest"]
  5. tag 0.6.1 is processed and ignored because it is less than cursorVer
  6. the digestVersions map is turned into the list tagVersions which is then sorted
  7. the check response is built from tagVersions
  8. as latestTag is set, its digest (hash1) is looked up in digestVersions and not found and so existsAsSemver is false and so response is extended with {"tag": "latest", "digest": "hash1"}
  9. the response, which is now [{"tag": "0.6.2", "digest": "hash2"}, {"tag": "latest", "digest": "hash1"}] is returned and so concourse now believes that {"tag": "latest", "digest": "hash1"} is the most recent version, which is wrong
• Eventually the tag latest is moved to the image with tag 0.6.2 and digest hash2
• The check runs again
  1. all the tags are retrieved from the docker registry
  2. tags is shuffled to put latest first because from is set to {"tag": "latest", "digest": "hash1"} by concourse following the previous check, note that the digest in from is not actually the digest linked to latest in the registry now
  3. tag latest is processed and variable latestTag set and its digest stored in tagDigests["latest"]
  4. tag 0.6.2 is processed, digestVersions["hash2"] is set to "0.6.2"
  5. tag 0.6.1 is processed, digestVersions["hash1"] is set to "0.6.1"
  6. Notice that this time digestVersions has entries for both 0.6.1 and 0.6.2 because cursorVer is never set due to the fact that from != nil && identifier == from.Tag && digest.String() == from.Digest is never true, even when identifier is latest due to the fact that it's digest is hash2 now but from.Digest is hash1
  7. the digestVersions map is turned into the list tagVersions which is then sorted
  8. the check response is built from tagVersions
  9. as latestTag is set, its digest (hash2) is looked up in digestVersions and found and so existsAsSemver is true and so response left unchanged
  10. the response is now [{"tag": "0.6.1", "digest": "hash1"}, {"tag": "0.6.2", "digest": "hash2"}] and so 0.6.2 appears as the most recent version in the check output
  11. Concourse does not update the version history because these two versions already exist in it and so {"tag": "latest", "digest": "hash1"} remains the most recent version

[mymasse]

I'm running into something similar just using a simple resource not a resource-type. I'm not specifying a source.tag in my resource and was surprised when my job ran with tag latest. Since the documentation mentions:

With tag omitted, check will instead detect tags based on semver versions (e.g. 1.2.3) and return them in semver order

Note that this was a brand new pipeline so all the current images tags were all existing.

[marco-m]

@odormond I have the impression that https://github.com/concourse/concourse/pull/8196 could be a manual workaround (until this issue is fixed).

[odormond]

@odormond I have the impression that concourse/concourse#8196 could be a manual workaround (until this issue is fixed).

Yes. That would be better than having to issue sql queries directly in the DB. :laughing: