[RFC] Define in the spec (or remove from code) that discovery will set version to "latest" if no version is requested

[sgotti]

Actually the spec, in the discovery section, doesn't specify that, if no version is provided, it'll do discovery providing a default version as "latest". But the appc/spec code does this.

This is used for implementing the "latest" pattern. So someone could just define a single template like https://{name}-{version}.{ext} and, if an user just requires "myaci" it will use:

https://myaci-latest.aci

I have some thoughts on pros and cons of this approach:

• It's handy since with an unique template, a server can implement "latest" just providing an url that can be a symlink/redirect to the real latest aci).
• it's different from the noarch approach. Here if someone wants a noarch aci it should provide multiple templates in this order:

"https://{name}-{version}-{os}-{arch}.{ext}"
"https://{name}-{version}.{ext}"

on discovery, not providing os and arch, the first template will fail to render, and the second will render and be used.

Using the noarch approach, the latest pattern can be implemented with another template:

"https://{name}-{version}-{os}-{arch}.{ext}"
"https://{name}-{os}-{arch}.{ext}"

or

"https://{name}-{version}-{os}-{arch}.{ext}"
"https://{name}-latest-{os}-{arch}.{ext}"

So I have two proposal:

• Define in the SPEC the current behavior. No impact on actual server implementations but will keep actual confusion like in coreos/rkt#2305
• Remove from the appc/spec code the defaulting of version to "latest" if not provided. This will impact current implementation since many expect to abtain the "latest" aci with just one template, but it's probably cleaner.

Before opening a PRs on first or second behavior I'd like to ear your thoughts.

[sgotti]

Since I find the second proposal cleaner I went ahead and tried to implement it. This also showed me some additional "problems" so I opened 2 PRs for doing this #580 #581 (and also found other related problems in rkt: coreos/rkt#2311 coreos/rkt#2317).

IMHO this is cleaner than documenting an hidded "latest" behavior and can open the SPEC to some meta discovery implementations that currently will not be possibile BUT can break some implementations that rely on this.

I'd really happy to ear your thoughts on this.

[mpasternacki]

The "latest" part always felt like a provisional solution to a real need. Provisional, because it overloads an actual label as a "tag" used only in discovery URL; the version label in the image manifest will be different than what was used to download. Actual need, because it's obviously useful to download the latest version. I'd even go further and say that it's very limiting to only have one "latest" pointer.

Docker uses word "tag" for what is more like a version control branch: a pointer to a most relevant version for a name at some point in time. A Docker image, like a Git commit, can have multiple tags, so an image can be tagged "latest", "stable", "3.0.1", "3.0.x", "3.x". AppC's labels cannot take multiple values (and if they could, it would complicate publishing an image), so the only applicable version label would be the most precise: "3.0.1". For later builds of same software version, to be able to uniquely identify an image, one would need to either add a revision inside version ("3.0.1-1"), or use new label for that ("version=3.0.1 build=1"). Don't even get me started on images that include more than one piece of software (we could use multiple labels, "foo-version=3.0.1 bar-version=2.3.8", which is precise enough). I like the convention of label set uniquely identifying an image; if I publish a new version, I bump at least one label. Otherwise, trying to figure out whether sha512-cafe or sha512-beef is more up to date is next to impossible.

How to combine the complexity of arbitrary labels with easy discovery and movable "pointers"? Let's assume we want the discovery URL to match actual labels (IMO it should be an error if we discover for "version=latest", and we get an image with "version=3.0.1"). We need to have a way to specify pointers in the discovery; an intermediate "map". This is just a quick sketch of what I think would be useful:

Add a new meta tag for a "tag map". It would contain a URL to a JSON document (and "$URL.sig" would be its GPG signature if public key is provided) with a mapping of "tags" to a set of labels (or tag aliases), something like this:

{
  "latest": "3.x",
  "stable": "3.x",
  "3.x": "3.0.x",
  "3.0.x": "3.0.1",
  "3.0.0": "3.0.0-1",
  "3.0.1": "3.0.1-2",
  "3.0.0-0": { "version": "3.0.0", "build": nil },
  "3.0.0-1": { "version": "3.0.0", "build": "1" },
  "3.0.1-0": { "version": "3.0.1", "build": nil },
  "3.0.1-1": { "version": "3.0.1", "build": "1" },
  "3.0.1-2": { "version": "3.0.1", "build": "2" }
}

Discovery for example.com:foo would use a foo tag from the map if there is a map. example.com:foo,build=2 would override individual labels. Specifying a literal version in a name would need explicit key (example.com,version=3.0.1,build=2). If there is no map provided, example.com:foo would fall back to example.com,version=foo.

[sgotti]

The "latest" part always felt like a provisional solution to a real need. Provisional, because it overloads an actual label as a "tag" used only in discovery URL; the version label in the image manifest will be different than what was used to download. Actual need, because it's obviously useful to download the latest version. I'd even go further and say that it's very limiting to only have one "latest" pointer.

I completely agree with you.

Add a new meta tag for a "tag map". It would contain a URL to a JSON document (and "$URL.sig" would be its GPG signature if public key is provided) with a mapping of "tags" to a set of labels (or tag aliases), something like this:

I really like you proposal. It's fair enough to help ACI servers to dynamically generate this map.

This will also probably fix open issues like: #557 #415 coreos/rkt#2305 and probably many others.

In #580 #581 I tried to implement the "latest" behaviour using only pure meta tags. But this led me to the need to change the actual template matching behaviour making it more strict (see #580).

(IMO it should be an error if we discover for "version=latest", and we get an image with "version=3.0.1").

Agree (and the spec states this). Regarding this I opened coreos/rkt#2311 because actually rkt doesn't check this on fetched images causing discrepancies between remote fetching with discovery and local store.

If your proposal goes ahead, this check will fail since an user can ask for `example.com:foo' and from the map fetch an image with version = '3.0.1'. Since I think that check is required to respect the spec the discovery process should return, in addition to the endpoints, also the "expanded labels" and the check will use them.