ref-engine-discovery: Add a ref-engine discovery protocol

[wking]

This is mostly from this comment, but pulls in ideas from the rest of the discussion with @xiekeyang and @cyphar there, as well as the inspirational specs mentioned in the README. I haven't spent much time on casEngines in this PR, because I consider that orthogonal to ref-engine discovery. Helpful ref engines may populate casEngines in their Merkle roots without breaking the deeper Merkle tree.

This PR will hopefully make it easier to track my proposal as I continue to pivot in response to review.

[wking]

I've pushed 8c18e71 → 37069dd, spinning the DNS-compatible image name bit out into it's own micro-spec. This information is more generic than my ref-engine discovery protocol (where it had lived before), and is very similar to (and inspired by) @cyphar's distribution-uri ABNF. Differences from his ABNF:

• I've uses host instead of authority before the slash. Where they're needed, I expect users to be providing userinfo and port information via other channels, especially since docker etc. currently use : to delimit tags.
• I've dropped segment-nz because the “default authority” option is not portable. I've mentioned it as a potential extension.
• I've added an optional fragment, which, depending on how folks process the index, may allow you to use the same index URI template for multiple image names (somewhat like Docker's tags).

[wking]

I've pushed 37069dd → 8d7eb73 adding a CAS protocol registry, oci-cas-template-v1, and the ability to optionally set casEngines in the ref-engine discovery response (as initially floated here). With this change, the current ref-engine discovery response is very similar to @cyphar's distribution object with its indexuris and bloburis, with the main differences being:

• casEngines is optional. I expect it to be set most frequently in ref-engine responses, and only occasionally in ref-engine-discovery responses, but you don't have to set it anywhere if you don't want. @cyphar's bloburis is required.
• I don't have @cyphar's discovery object indirection, although you could add something like that to my suite of protocols if you want to support folks with dumb well-known ref-engine discovery servers, dumb oci-index-template-v1 ref-engine servers, and unstable CAS engine providers.

[xiekeyang]

I'm drawing a commit 1 (not finish and not put PR yet) on branch discovery It do similar thing as this PR. But it doesn't matter, we just hope to present a good specification.

I had thought that we should build 2 well known URIs for retrieve ref-engine object, and cas-engine object. See 2.

https://example.com/.well-known/oci-index/reference?...

and

https://example.com/.well-known/oci-index/descriptor?...

Because I think consumers want to them for different purpose.

[wking]

I had thought that we should build 2 well known URIs for retrieve ref-engine object, and cas-engine object.

I've spun this off into this comment.

[wking]

I've pushed 2bb26ec → f815358, changing .well-known/oci/ref-engines to .well-known/oci-ref-engines so we can register the ref-engine discovery spec independently from other OCI .well-known specs. Based on this comment.

[wking]

I've pushed f815358 → 906725d fixing a path-rootless → path typo and adding two examples of serving everything from a single static Nginx server.

[xiekeyang]

@wking I think you can add some explain about goal of ref engines. It is for retrieving image name's and basic information's URI , with mutable content. And clarify that in most cases refEngine has no relationship to casEngine. This language can make readers clearer, IMO.

[wking]

I think you can add some explain about goal of ref engines. It is for retrieving image name's and basic information's URI , with mutable content.

There's something like that in the glossary entry. I looked that over again, but cannot thing of anything more to add. Maybe you can recommend some wording?

And clarify that in most cases refEngine has no relationship to casEngine.

There's something like that in the out-of-scope line (which is now the final line in the Merkle root glossary entry). But while the Merkle root's opinion on casEngines is independent of ref-engine discovery, casEngines itself is no longer independent (since b251fe1 which allows a casEngines entry in the ref-engines object).

This language can make readers clearer, IMO.

Language suggestions welcome :).

[wking]

I've pushed f56204e → 89e3eb5 with some header tweaks, a list of specs and registries in the README, and a Python implementation of everything except the CAS engines.

[xiekeyang]

LGTM :1st_place_medal:

It has reached almost what we discussed before, and can be merged already. In future we can go on improve and extend the capabilities. And one question: why oci-discovery implementation 1 select python but not golang? If anyone like and want to call API, maybe golang is more popular in OCI ecosystem. Of course the implementation 1 is amazing, and we can add golang later if necessary.

[wking]

And one question: why oci-discovery implementation 1 select python but not golang?

I write Python much faster than Go ;). Agreed that a Go implementation would be good too.