Add support to provide search capabilities for the stored artifacts

[hectorj2f]

With the increasing proliferation of new artifacts linked to manifests (e.g. sbom, scan reports, attestations, signatures...), there is a need to efficiently index the content of these artifacts to provide certain search capabilities. Obviously due to the heterogeneity of these artifacts, these search capabilities should be specific to each artifact type.

As an example, some of our use stories are:

• Get a list of images affected by a specific CVE .
• Get a list of images affected by a compromised signature.
• Find the latest scan report linked to a specific image digest.
• Get a list of images affected by a specific dependency.
• ...

Likewise, we are not sure that the usage of annotations for each artifact manifest would be enough to satisfy the aforementioned user stories.

[SteveLasker]

This would be a great example to leverage the distribution extension api, as the referres api does. @hectorj2f, would you like to make a PR as a proposal?

Some other discussions on the topic:

• opencontainers/distribution-spec: Requirements: Search #71
• Common Registry Search and Eventing APIs

[hectorj2f]

@SteveLasker Yes, I'll have a look at the current examples and build something from it.

[michaelb990]

I see this capability as separate from the artifacts and references work. Happy to keep having this discussion here for now if it helps, but the eventual solution would probably be better suited to a separate project or working group focused around a search API that can be added on to a registry as an extension. We're not trying to define everything that can ever happen with artifacts, just the base primitives to allow for storing and simple discovery of artifacts that refer to images.

[SteveLasker]

Yes, to both.

[toddysm]

This ask was brought to my attention in several conversations and there is some interest in moving this forward. @michaelb990 proposal to create a WG sounds reasonable so I would like to bring this up in the next ORAS meeting for discussion. Let's use this issue in the next two weeks to gather interest for WG participation.

[rchincha]

Aye.

As a data point, this is how zot does it ... as a dist-spec extension, https://github.com/project-zot/zot/blob/main/pkg/extensions/_zot.md https://github.com/project-zot/zot/blob/main/pkg/extensions/search/search.md

[michaelb990]

Most of the open items in this repo have moved elsewhere -- I think we should do the same for this issue. The work that this project was created to do was folded into the OCI references working group here and is now part of the spec, pending the next release.

It looks like I don't have access to archive this repo, but I think it's time to do that.

[afflom]

Please sign me up for the working group.

For context, I've been involved with developing the Emporous (formerly UOR Framework) concept for the past 10 months which implements additions to the Image and Distribution Specs. Emporous also imports oras-go.

Prototype References: Emporous Collection Spec (Draft) (Proposed change to Image Spec) - https://github.com/emporous/collection-spec

Emporous' approach:

• Typed attribute field on descriptors (Similar to annotations, but value is json.RawMessage/byte slice)
• Dynamic schema registration in the distribution server (Corresponds to typed attributes)
• Cross namespace linking

cc @jpower432 @sabre1041 @dmesser

[toddysm]

@michaelb990 I will be happy to bring the proposal to archive the artifact repo to the ORAS community in the next meeting and have the community decide at that time. The next meeting is on Jan 31st.

[toddysm]

The proposal that came up from the community meeting is as follows:

• Participants agreed to start the new WG under ORAS to collect and document scenarios
• Once the scenarios are considered complete, the WG participants will present the proposal for WG to OCI community
• Depending on the OCI response participants will make a decision if the specification should be implemented under OCI or will be started in ORAS.

See notes from Jan 31st 2023 in https://hackmd.io/P-O6n222TcSMoJgHmTTduw?both#Meeting-Notes

Please let me know if I misstated something - @jpower432 @sabre1041 @dmesser @afflom @rchincha @SteveLasker @AaronFriel @sabre1041 @FeynmanZhou @sajayantony @yizha1 @Wwwsylvia

I will work with @FeynmanZhou to create a repository for capturing the scenarios.

cc:// @michaelb990

[imjasonh]

• Participants agreed to start the new WG under ORAS to collect and document scenarios
• Once the scenarios are considered complete, the WG participants will present the proposal for WG to OCI community
• Depending on the OCI response participants will make a decision if the specification should be implemented under OCI or will be started in ORAS.

Why not have this discussion in OCI directly, from the beginning? Having a separate external group for requirements-gathering before an OCI WG can be approved and convened (whereby requirements will undoubtedly be gathered again, by a different group of people) just seems like unnecessary bureaucracy.

Or, going entirely the opposite direction: just define an extension without OCI's involvement or approval whatsoever, iterate there until it's perfect, and once it's adopted by clients and registries, propose it to OCI with a groundswell of support from happy users and registry operators.

Trying to do this work under OCI's umbrella while also pre-deciding most things about the API before involving OCI will probably only lead to headaches. (Ask me how I know! 🙃)

[imjasonh]

Also: how does this update square with the @michaelb990 's proposal in https://github.com/oras-project/artifacts-spec/pull/119 ?

[michaelb990]

I've updated #119 to reflect the views of the community from the meeting notes. I think it should be ready to be merged once folks have a chance to take a look.

[toddysm]

I totally agree with you @imjasonh and both paths were discussed in the meeting. Though, the decision above was made collectively by the people who are interested to see such capability implemented in the registries.

I will let everyone else who was in the meeting chime in.

My personal position is that I would like to see this work done in OCI from the beginning. Though, my reluctance comes from the current state and discussions of OCI 1.1 (I believe this is what you meant😀). I am also not opposed to the extension approach. Though, I think that it fractures registry implementations and if it is brought to the OCI later on, we will still need to go through the same discussions and lengthy process. Once again, this is my personal position.

To be clear, the collective decision is to document the scenarios only and not do any work on specifying APIs or anything else.

I am not sure I follow how this issue is related to #119 except that #119 is the precedent for such work🙃

[toddysm]

Sorry, I forgot that this issue is in the repo that @michaelb990 wants to archive with #119. I see the connection now. My bad.

[sabre1041]

@imjasonh I can provide some thoughts surrounding opinion on why this effort was intended to start within this project prior to bringing up to OCI directly. First and foremost, there is no intent to dissuade participation in this topic and as evident in the discussions at the last community meeting, there is certainly interest in investigating and forging a path towards finding an approach to solving this challenge.

Given the current state of the OCI 1.1 release (still in progress) and continued outstanding items that need to be addressed before release, it would be improper to distract with an entirely new feature knowing that it would not be part of the 1.1 specification.

Therefore, to allow 1.1 to continue to progress and allow those who are able to begin efforts on this initiative, initial discusses and approaches could be forged to provide the initial framework and potential approaches and once 1.1 is out the door and there is an initial set of assets related to this effort, it would be brought to the larger OCI audience for which continued discussions to occur.

Anyone who wishes to participate in these initial discussions are certainly more than welcome!

[scottrigby]

Hey folks, just noting here that discussions are going on about bootstrapping a working group. Will follow up in https://github.com/oras-project/community/issues/45.

[toddysm]

I believe we can close this one to avoid randomization. The most recent discussions are in https://github.com/oras-project/community/issues/45 If there are no objections, I will ask the maintainers to close this in the net ORAS meeting.