search

opencontainers / tob

Technical Oversight Board (TOB)
https://groups.google.com/a/opencontainers.org/forum/#!forum/tob
Apache License 2.0
80 stars 50 forks source link

projects: add ORAS proposal #68

Closed jdolitsky closed 4 years ago

jdolitsky commented 4 years ago

Continuation of #66

Vote Total (Closes 2020-06-19) (please make sure you also leave an LGTM comment):

/cc @opencontainers/tob

jdolitsky commented 4 years ago

@cyphar @ArangoGutierrez comments addressed! thanks

jdolitsky commented 4 years ago

@cyphar sorry for delay, comments addressed

vbatts commented 4 years ago

apart from the maintainer balance, LGTM

cyphar commented 4 years ago

@SteveLasker You probably want to squash the PR to a single commit for the proposal.

estesp commented 4 years ago

@SteveLasker can you add the same voting markdown from #67 to the top comment so we can align these 2 proposals with the Friday vote deadline?

Thanks!

SteveLasker commented 4 years ago

Just getting several cleanup items done, but added the voting to follow the dates committed.

SteveLasker commented 4 years ago

@SteveLasker You probably want to squash the PR to a single commit for the proposal.

They're actually 2, one from @jdolitsky the other from me. These can get squashed on merge, no?

jdolitsky commented 4 years ago

I've squashed Steve's commit which fixed DCO check

jdolitsky commented 4 years ago

Update: addressed Sam and Aleksa's comments by updating the introduction and FAQ, also added Vince as initial maintainer.

SteveLasker commented 4 years ago

apart from the maintainer balance, LGTM

@vbatts, looks like the squash removed the [x] vote

estesp commented 4 years ago

Just a quick note that we may have a vote close and timezone issue here as @cyphar has made comments and requested changes, but it is now the day of the vote close and I assume @cyphar doesn't stay up all night in his timezone :) So, I'm not sure how that will be resolved prior to noon US Pacific today. I have replaced @vbatts LGTM vote that was lost during the commit squash/edit; I assume no one else has voted yet?

cyphar commented 4 years ago

I think it may have been a mistake to run both votes in parallel -- not to mention it's pretty clear we didn't really do a proper review of both proposals until this week (in fact, I have some more questions that I'd like answered about ORAS before I vote).

I think it wouldn't be a bad idea to call another vote on ORAS once we've completed a proper review. I also think that we may have also jumped the gun a bit on the umoci proposal too (given the significant rewrites to clarify many questions I've seen in the past week).

SteveLasker commented 4 years ago

I'll propose this: When we discussed voting these projects into OCI, the discussion was about the spirit of the projects being applicable to the framing the TOB made for the types of projects that make sense for OCI, vs other bodies like CNCF. The edits over the last week are all great edits and feedback. Did we expect the projects to be complete before adopting? For myself, I've been really busy this week and didn't expect this much churn to vote the projects on their merits and Josh jumped in quickly, helping as it was his initial PR. I could see two approaches:

  1. Vote the projects based on the framing we made to the TOB, continuing the PRs as any project lives on.
  2. Extend the voting another week to address the feedback.
SteveLasker commented 4 years ago

LGTM Based on the spirit of adoption of the project, with additions & edits as with any project.

estesp commented 4 years ago

I could be swayed either way (delay vote conclusion by one week or try and get people to vote in the next 3+ hours with @SteveLasker's reasonable point about ongoing discussion/PRs to clarify any open questions). I think the issue is at this point I assume the vote may fail given 2 participants will be in the middle of their night at vote close.

I would like everyone on the TOB to feel comfortable that any larger questions are put to rest (with total agreement that there will always be nuance and changes over time that can be handled post-vote), and it feels like trying to get a complete vote will effectively have to be rushed at this point.

It is a bit troublesome that after months of discussion most questions have come up in the 48 hours before the vote, but that's life in open source! :) Agree with @cyphar that given a lot of questions were raised on both proposals running them in parallel may have added to the difficulty there.

Thoughts? Opinions?

SteveLasker commented 4 years ago

With no downside to extending, and in the spirit to make everyone comfortable with their questions, I'd vote to extend the vote a week to make it imminent to cause focus, but time to iterate.

samuelkarp commented 4 years ago

I think extending the vote another week to settle on the questions makes sense.

cyphar commented 4 years ago

:+1: on extending the vote, and I don't think we need to vote to extend the vote (that'd get a bit silly pretty quickly) -- I think @estesp as TOB Chair can just announce that the vote will be extended since he called it in the first place.

SteveLasker commented 4 years ago

I've extended the vote to next Friday. We have a meeting scheduled Monday evening to discuss the various issues on annotations and library/reference implementation. While I scheduled it for ORAS maintainers, anyone is welcome:

estesp commented 4 years ago

Thanks @SteveLasker; I did pass the extension by Amy and Chris A. today to make sure there were no governance issues I was unaware of with a decision to extend. I will send a note to the TOB list just to make sure it is clear.

estesp commented 4 years ago

What are the remaining questions from TOB members? We need to finalize questions today (and a maximum of maybe 24 hrs from now IMHO) so that answers can be provided and we give the ORAS experts at least a day to provide answers.

I haven't seen any substantial feedback or responses since we extended the vote, but we need a conscious acknowledgement from TOB members that their specific questions are answered and they are all comfortable to make a voting decision.

cyphar commented 4 years ago

Sorry for not sending this earlier, but I've been struggling to get ORAS to work so I can properly review it. Namely, unless I'm mistaken it appears ORAS cannot actually pull OCI images at all:

% oras pull docker.io/library/debian:latest -ao oci_store
Downloaded empty artifact
Pulled docker.io/library/debian:latest
Digest: sha256:46d659005ca1151087efa997f1039ae45a7bf7a2cbbe2d17d3dcbda632a3ee9a
% ls oci_store
ls: cannot access 'oci_store': No such file or directory

The digest is correct, but I don't know where the blobs are supposed to have gone (I found that ORAS does support using an OCI image-spec layout as of https://github.com/deislabs/oras/pull/108 -- though that really should be the default IMHO). I imagine this can't be intentional, hence why I've put off replying until I got it working (I assumed that I was doing something wrong). But with the vote deadline being so close now, it's probably better to post my incomplete opinion on this proposal.

Regarding "library" or "reference implementation", this project can only be one of the two. A reference implementation may provide a library interface (as umoci and runc do, to varying degrees) so that's not really the sticking point for something to be a library. Whether something is a library is more about its purpose, size, and other such factors (and I agree with @estesp that opining about that here probably isn't helpful).

However, @SteveLasker has discounted this project being a library (I'm not sure if you saw this @estesp), and instead saying that it is a reference implementation:

The vote here agrees to merits of ORAS being a reference client implementation of the OCI distribution-spec and OCI Artifacts

Which means it's very critical that I ask whether there are specific plans to make oras pull docker.io/some/oci-image into/this/oci-image/layout work out of the box? Given that this is what most people are going to use the distribution-spec for, I think it's fair to say that this should work and the artefacts features (while obviously important) should be something you have to explicitly specify. Namely:

I don't think all of the above needs to be addressed before the project is accepted, but there should be broad agreement that this is what ORAS should aim towards in the future. Without the above points settled, users will have to be told "yeah we have a reference implementation of the spec -- ORAS -- but if you actually want to download OCI images use skopeo" which is a very split-brain message.

Again, sorry for springing this on you so late in the process but I do think this is important to clear up.

SteveLasker commented 4 years ago

Thanks Aleksa, I really appreciate all the detailed feedback. This is the discussion we've been trying to narrow in upon, container runtime tooling, vs. tooling for other artifacts to leverage distribution with Artifacts.

ORAS is NOT intended to be yet another runtime-container tool. It's not focused on the runtime aspects of the image-spec. It's really focused pushing additional content into the distribution-spec, using the manifest format for defining a thing, and the layers that make up that thing. See Artifacts Scope

We can certainly figure out how to fix the default OCI Image pull experience. However, the conversation for vote should be focused on:

Enabling new uses of the distribution spec. Enabling the community to build new things they need to distribution. ORAS enables custom artifact-cli push/pull experiences. We do this by providing a set of libraries that implement all the required functionality to enable customers building OCI Artifacts to:

In addition to reference implementation and library, perhaps we're talking about the tool category.

Let me address the specific topics raised for details:

The digest is correct, but I don't know where the blobs are supposed to have gone

ORAS isn't intended to account for container-runtimes. It's intended to account for file based artifact types, including directories. The recent conversation on annotations will address this. I don't know how runtime images are tar'd up, but if they have a filename or directory, ORAS would support this model. Right now, I'd consider it a non-supported scenario, or a bug we could support if it were felt to be important. Remember, the ORAS binary is a prototyping binary for building your custom artifact-cli push/pull experience.

However, @SteveLasker has discounted this project being a library (I'm not sure if you saw this @estesp), and instead saying that it is a reference implementation:

I'd suggest we need a bit more clarity on what makes a library vs. reference implementation.

Rather than require an artifact author to start with a go project, adding all the libraries, we provide a client reference implementation of the distribution-spec, which supports arbitrary artifact types.

When the artifact author is ready, they can create a go project and include the same libraries to build their artifact-cli.

So, this could be a reference implementation that has the libraries to copy/paste/edit your own. I'm really open to calling it either library, client reference implementation or tool.

...ask whether there are specific plans to make oras pull docker.io/some/oci-image into/this/oci-image/layout work out of the box? Given that this is what most people are going to use the distribution-spec for, I think it's fair to say that this should work and the artefacts features (while obviously important) should be something you have to explicitly specify.

Noted above, we can enable this, if we feel it's important. While most currently use runtime, ORAS and Artifacts are all about expanding what people are doing tomorrow. ORAS was not intended to be a generic container-runtime tool. We have docker and others for this ORAS focuses on leveraging registries for new artifact types.

The whole -a option is a bit concerning... I saw deislabs/oras#130 but that seems to narrowly-scoped -- the issue is more generally that OCI images should be the default (or at least, if there is a default mode it should be OCI images).

ORAS is not intended to default to oci-runtime images. See: [https://github.com/deislabs/oras/issues/115](ORAS mediaTypes should default to unknown #115) It's everything else ORAS enables.

Consider ORAS a prototyping tool. When you create artifact-cli, you would embed the specific manifest.config.mediaType in that cli. The same way docker build sets the manifest.config.mediaType to ...vnd.docker...

Honestly, I'd like to remove the -a option. Once you specify an artifact with it's name, it should pull regardless.

The storage should be an OCI image-spec layout by default, because that is the primary way that OCI images are stored on local machines.

should be broad agreement that this is what ORAS should aim towards in the future. Without the above points settled, users will have to be told "yeah we have a reference implementation of the spec -- ORAS -- but if you actually want to download OCI images use skopeo" which is a very split-brain message.

These points are focusing on the image-runtime being the priority. ORAS is about all the other artifact types.

cyphar commented 4 years ago

@SteveLasker

I think you're conflating the term "container runtime" with quite a few other things, which is understandable given the term's more generic use within the wider industry. However, within OCI a container runtime is a very specific thing -- it is a program or service which implements the OCI Runtime Specification. Images are not part of the container runtime, so when you said:

What ORAS isn't:

  • A container runtime client

I was a little confused, because nobody should expect that to be the case (it's not going to spawn containers based on OCI runtime-spec bundles, so of course it isn't a container runtime). I just want to make sure that we're understanding each others' concerns and using a term like "container runtime" is a bit confusing when it has a specific meaning OCI but isn't being used that way is confusing (at least to me). In particular, this line also seems a bit confusing to me (again, "runtime image" is a term that I think you're using to refer to the OCI image-spec?):

I don't know how runtime images are tar'd up, but if they have a filename or directory, ORAS would support this model.

Unless I'm mistaken, this is already implemented by deislabs/oras#108. If that mode of operation was the default, or was used based on the media-type (if it matched the OCI image-spec ones) then I would be totally okay with this because it would do what IMHO most users would expect. I think you might be talking about the org.opencontainers.image.title annotation here? Each layer is a tar archive of the container's root filesystem, they don't have individual file names -- and the usage of org.opencontainers.image.title is a little questionable but let's not get into that. I most definitely would not expect ORAS to extract the tar archives, but I would expect it to download them into some content-addressible store so that a tool like umoci can operate on them -- which is what deislabs/oras#108 appears to do.

But I guess your point is that ORAS has nothing at all to do with OCI images directly, it just so happens that you can in theory (though apparently not practically -- see above) use it with OCI images. That's fine. But distribution-spec explicitly does have a lot to do with OCI images. Yes, you can store other artefact types (and that's perfectly fine) but the primary usage people have today (and likely will always have) is related to OCI images. I don't think it's unreasonable to ask "why doesn't that work at all with ORAS right now?".

I'm genuinely not trying to be difficult or anything, I just want to make sure we don't end up in a situation where we have a project in the OCI that explicitly wishes to not primarily operate on OCI images (which is fine), but at the same time is defined as a "reference implementation" of a specification which most people wish to use with OCI images. Because in that situation, if a future TOB had to consider adding a new project that does allow you to push/pull OCI images it would almost certainly be rejected because "hey, we already have ORAS".

I'm not even sure if the problem is related to the "bucket" we put ORAS in -- when someone comes to the OCI and says "oh, I want to use this neat OCI thing -- let me go download an image to go run a container" our answer to that is going to be "well, we have ORAS but it doesn't actually do that -- you need to go elsewhere or write your own tool based on ORAS". That doesn't seem like a reasonable path forward for me.

While most currently use runtime, ORAS and Artifacts are all about expanding what people are doing tomorrow. ORAS was not intended to be a generic container-runtime tool. We have docker and others for this ORAS focuses on leveraging registries for new artifact types.

But Docker isn't being proposed for inclusion into the OCI. ORAS is. If we just say that ORAS is a "reference implementation of the artefacts spec" then that's one thing, but having a project that looks and smells like a tool that could be used to download OCI images (but can't) without having a project that can seems like a bit of a bad idea -- we'd be ignoring what the vast majority of users actually want to do.

SteveLasker commented 4 years ago

Lots of good thoughts, that I didn't read all. It's the end of the day, so for now, I'll say:

I'll review tomorrow in detail and respond. I'm not sure how that impacts the voting timeline, but we're having the right conversation to narrow it in.

estesp commented 4 years ago

@cyphar at this hour in my timezone there is way too much text to respond to for completeness, but let me summarize that it seems like there is a huge disconnect when the ORAS proposal goes out of its way to use phrases like "custom content", "artifacts", "other types of content" and consistently never talks about implementing any functionality related to the image-spec in reference to ORAS, yet almost the entire review revolves around (drum-roll please): ORAS doesn't seem to work with the image-spec! Yes, exactly. We already have plenty of tools for that all over the container ecosystem.

ORAS is a library to handle the client-side interactions with any given distribution-spec implementation, and provides a common library of functionality on which people can implement and explore the ideas and media types being hashed out in the OCI artifacts project. It has no default purpose, nor would it make sense for it to handle pulling OCI image-spec bundles as it was never designed or even recommended as a tool for that purpose. It would take about 15 minutes of code to use ORAS to make an image-spec "compliant" client (I've actually done this with ORAS and a custom image handler function which properly walks the image-spec), but given that wasn't the purpose, I'm confused as to why that has to be the "default mode" of ORAS when it wasn't designed to be a container image client.

cyphar commented 4 years ago

@estesp

I really should try to make more comments more brief, sorry about that.

I do now better understand that ORAS explicitly does not wish to be related to OCI images. That's fine, but if we're going to throw around the word "reference implementation" in the context of a distribution-spec client I do want to make sure we agree what that actually means. While most of this text pre-dates the artefacts-spec, the distribution-spec explicitly talks about OCI images and their storage. It seems very odd to me that we would say that "a reference implementation of a distribution-spec client" doesn't actually do anything with OCI images.

All I want to avoid is that we end up in a situation where we:

If we can avoid that (even if it's just saying "we don't discount the addition of other projects that implement OCI image-spec specific download features"), then I'd be fine with this proposal. Honestly I'd even be okay with a tiny oras-image program that is part of the ORAS repo that implements the ~35 lines needed to be able to download an OCI image. Or oras oci-image pull or something.

I'm not sure if I'm missing something, but am I really the only person who finds it strange that you cannot download an OCI image from an OCI distribution-spec repo with a tool that we are discussing adding to the OCI as a reference implementation of the OCI distribution-spec?

It would take about 15 minutes of code to use ORAS to make an image-spec "compliant" client (I've actually done this with ORAS and a custom image handler function which properly walks the image-spec)

This was my impression as well, looking at the examples and parts of the implementation. That's why I'm finding this all a little bit strange, as though I'm missing some greater context.

SteveLasker commented 4 years ago

Thanks @estesp, and @cyphar We can add code that downloads an OCI image (image meant to be run as a container through containerd/docker/moby). It's not out of the scope, just not the primary scope.

However, downloading an oci-image is needed for air-gapped environments where someone wants to download content, doesn't have or want the docker client as it requires a daemon, and needs to send the content through some gateway to the air-gapped environment. We're seeing this in IoT scenarios that implement Purdue networks.

However, oci-image is not the primary scenario and should not be the default for push because:

To summarize:

jdolitsky commented 4 years ago

I feel this has been filibustered :)

In my opinion, the value of ORAS is not related to images or the image-spec, but rather that it can publish other types of content to registries. It plugs into registries powered by Docker distribution, regardless of OCI specs in play.

To that end, I can see CNCF as an alternative home for ORAS, considering it's built on containerd and used for Helm and Open Policy Agent (all CNCF projects). On the other hand, that might also signal that it has no use outside the "cloud-native" realm. In any case, ORAS can change its course to better represent its employer. It's certainly operating in a gray area right now.

vbatts commented 4 years ago

agreed. I tried to follow, but I think once the words began to spill: "when it rains, it pours".

As to CNCF vs OCI, this lends to the funny relationship that exists between these two groups already. I personally feel that oras makes more sense to live near distribution-spec (despite leaning particularly on code from containerd).

As for pulling OCI images, that should be expected behaviour.

jdolitsky commented 4 years ago

Should this be closed? Does opencontainers want oras? Otherwise I offer 3000 dogecoin for it

SteveLasker commented 4 years ago

Otherwise I offer 3000 dogecoin for it lol, We took the feedback to breakup ORAS to be more specific in the contents it consumes. There were a few open issues to cleanup the separation, including:

So, the action item is to move some of the libraries to oci/articacts and streamline the ORAS project. That said, everyone has been pretty busy with Notary v2, Docker TOS and other priorities that we haven't gotten back to this.

My belief is the Notary v2 implementation will require these changes and it will become a forcing function. Whether it means we maintain this as open, close it and re-open, or open a new issue, is a good question.

For the sake of cleanliness, I'd be fine with closing this and creating a new issue when we're ready.

jdolitsky commented 3 years ago

Hi all - I've added to this week's call agenda the possibility of continuing with this effort. Perhaps we can discuss what requirements people would require to see in oras to make it a suitable opencontainers project? cc @cyphar @samuelkarp

SteveLasker commented 3 years ago

We had some action items from the last discussion:

It would be good to capture the specific steps to come to a consensus on what's required as I believe we agreed it fits the charter of OCI, but needed some cleanup.