search

hexpm / specifications

Specifications for using and implementing Hex protocols
48 stars 26 forks source link

Add registry v2 specification #10

Closed ericmj closed 7 years ago

ericmj commented 8 years ago

/cc @hexpm/contributors @hexpm/rebar3

ericmj commented 8 years ago

Also going to add namespaces as part of this but let's start out with discussion the new format.

ericmj commented 8 years ago

There's also the question on how to handle the signing. All files need to be signed. The old registry format had an optional header and a registry.signed file with the signature. The header is an optimization over the file so we don't have to do an extra HTTP request and we need the file so we can support protocols other than HTTP. There's also the issue of the race condition between updating the registry and the signature objects in the S3 bucket.

My proposal is to say that the first 256 bytes of each file is the signature and the rest of the file is the protocol buffer encoded message.

ericmj commented 8 years ago

The signing is ready for review.

ericmj commented 8 years ago

Added yanking. Unlike the initial ideas for yanking we decided that yanking should not change resolution. The reasoning be that some people do not lock their package and instead use "== 1.2.3" for their requirements. If a yanked package no longer resolves it would break repeatable builds for them.

With this change packages are no longer really "yanked", it's more like adding a notice or a warning to an existing release. With that in mind, can we find a better name than yanking?

ericmj commented 8 years ago

Added namespaces and private packages.

This is now done pending reviews and feedback. Would really like the other client maintainers to review this since they will be needing to implement and support this in their clients eventually.

ericmj commented 8 years ago

Another question to discuss is if the index resources /names and /versions should include public namespaced packages.

ericmj commented 8 years ago

This is ready for merging, but I am waiting until we have complete implementations on https://github.com/hexpm/hex_web and https://github.com/hexpm/hex to see if there's anything that needs tweaking.

We can now generate the new registry format in production on hex.pm. It also seems to be working in the client but I have a few bugs to work out there. Still need to implement package yanking. Namespaces and private packages are going to be marked as experimental in the spec until I have it implemented on hex.pm, that's at least a few months out.

idyll commented 8 years ago

Can I suggest deprecate or retire or renounce instead of yank for this.

josevalim commented 8 years ago

@idyll we had a naming discussion at some point and I believe none of us were happy with yank. FWIW, I like retire. deprecate may have other meanings which wouldn't apply to security cases and renounce may mean we gave up on the package altogether.

ericmj commented 8 years ago

+1 on retire. @idyll would you mind sending a PR updating the terminology?

On Thursday, 1 September 2016, José Valim notifications@github.com wrote:

@idyll https://github.com/idyll we had a naming discussion at some point and I believe none of us were happy with yank. FWIW, I like retire. deprecate may have other meanings which wouldn't apply to security cases and renounce may mean we gave up on the package altogether.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/hexpm/specifications/pull/10#issuecomment-244192016, or mute the thread https://github.com/notifications/unsubscribe-auth/AATV2r52EImFBAwx8TglLe-98-7uSsZNks5qlyy7gaJpZM4JOQJA .

Eric Meadows-Jönsson

idyll commented 8 years ago

Will do. retire works for me.

idyll commented 8 years ago

I've opened a PR for this.

https://github.com/hexpm/specifications/pull/11

jadeallenx commented 8 years ago

I know I'm a Johnny-Come-Lately, but I would really appreciate understanding the rationale why the registry needs a per package/namespace index and two global indices for names and names+versions.

I'm also curious why did you pick protobufs instead of msgpack or bson or plain ol' JSON?

ericmj commented 8 years ago

I would really appreciate understanding the rationale why the registry needs a per package/namespace index and two global indices for names and names+versions.

Not sure what per package/namespace index you are referring to. The names/versions indexes is used for discovery of packages, without it would not be possible mirrors that want to cache the whole repository.

EDIT: Clients should ideally not use the names/versions indexes so it may not be necessary those endpoints for private repositories.

I'm also curious why did you pick protobufs instead of msgpack or bson or plain ol' JSON?

Protobufs is a more efficient in the space it uses.

jadeallenx commented 8 years ago

I'm talking about line 7 in the registry-v2.md document.

ericmj commented 8 years ago

I'm talking about line 7 in the registry-v2.md document.

I'm not sure I follow, that paragraph does not mention namespaces.

Are you asking why we need separate indexes for namespaces? The reason is that namespaces are completely separate from each other, a user that has access to namespace A may not have access to namespace B. We are going to use this to implement private packages on hex.pm. You do not need to use namespaces if you are implementing your own repository.