Version mapping between distribution channels and registry

unional commented 8 years ago

I believe the registry entry (e.g. domready.json) should be tied to the source code (the repo) instead of the distribution channel (npm, bower, jspm, Duo, even github (as distribution channel)).

So here is an attempt to describe how will it look like in that scenario.

For versioning (draft):

{
  "versions": {
    "npm": {
      "0.1.2": "..."
    },
    "bower": {
      "x.y.z": "..."
    }
  }
}

jspm may need to resolve to local or endpoint to get the right source + version. Duplicating it here could be a lot of maintenance work.

blakeembrey commented 8 years ago

This issue definitely needs more context. It seems to be missing any information to know what it's about :stuck_out_tongue:

unional commented 8 years ago

Updated the comment. Is it better now? :smile:

blakeembrey commented 8 years ago

Yes, thanks. I went against this approach because it's tightly coupled with GitHub namespaces and GitHub repo names are generally more fragile than registries. Also, doing this is impossible in a lot of cases - there's a lot of projects on NPM that don't have a registry field.

How would one install using an NPM name to the GitHub name in the registry here?

unional commented 8 years ago

Agree to what you say. There are many projects don't live on github and only available through the distributing channel (npm).

There is no need in back-referencing the source of the code, e.g. github or gitlab or private. When I say it ties to the source code or repo, I mean it ties to them conceptually. i.e. 1-1 map between them.

To the user, they would only know (or care) where they get the package from. So typings i npm:domready@1.2.3 make sense to them.

At such, and for searchability, it is much better to structure the registry as it is right now (/[distribution]/[name].json) compare to what I suggested above.

So to maintain the 1-1 mapping, we could record the distribution channel(s) in the typings.json of the typed-repo. typings publish would update registry/[distribution]/[name].json based on that information.

This means, in typings.json of the typed-repo:

{
  "sources": {
    "npm": "1.2.3", // or semver syntax
    "bower": "1.2.4",
    ...
  }
}

And in the registry, it remains the same.

UPDATE: can add a method typings config version [npm] [semver] which will update the typings.json.

unional commented 8 years ago

The benefit of keeping the version in the typed-repo's typings.json is to help the typed writer reminds himself/herself of which version the typings is written for. Checking the version on registry is hard as the typed writer need to reference back based on the SHA.

Is there a better way to do this?

blakeembrey commented 8 years ago

@unional having multiple fields for versions makes it a little tricky and I don't think anyone has a setup where the versions will be different anyway. I had an issue open to allow the version field to be a semantic version range and use that for warnings.

unional commented 8 years ago

I think it does exists. For example, TypeScript node and NuGet version are different, and modules that removed from specific source (e.g. bower) after a while.

Another example is git. it has different versions available on different platforms.

I'm not sure if all these are related or relevant, so please correct me if I'm wrong. I thought this might be an issue so I bring it up. :smile:

blakeembrey commented 8 years ago

The versions are in sync though, it's not a matter that you have two different distributions of the source code with different versions. E.g. https://www.nuget.org/packages/Microsoft.TypeScript.Compiler/. The only case I can think of is if a platform used a different versioning scheme, but we should specify we only handle semver anyway.

blakeembrey commented 8 years ago

Did we resolve this? I'm going to close unless you think we should re-open :smile:

unional commented 8 years ago

Yeah, this can be closed. Thou still not 100% sure about version matching. e.g.

https://github.com/Microsoft/TypeScript/releases/tag/v1.8.2

Note: This release has the version tag of 1.8.2.0 for npm and Visual Studio 2013 and 1.8.4.0 for NuGet and Visual Studio 2015

May be bring it up in NPM registry discussion?

blakeembrey commented 8 years ago

No, that's the wrong place. Let's always use the NPM version, since it's semver (it's 1.8.2 on NPM, not 1.8.2.0). The versions in NuGet and Visual Studio are not compatible with semver.

unional commented 8 years ago

I agree that the version in registry should be normalized (e.g. using "npm version").

What I try to say is not that. It's about mapping. In other words,

I'm thinking about discovery, not delivery. When user install packages from NuGet (or github using tag, bower, jspm, etc, AND local discoveries for all distribution channels), how to:

map the discovered version (distribution channel dependent) to the normalized "npm version" in registry?
- is the "npm version" readily available (before accessing the registry)? If not, then the mechanism for including such mapping need to be available in the registry.

blakeembrey commented 8 years ago

So, should the registry use compatibility ranges? Or should the compatibility range live in typings.json and be printed during installation?

unional commented 8 years ago

typings.json would probably be the only viable option.

thinking out loud: The mapping between source and typings is many-to-one (many source versions map to the same typings version). API/signature shouldn't be changed in patch version changes anyway.

Version range make sense
Need to wrap my head around to make sure it is working as in this case it is a "reverse range lookup"
- Need validation to make sure the ranges don't overlap
- Pick the first matched range might not a good solution because it is likely not the intent of the typings author
- ^version, >version, >=version, ~version might not work well
- Pick the best match from newer version range to older version range might work?
- It is not easy. e.g. [1...4] vs [2...3], source version is 2.5, which one to pick?

Re-opening the issue. :rose:

unional commented 8 years ago

I think the NPM registry concept might not able to address this.

In this context "Typings" refers to the code, either typings or tsc.

NPM procedure:

User: npm i batch -S // Installed batch@a.b.c
Typings: detects npm:batch@a.b.c is installed
Typings: lookup @typings/batch@p.q.r which have the appropriate peerDependencies that matches a.b.c a. Is there such a call? Do we need to lobby npm to add such functionality? Or we have to iterate each version starting from @latest?

Non NPM procedure (using NuGet as example):

User: nuget i batch // Installed batch@w.x.y.z
Typings: detects nuget:batch@w.x.y.z installed
Typings: lookup @typings/batch@p.q.r which ....nuget:batch@w.x.y.z cannot be placed in peerDependencies, and peerDependencies is reserved for npm version resolution only. a. Have to iterate each version to find p.q.r b. This would work like the NPM procedure above if in step 2 we can identify nuget:batch@w.x.y.z => npm:batch@a.b.c before accessing the registry

For this point, the current registry is a better approach.

blakeembrey commented 8 years ago

On the NPM side, it's easy to get a specific version list - as far as I can tell the only tricky part is (re)implementing search. For example, https://registry.npmjs.org/express.

On non-NPM, it should look the same, except we can optimise the server-side. For example, I can implement an endpoint like /versions/:source/:package/:version/latest. The thing here is, should the endpoint always get the "closest" version, should it work like semver (no major changes) or should the registry keep track of ranges too and do an intersection?

Edit: FWIW, I believe the intersection is the best solution. Does that mean the registry begins to support ranges as keys or do we keep that info only in typings.json? Probably be best to update the registry schema.

unional commented 8 years ago

Also one more point is does nuget:batch maps to npm:batch. There could be a name mismatch (because the name has be taken in one of it). Don't know how to solve this thou. :stuck_out_tongue:

blakeembrey commented 8 years ago

We could add maps to the registry to.

{
  "alias": "npm/batch"
}

Is that what you mean? Like one definition is an alias of the other in terms of the type definition?

unional commented 8 years ago

Yes, but the other way around:

// @typings/{npm-batch,batch}/typings.json
{
  "alias": "nuget/batch"
}

Update: /versions/:distribution-channel/:distribution-channel-package-name/:distribution-channel-version

{
  "alias": ":distribution-channel/:distribution-channel-package-name"
}

blakeembrey commented 8 years ago

Sorry, not follow "on the other way around". I thought it was immaterial? Are you saying the alias is from the same source as the versions?

unional commented 8 years ago

Sorry, not follow "on the other way around". I thought it was immaterial? Are you saying the alias is from the same source as the versions?

We are talking about step 3 in https://github.com/typings/core/issues/5#issuecomment-190937859, right? At that time typings only have npm:batch@a.b.c or nuget:batch@w.x.y.z. So it is a distribution-channel => npm registry lookup.

blakeembrey commented 8 years ago

Let's get to aliases later, it won't work with incompatible version schemes. Also, none of this works without semver either. So let's forget aliasing.

unional commented 8 years ago

Fair enough. And as you pointed out, the NPM route is do-able. :+1:

blakeembrey commented 8 years ago

Yes, both would work the same - just offload certain pieces here or there. The latest deployment to the API demonstrate a more NPM-like approach where every version is being published. I'll investigate using semver ranges in the registry and doing intersections.

typings / core

Version mapping between distribution channels and registry #5