Need a long term solution to supporting users on bleeding edge versions of typescript

[ericanderson]

It makes sense to not want to break existing users with a "^" dependency on something like @types/react, but it hinders the very users and organizations that help the TS ecosystem. The introduction of Partial<> and co in particular is a big game changer, and many users would benefit from updates to react and lodash in particular.

The current policy has been stated as waiting 30 days before allowing new features into DefinitelyTyped to avoid breaking people. While I disagree with the policy, I think it we as a community can do better to support those on the bleeding edge.

Considering the situation, I'd like to propose a few options.

Option A, do nothing. There are work arounds if you want to start moving fast, such as hacking paths and typeRoots in your tsconfig. I have documented one such solution here: https://medium.com/@ericlanderson/using-custom-typescript-definitions-with-ts-2-x-3121db84015d#.tvqw3rmth

While this works, its a hassle.

Option B: Someone (maybe me) forks DefinitelyTyped and allows changes for the latest and greatest right away. I could publish this as something like "types-edge" in npm and provide documentation for how to configure your tsconfig to allow for it to work.

While this works, it half forks the community, and creates a burden on myself to keep up to date with DefinitelyTyped. It also hinders discoverability. I think this hurts the ecosystem, but its better than everyone manually forking type definitions.

Option C: DefinitelyTyped and Microsoft support the development of these edge versions on a branch within DefinitelyTyped (lets call it "edge" or even use "master"). Microsoft could then register the org @types-edge and publish the edge branch. The DefinitelyTyped and Microsoft people could then as a group keep merging types-2.0 (which should really be called "stable" in this world) into edge, while allowing the community to grow and move quickly. If desired, there could also be a "nightly" branch and a corresponding @types-nightly (I would have personally liked this as the second Partial<> and keyof got merged, I started using it immediately). This would also mean the types are already being updated and ready to go when the next TS release happens.

There could also be a release of typescript 2.1.1 pretty quickly that adds "./node_modules/@types-edge" to the default list of for the paths and the typeRoots in tsconfig.

I think this is the best short-term solution.

Option D: Add macros to TS. (Yeah this is dirty). But the macros could allow things like #if typescriptVersion >= 2.1.0 and allow additional definitions to be placed into the type files. This would still require the 30 day grace period before it could be introduced (maybe in TS 2.2?) but would future proof this problem and allow just a single npm org for publishing types. Its also not backwards compatible, but it would be the easiest to maintain in DefinitelyTyped.

Option E: Add additional meta-data support to package.json to declare types with a minimum version. This could look something like:

{
  "...": "...",
  "types2": {
    ">=2.1.0": "index-2.1.0.d.ts"
  }
}

Newer versions of typescript can look at this instead of types and choose the highest version that maps.

This would be a PITA to support, IMHO. You could do it with branches in DT, but now we have to support forward merging changes to types-2.0 to types-2.1 and then types-2.2, etc. Or you have to keep the different versions in the same branch and remember to change them all when adding new functionality.

Another way to do this, which I have no idea if it would work given future changes to TS are unknown, is write scripts that convert the latest typings into older versions. For example, pretty much every typing that would want Partial<T> could just be mutated at the time the publisher runs to replace it with T. Pick is probably any in the old world. ReadonlyArray support wouldn't have been broken cause it could just be Array. This would require custom scripts for each new release but it might not be awful.

There are probably other ways to do it, but these are the ones that came to me today. I would really love to hear the opinions of a bunch of people: @andy-ms, @ahejlsberg, @pspeter3 @vsiao @johnnyreilly, @jkillian, and more.

Thanks

[jkillian]

Thanks for writing this up @ericanderson. Over in a different thread, I proposed somewhat of an Option F: simply allowing new TS structures in typings right away. To justify this proposal a little further: the version numbers of @types packages specifically are not semver structured. That is, the version number of the major and minor version of the @types package corresponds to the major and minor version of the actual package and not to typical semver meaning. (The patch number is left for making incremental changes to the typings independent of the actual package.)

This means that when we want to add breaking changes to a @types package, say TS 2.1 features, we have no way of indicating this breaking change. We can't bump the major version because that would correspond to the actual library getting a major version bump. In effect, this means users can't tell when breaking changes will be introduced and thus should depend on a fixed version number of an @types package. (Which I think is a reasonable thing to ask.)

All this put together, DT could start merging and releasing packages with new TS features right away.

Edit: As noted below, utilizing peerDependencies to specify the minimum TS version for an @types package along with the above is a smart idea

---

I'd also support Option C. My worry is that maintaining the two running versions of the types would be a lot of work for the already big job the DT maintainers have. There'd also have to be a fast way to handle the inevitable merge conflicts between the two. Overall though, from a developer standpoint, I'd be happy with an @types-nightly. (As a sidenote, "nightly" seems that it should correspond to the nightly releases of tsc, which opens up a whole new can of worms...)

[masonk]

NPM already has robust versioning support. I would like to see DT utilize that more. The essential issue is that @types packages don't know what version of TypeScript they are for.

If a user on 2.0 were to download your 2.1 update today, their types would break and it wouldn't be obvious what happened. There would still be a version of @types/react which works with their version of TS, but they (1) wouldn't have been told that that is the problem and (2) wouldn't be able to easily locate the NPM version that they do need. If you run npm view @types/react today, you'll see a fairly daunting list. I would not want to hunt for the last version that worked with TS 1.7, to take an example at random.

However, very little would be needed to make this versioning story awesome. First, there must be some way for DT contributers to signal the minimum TS version that their patch works on. I think some metadata file at the top level of the DT package folder which lists a typeScriptMinimumVersion would be sufficient. If you make a change for a newer version of TypeScript, that change must update this metadata file as well. Then, software that builds the npm from the DT project would be updated to look at the tSMV, and put that into the package it's building as a TypeScript peer dependency. Boom, we're now using NPM to do versioning. That is the dream.

Since NPM is doing the versioning, users who subsequently use npm to install a version of an @types package that is dependent on a higher version of TypeScript than is currently installed will be warned by NPM about the exact cause of their issue. They would then have the option to upgrade TypeScript, or to find the version of their @types package which supports their installed version of typescript. This would be a lot easier because as they view out specific versions of their @types package, it just tells them the info they needed to know, via the peer dependency line. A small tool could be made available that does that the hunting for you, since all it would have to do in this situation is read through the NPM archive looking at peer dependencies.

Edit: Additional thought:

This system could also be used to signal what version of React these typings are for. Just have a metadatafile that lists all peer dependencies. I know that right now, the package version apes the host project version that it declares. But there's nothing stopping you from downloading the wrong version, and nothing that tells you that you downloaded the wrong version. NPM peer dependencies do exactly that.

[pspeter3]

I agree that this seems like an issue that should be solved by npm and not @types or DefinitelyTyped. Since the 2.0 types are published, nothing is force upgrading users. Eg, if you have pinned the version of TypeScript, you have hopefully pinned the version of your type definitions.

[ericanderson]

I don't know if I agree with the policy, but I think the argument here against just upgrading the types, even if you pin, is that you shouldn't be prohibited from upgrading to the latest react unless you upgrade typescript.

Now. That said, I don't think upgrading to TS 2.1 from 2.0 is a dangerous upgrade so the point may be moot, but I think that's what the policy is intended to allow.

[masonk]

When, e.g., React 16 drops, I would expect that contributors will be most likely to write new declarations against the oldest available version of TypeScript that has all the features necessary to most accurately declare React 16, and not anything older than that.

If some particular org needs typings for React 16 declared in an older version of TypeScript, I would expect them to fork and do it themselves.

It's true that they have no way of publishing it back through DT right now, but is that such a big deal? Do we need to envision a system that lets DT post packages for an n x m matrix of TypeScript and project versions?

[mkusher]

First, there must be some way for DT contributers to signal the minimum TS version that their patch works on. I think some metadata file at the top level of the DT package folder which lists a typeScriptMinimumVersion would be sufficient.

I think package.json's peerDependencies can do the trick

[ericanderson]

Will peer dependencies break if it says "^2.1.0" but I am using "2.2.0-dev.20161212"

[mkusher]

@ericanderson well, I think version would be ">=2.1.0-dev" or something like this

[ericanderson]

@mkusher ">= 2.1.0-dev" won't work, as it would only include the -dev line. I guess the version for the peerDependency could be "^2.1.0 || ^2.2.0-dev"

[ericanderson]

After thinking about the problem some more this morning I am pretty sure Option C is the best choice for now.

I have a hack week next week, I will pledge my time to the project if there is agreement (and I have access to repos).

I would: [ ] start a branch "types-2.1" on DT so that new types can start getting merged. [ ] update the publisher to be date aware so that on Jan 7, it starts pulling from "types-2.1" instead of "types-2.0". [ ] create a "types-2.2" branch in case anyone is using nightly and wants to use new features. [ ] request that either my employer (Palantir) or Microsoft purchase @types-edge and @types-nightly on npm. [ ] update the publisher to publish to the above said branches to their respective @types organization [ ] create a pull request on TS to have "@types-edge" added to the current stable version [ ] create a pull request on TS to have "@types-nightly" added to the current dev version [ ] write a script that nightly creates PRs to forward port changes from 2.0 to 2.1 and 2.2. [ ] pledge, for at least 1 year, 5 hours a week of my time to help manage this (fix merge conflicts on the forward porting and support in what other ways help the community)

[jkillian]

">= 2.1.0-dev" won't work, as it would only include the -dev line. I guess the version for the peerDependency could be "^2.1.0 || ^2.2.0-dev"

Even this has issues, as it won't pick up 2.3.0-dev or any later dev release. We ran into this exact problem with TSLint actually and never found a good solution - so we just do something like this:

"typescript": ">=2.0.0-dev || >=2.1.0-dev || >=2.2.0-dev || >=2.3.0-dev"

[mkusher]

@JKillian is it because of "dev" postfix? I thought prerelease tags(e.g. alpha, beta) should work in such case...

[jkillian]

@mkusher the specific words in the tag don't matter actually. NPM just treats prerelease versions with different semantics than regular versions and makes it really hard to specify ranges of them

[masonk]

Of all the options you listed, C sounds like the best. But I still think that peerDependencies is the way to go.

">=2.0.0-dev || >=2.1.0-dev || >=2.2.0-dev || >=2.3.0-dev" looks fine to me. It puts the burden of thinking about versions on one guy, the package contributor, instead of every person who downloads @types and has to think about which branch of DT they're downloading from.

[ericanderson]

We could put in the peerDependencies with option C. That would help people after Jan 7 understand why @types/react doesn't work if they are still TS 2.0

[RyanCavanaugh]

Talked for a while with @andy-ms and we think we have a good low-cost (for .d.ts authors / future us) solution.

Key scenarios we're thinking about here:

• "Bleeding-edge" compiler users or definition file authors who want to use bleeding-edge features
• "Normal" users using the most recent official release
• "Legacy" users on a previous version of TS (though at least 2.0)
• "VS" users (including VS Code) who are using Automatic Type Acquisition

Important constraints:

• npm install @types/foo must work for "normal" users
• VS users don't get broken, ever
• Keep the number of merges to an absolute minimum (they are an enormous time sink for us)

Proposed solution is as follows:

• New header field in definition files for compiler version
  • // TypeScript Version: 2.2 or similar (bikeshed away)
  • major.minor versions only
  • This is assumed to be 2.0 if not present
  • A version higher than the last release will be considered equivalent to the current nightly build (e.g. specifying 2.3 today is the same as typescript@next)
• The definition tester will test using the specified compiler version
  • If something fails, the tester will try using the latest compiler and issue a helpful warning
  • A useful error will be issued if a file depends on another file that requires a newer compiler version
• The definition publisher will:
  • Publish a -edge prerelease version if the specified compiler version is above the latest release
  • Publish release tags for all major compiler releases (e.g. @ts2.0, @ts2.1, etc) on versions corresponding to the last version of the file that worked with that version of the compiler, as well as @edge which is the newest file. @latest points to the newest file which is still compatible with the current released compiler
• No additional changes to existing branch structure on DefinitelyTyped

The workflows for users then becomes:

• "Normal" users: npm install @types/foo
• "Bleeding edge" users: npm install @types/foo@edge
• "Legacy" users: npm install @types/foo@ts2.1
• "VS" users: npm install @types/foo@tsX.Y where X.Y is derived from the currently-installed TS SDK

Thoughts?

[mohsen1]

Publish release tags for all major compiler releases (e.g. @ts2.0, @ts2.1, etc) on versions corresponding to the last version of the file that worked with that version of the compiler, as well as @edge which is the newest file. @latest points to the newest file which is still compatible with the current released compiler

There should be a more npmy way of doing this. With this, today if I'm using 2.1 I have to install type dependencies with @ts2.1 suffix all the time and if I miss it I'll see weird bugs. Can we also enforce TypeScript requirements of those packages with peerDependencies?

// cc @othiym23 @iarna

[masonk]

Given those constraints (and they are good constraints), I really like your solution. Since the publisher is going to be going to the trouble of inferring the minimum required compiler version, I hope it will slap that into the peer dependencies field to make these issues more discoverable. Not all normies are going to be on the latest official release, even after a one month grace period, and they might as well get the NPM help when they try to add a new package. And as a bleeding-edger, I wouldn't mind being reminded of these things when I inevitably screw them up, either.

[iarna]

So, the default tag is latest. You can specify a different default tag by passing --tag <tagname> into npm or (more likely) having it in your .npmrc. So to draw on the scenarios that @RyanCavanaugh outlines:

"Normal" users: npm install @types/foo "Bleeding edge" users: tag=edge in .npmrc, then install as usual "Legacy" users: tag=ts2.1 in .npmrc, then install as usual "VS" users: npm install @types/foo@tsX.Y where X.Y is derived from the currently-installed TS SDK

There's one drawback to this currently. The tag config applies to ALL packages that you're installing, including one's that don't have the tags specified. If the specified tag doesn't exist then the semver range is applied against all available versions. This means that if any of them have a version that's higher than latest then it'll be installed, whereas otherwise you'd get whatever was tagged latest.

In practice, I suspect this won't cause any problems.

(Long term, I would consider proposing allowing multiple tags for the --tag option, which would likely make it substantially more useful.)

[jkillian]

@RyanCavanaugh I like your solution. Good call from my perspective to stick with only maintaining one branch of typings, seems like that'll help maintain sanity for all.

@iarna this is slightly off topic, but it might be neat if you were able to apply a tag in a .npmrc file to a specific scope -would definitely help with the above situation anyways

[ericanderson]

This solution feels pretty good to me with a few thoughts/concerns.

I assume you're still defining the normal version as the latest stable that has been out for 30 days?

The legacy situation feels pretty good. It assumes that if you're not willing to upgrade to the latest TS that you probably aren't upgrading things like react either. This may be the only gotcha if this isn't true.

If the 30 day grace period goes away, this next concern does too: with the grace period in place, if I start preparing setState for the next version of TS and a new version of the library comes out, now we have to unroll my next TS changes in order to implement the new library definition at the current version.

If the 30 day grace period goes away, it's reasonable that if I am on the nightlys that I can just deal and manually manage my type definitions.

As much as I love the peerDependencies thing I think it often hurts more than it helps when you're dealing with prerelease versions.

In conclusion, I think this is better than the current situation but I think it still introduces some weird edge cases when TS updates and then a day later a new version of react drops.

[iarna]

@JKillian Yeah, we'll definitely keep this in mind when we're drawing up user stories for it.

[ghost]

Resolved in #229, although we don't have an "edge" tag, just ts2.1.