Improve Declaration File Acquisition

[DanielRosenwasser]

In TypeScript 2.0, we want to make acquiring declaration files easier. We'd like to consolidate .d.ts management with general dependency management through npm.

Background

TypeScript uses declaration files (files ending in .d.ts, also called "definition" files) to describe the shape and functionality of other libraries. Usually, new declaration files are submitted to DefinitelyTyped, a community-driven repository of declaration files. Unfortunately, there are issues with the way a definition file gets from there to your project today.

Once a declaration file is on DefinitelyTyped, you can get it through tools like Typings or tsd - package managers for declaration files. While very useful, these were extra tools to learn, and this added friction for new users. There were also some technical issues, such as lack of versioning and conflicting declarations, that existing tools couldn't manage.

The new world

So let's jump to the new world to contrast against the current one. Grabbing a declaration file won't require using tsd or Typings at all. It's just an npm command away:

npm install --save @types/lodash

That command does two things:

1. Grabs the declaration files for lodash and saves it to a directory named @types/lodash in our package's node_modules.
2. Saves that as a dependency in our package.json.

@types/lodash is nothing more than a scoped package. We have taken the time to import files from DefinitelyTyped into their own scoped package. But why does this matter, and why is it any different from how things worked before?

Well in TypeScript 2.0, this @types folder is going to be significant. Usually when we try to try to import "lodash", we look in ./node_modules/lodash, ../node_modules/lodash, ../../node_modules/lodash, etc. for type declarations. Instead, each time we peek into a node_modules folder as we climb up, we'll look for @types/lodash first, and then for lodash.

That might sound surprising, but the logic is that if the lodash package itself had type declarations, you wouldn't have installed @types/lodash.

Global declaration files

Some declaration files only affect the globals, and don't use modules (e.g. Jasmine). In most cases, TypeScript will automatically pick them up from the typeRoots option. By default, typeRoots will just be node_modules/@types/ and node_modules/, relative to a tsconfig.json or for loose files, the files themselves.

If for some reason, you need to include Jasmine but it's not present in your typeRoots, then you can use the types compiler option to trigger the same sort of resolution that happens for modules, where TypeScript will look in ./node_modules/@types/, ./node_modules, and keep climbing up.

The types option could be used something like this.

{
    "compilerOptions": {
        "types": ["jasmine"]
    }
}

Going forward

Library authors will be getting attention in 2.0 as well. We're also introducing new features that simplify the way .d.ts files can be authored for libraries that are both modules and globally defined (also called universal modules). Apart from that, everything stays the same - for instance, new declarations and fixes should still go to DefinitelyTyped. In the future, we'll go into this in more detail and have some prescriptive documentation.

Acknowledgements

We also owe a great thanks to those who dedicated their own time into efforts like Typings and tsd. These tools helped bring TypeScript where it is today and ultimately helped guide us on the direction for this effort. Specifically, Blake Embrey, the maintainer and creator of Typings, has worked closely with us during this entire process and given us extremely valuable feedback. In addition to this, Diullei Gomes and Bart van der Schoor, maintainers of tsd, have helped lay the foundation from the beginning.

Finally, the entire DefinitelyTyped community - a group that has grown to over 2000 contributors at this point - have demonstrated, and continue to show, the kind of enthusiasm and support outside of the core team. We're grateful for all the great work that's been done here.

[felixfbecker]

typings allows us to version declarations independent from packages. So we can have typings for version 3.2 and 4.6 of the same module. If there was a bug in just the typings, we can update just the typings by adding a timestamp, and they still reference the same package version.

How will that work with npm? You would have to use the package.json version as the package version of the module, so how will you fix a bug in the typings that still targets the same version of the module? We have to release it as a new version, and then the versions would be out of sync with the module version and we have no way anymore to reference the actual module version, because a declaration update can be a breaking change. Furthermore, if you maintain a 3.x and a 4.x branch, how would you introduce a breaking change to the declarations in the 3.x branch?

Typings solves this much better. Either more frameworks start bundling their typings with their modules or we need a seperate typings manager, that allows declarations to evolve independent of the module.

[RyanCavanaugh]

NPM has version numbers too. We intend to use the X.Y.z scheme where X.Y is the version of the targeted package and z is the version of the typings file. We'll also be adding support in DefinitelyTyped (and other type definition sources) to have multiple types checked in, so you could have e.g. /jquery/1.x/ and /jquery/2.x folders so they can be maintained separately if that's desired.

I should add that if you want to continue using typings because of some complex versioning scenarios you're anticipating, nothing in these changes will prevent you from doing so.

[myitcv]

How are type definitions for different versions of TypeScript handled?

Related discussion: https://github.com/typings/typings/issues/49

[RyanCavanaugh]

We'll have tags for the latest version of a file compatible with each released version of the language starting at 2.0. So you could npm install @types/foo@ts2.1 or whatever if you need the 2.1-compatible file.

[myitcv]

@RyanCavanaugh thanks

[anaisbetts]

This is a super rad idea, but how will people upload things to the @types scope?

[mhegazy]

Packages in the @types scope are auto generated from definitions on https://github.com/DefinitelyTyped/DefinitelyTyped using https://github.com/Microsoft/types-publisher.

So to get a package published to @types you need to submit a PR to DefinitelyTyped.

[ejsmith]

I don't like how this feature is dependent on a single github repo and forcing us to put our typings there. Does this seem like a bad idea to anyone else?

Also, it sounds like you are recommending that even lib authors should put their typings here even if we are generating our own typings? Shouldn't they just be included in the npm package if possible?

[mhegazy]

I don't like how this feature is dependent on a single github repo and forcing us to put our typings there.

Issue https://github.com/Microsoft/types-publisher/issues/4 is tracking supporting redirects.

Also, it sounds like you are recommending that even lib authors should put their typings here even if we are generating our own typings?

No. Publishing declaration files with your packages would be the preferred course of action. for packages that do not include their declaration files, @types, would be the correct place for them.

[ejsmith]

Ok, thank you for the clarification.

[felixfbecker]

@RyanCavanaugh

NPM has version numbers too. We intend to use the X.Y.z scheme where X.Y is the version of the targeted package and z is the version of the typings file.

So you would have to always use an exact dependency instead of a caret ^ or tilde ~ because a declaration change is often a breaking change in a way that it wont compile without code changes. So when a new module version comes out that automatically gets installed because your package.json semver range, you wont get the new typings for it, because you have to lock the version to prevent breaking changes. typings on the other hand, because the declaration version is tracked independently, could in theory allow us to specify a caret dependency and lock the declaration to prevent breaking changes: "sequelize": "registry:npm/sequelize#^3.0.0+20160604152601"

Imo using the patch section of the version for declaring the declaration version (that may also be a breaking change) is completely against semver and npm's idea of the version field in package.json.

We'll have tags for the latest version of a file compatible with each released version of the language starting at 2.0. So you could npm install @types/foo@ts2.1 or whatever if you need the 2.1-compatible file.

Ok, so you can tag the last compatible version of a typing, but how do you tag the last compatible version of the 3.x branch and the 4.x branch of a module?

We'll also be adding support in DefinitelyTyped (and other type definition sources) to have multiple types checked in, so you could have e.g. /jquery/1.x/ and /jquery/2.x folders so they can be maintained separately if that's desired.

Putting all typing definitions in a single gigantic repository, where every module often only has a single .d.ts file with over 3000 lines of code and every change is done through a PR just doesn't scale. That's why typings moved over to one-repository-per-definition scheme, where the declaration structure mirrors the actual module structure and typings taking care of wrapping the definitions in declare module '...' blocks. They can have their own tests, CI, PRs etc and are tracked in the typings registry. Instead of one file per version, they have a typings.json and actual versions, branches etc. This is much much better than DefinitelyTyped, and every definition on DT should be moved over to this new way.

[felixfbecker]

Currently we have

• DefinitelyTyped, which has a number of problems I outlined above
• typings, the new definition manager that has great potential, but is still young and most of the typings are still in DT
• bundled definitions in modules, which is not every module maintainer is willing to support

I feel like instead of introducing yet another way to publish and install typings the TypeScript team should rather improve typings even further (for example, we could make installation easier by automatically reading package.json and searching for typings) and make it the recommended way™ to install typings.

TL;DR Scrap this idea and put the effort into typings

[RyanCavanaugh]

For large and popular libraries, I agree that having a separate repo is the way to go. We fully intend to support that scenario by allowing redirection from DefinitelyTyped to other repos so that well-maintained libraries can have their own repo.

For the ~80% of other libraries on DefinitelyTyped that don't have strong specific ownership and are managed on a more ad-hoc basis, having a separate repo for these is bad because it scales badly in the other direction. We could easily end up with six people having their own repo for calq, not knowing which is preferable. What happens when the owner for that repo doesn't use the library anymore? If I want to contribute a fix to this file, whose repo do I go to? If I'm a maintainer for DefinitelyTyped, how do I see all the pending PRs for the un-owned libraries out there?

Having individual repos for well-owned definitions and a catch-all repo for ad-hoc-managed definitions is really the best of both worlds. I fully expect e.g. React to have its own repo, but having a separate repo for every definition leads us to the exact XKCD you just posted.

[felixfbecker]

@RyanCavanaugh

We could easily end up with six people having their own repo for calq, not knowing which is preferable.

that's where the typings registry comes in. The idea is simply that instead of hosting the whole definition in the repo you just host a reference.

If I want to contribute a fix to this file, whose repo do I go to?

typings info <package name> shows you the source repository.

[RyanCavanaugh]

I'm just not sure what you're going after here since we're basically supporting a superset of scenarios over what you're describing. Separate repos will be supported!

Because there are many type definitions that don't have (or need) a specific owner, we don't want to raise the barrier to entry here to the point where submitting a four-line definition requires setting up your own repo, sending a separate PR to a registry, and implicitly agreeing to maintain it for the rest of your life. Having non-isolated definitions means we can have shared maintainership of the long tail of files, in one place. We're going to be stepping up our commitment to DefinitelyTyped in the future to decrease the latency of PRs; having the additional friction that we suddenly need a massive cross-repo PR query and pull request merge rights on 1,600 other repos isn't actually getting us anything in return.

Again, you can still have a separate repo for your type definitions. That's a key scenario for well-owned definitions, but we have to consider the "long tail" of definitions as well.

[felixfbecker]

@RyanCavanaugh Valid points you raise there about DT, I didnt look at it that way until now. It may really be beneficial to keep DT for small definitions (even though, I would ideally like to see module authors bundling that four line definition, I mean come on guys, it doesn't cost you a penny :D). But typings supports DT... So I still don't see why we need npm.

[anaisbetts]

@mhegazy That's awesome

[sjansen]

I welcome needing to install one less tool. Consolidating on npm is a simple, clean solution.

[blakeembrey]

@RyanCavanaugh FWIW, that definition you reference is actually in the repo itself already 😄 We should make sure deprecation is handled effectively by the publishing tool also.

Aside from that, the way I've been working with separate repos is by adding a large collection of people to @types and people can add repos there knowing the rest of the community already has access. Of course, this doesn't work for first time creation where you'd need to make your own repo, but you could always move the repo later if you get added to @types (for example). Do you imagine owning a @types like organisation (or I can add the MicroSoft team to @types)?

[kourge]

The pros of this are immediately clear:

• Use npm for what it's best at: dependency and version management
• Does not require additional tool and thus lowers entry barrier
• Establishes a more confident "authoritative" typing status without diminishing existing sources of typings
• For those who have their own npm org / user scope, typeRoots provides a promising way of leveraging npm to add your own custom typings.

I can only think of one possible con: assuming the worst possible scenario wherein none of your dependencies ship with their own typings, this approach results in roughly double the number of package.json dependencies, in contrast against the typings approach, where your typings.json would roughly parallel package.json but would remain as a separate file.

[blakeembrey]

@kourge Can you point me to where you found typeRoots? Last I knew it was being omitted because of dependency complexity.

[mhegazy]

typeRoots is not omitted. it is still in.

[blakeembrey]

@mhegazy Did you resolve how you'd be handling sub-dependency being emitted where the typesRoot property has been customised?

[unional]

definition you reference is actually in the repo itself already

And redirection is also pointing to separate repo. :smile: Also I would guess redirection will be something pretty close to typings/registry.

[unional]

Another benefit @felixfbecker already mentioned and I want to emphasis on: testing

Testing in DT (having typings in one giant repo) just doesn't scale, and the "shape test" done in DT is simply not sufficient: https://github.com/typings/generator-typings#about-writing-tests-for-typings

[kourge]

@blakeembrey in the root comment of this issue, under the Global declaration files section, leading up to the types option.

I think the potential in using typeRoots to establish a secondary @types scope for the purpose of typings resolution only serves a very fringe scenario. For example, in one of the projects I own, we have a webpack resolve alias from foo to ./bar.js. The way we keep the foo module typed is by using typings combined with a file: source, e.g. typings install --save foo=file:custom-typings/foo.d.ts. With typeRoots, I could instead package foo.d.ts into an npm package of my choosing, say @org/foo, and add node_modules/@org/ to typeRoots. But another way to solve this would be to simply move bar.js into such a package and make it a full-fledged npm package that comes bundled with typings, and this way does not require typeRoots.

Are there any concrete features that are to be introduced for universal modules? (If I understand those correctly, those usually fall under global dependencies in typings, e.g. mocha, right?) The types compiler option is a good start.

[blakeembrey]

@kourge Sorry, I was aware of the feature, I was just commenting on the complexity around it. Last I talked with the TypeScript team I believed it was going to be omitted because of the complexity. For example, say you have a module which has a dependency that has customised the typeRoots property - without that working transitively (E.g. it needs to be stored within each .d.ts file or somehow accessible) then any sub-dependencies of the dependency will fail to resolve. I'm glad if it's been resolved 😄

Edit: Though I did miss the mention under the section you pointed out, cheers! I wrongly assumed the post was a mirror of the blog post. I think it also had another name before this announcement.

[cnshenj]

What about d.ts files created by internal projects which are not published to npm?

For example, we have a shared project which provides common modules. We simply copy the d.ts file to other projects' "typings" directory and other projects can import these modules. We do have mechanism to guarantee that shared project is always loaded and available in run time.

Can we do the same when using TypeScript 2.0? Will TS 2.0 provide better way to generate combined d.ts file for a whole project instead of d.ts files for each ts file?

[mhegazy]

What about d.ts files created by internal projects which are not published to npm?

You can still import these the same way you do today. no change is needed.

Alternatively you can define them in the same format @types packages are, and use --typeRoots command-line option to point the compiler to their location.

Can we do the same when using TypeScript 2.0? Will TS 2.0 provide better way to generate combined d.ts file for a whole project instead of d.ts files for each ts file?

You can use --outFile today to combine your declaration in one file.

[mhegazy]

For node modules, please note that npm-link should now work as expected. see https://github.com/Microsoft/TypeScript/pull/8486.

[aluanhaddad]

While I appreciate the effort to standardize on conventions, distribution, and esspecially semver, and while I think npm is both a natural and sound choice as a delivery mechanism and versioning system, I am concerned that this proposal is slightly overly coupled to npm itself. Specifically, the use of the node_modules folder. Nested directory structures which simply use the package name itself like the example for lodash can be problematic. I'm concerned with multi version support and I'd like to see support for browser oriented technologies, such as jspm, that use a flat directory structure with versioned folders. Basically what I'm saying is that TypeScript is about JavaScript not specifically about node.js and I would like to see framework agnostic support of some kind. It makes a lot of sense to use npm but I think TypeScript should support additional configurations and be extensible for new conventions.

[mhegazy]

I would like to see framework agnostic support of some kind

The problem here is that npm, jspm, bower, etc.. are not frameworks per se, they are package managers. Any solution that supports them seamlessly needs to be coupled with the internals of the package systems, package definition, naming, special folders, resolution semantics, etc..

At the core, what the TypeScript compiler is trying to do is locate a declaration file given a module name, and a package manage.

JSPM support is tracked by https://github.com/Microsoft/TypeScript/issues/6012; and solving this will need to understand JSPM naming model and how to consume JSPM config files.

I'm concerned with multi version support and I'd like to see support for browser oriented technologies

TS 2.0 has added support for also baseUrl and path mapping, these are not npm-concepts, these are AMD/require.js initiated concepts. So i do not think it is all about npm.

As I mentioned earlier, we have tried to do resolution in a non-host-specific fashion, and that end up not working on all hosts. so you have to bake in knowledge about these package manager to get an seamless experience.

[aluanhaddad]

Absolutely. I have been using baseUrl and path mappings for several months now, and they work quite well. I also understand that baking in knowledge about every package manager is neither desirable nor practical, but I would like to see an effort to define extensibilty points which allow tools to hook into the .d.ts resolution file process when installing dependencies. This is especially relevant for transitive dependencies as @blakeembrey mentioned earlier.

I do not expect seamless support for everything out of the box, but it would be nice if, for example, resolution lookup would not rely on the official published name of a dependency, nor on the assumption that transitive dependencies are installed within direct dependencies. At the very least, remapping names seems very important and is a key concept of ECMAScript modules in the sense that they are anonymous.

[aluanhaddad]

Also I really appreciate what the team has already done by supporting baseUrl and paths, they are very powerful. The TypeScript team has shown an unending dedication to supporting the breadth of the JavaScript ecosystem. The support for different module formats, the effort to remain compatible and agnostic, and the real attention payed to community suggestions and concerns is truly fantastic.

[mhegazy]

thanks for the kind words. and hope we can make most of these scenarios work correctly and seamlessly.

[joost-de-vries]

I'm looking forward to using the npm way of getting type definitions. Unfortunately the npms are missing the repository metadata. Unfortunately the build infrastructure that I use requires npms to contain this standard metadata.

So my request is for the repository metadata to be available in @types npms.

[weswigham]

@joost-de-vries You should consider adding an issue for your requirement at Microsoft/types-publisher, which is where the tool that generates the packages which get published to @types currently lives.

[joost-de-vries]

@weswigham ah, sorry. I didn't know that was the repo. Thank you, I will.

[seocamo]

To improve typescript typed file .. add a control flag to tell typescript to use the d.ts file it got but only warn about any missing file and just see them as "any" ... This way you can get types in places where it helps and not be a pain when it not wanted ... This way typescript can also be use as a superset of js

[mhegazy]

@seocamo for these scenarios, i would recommend using shorthand module declarations, see https://github.com/Microsoft/TypeScript/issues/6615. you can choose what module you want to treat as "any" and what you want to type check. so you can do something like:

declare module "lodash/*"; // any import to a lodash submodule would not be an error, and will result in any

[remojansen]

Usually, new declaration files are submitted to DefinitelyTyped, a community-driven repository of declaration files.

I think you guys need to focus not only in what happens after the .d.ts files reach DefinitelyTyped. We need something that will allow us to have control over the source and release of the .d.ts files.

I have noticed over the last months that it now takes over weeks for a PR to be merged into DefinitelyTyped:

.

Unfortunately, there are issues with the way a definition file gets from there to your project today.

About what matters the most for the distribution is for sure the correct versioning. It would be great if we could use versioning to ensure that the type definitions that we are using are compatible with the version of the library that we have installed. This is possible using npm.

[DanielRosenwasser]

@remojansen we are planning to have the team be a lot more seriously invested in contributing to DefinitelyTyped, so things should speed up in that area.

[CreepGin]

It would be ideal if there can be an automated way to transition existing typings or tsd setup to the npm way. Also what will be the equivalent of typings search? npm search is dramatically slow.

[DanielRosenwasser]

@CreepGin https://aka.ms/types is currently one way to do searching. npm will potentially support searching scoped-packages in the future.

[CreepGin]

@DanielRosenwasser Thanks, I'll save that link. I just tried to convert my typings setup to npm and noticed that the new setup breaks module augmentation.

For example, I had the following module augmentation for redis:

declare module "redis" {

    export interface RedisClient extends NodeJS.EventEmitter {
        setAsync(key:string, value:string): Promise<void>;
        getAsync(key:string): Promise<string>;
        existsAsync(key:string): Promise<boolean>;
    }

}

It worked fine augmenting the old redis definition. But with the new npm version, it would just "steal" the module name, rendering the redis definition from npm useless.

I usually use module augmentation to deal with alpha releases of libraries so I can quickly patch things and test. If this is no longer possible, what's the alternative?

[CreepGin]

Module augmentation still works fine on definitions with declare module on top. But it fails when some of the npm types (ex: redis and joi) only contain a bunch of exports.

UPDATE: Later on I was able to get module augmentation working (for redis and joi) by moving the code from the .d.ts file to the place where redis is actually being used. Can someone else see the same behavior?

[frogcjn]

Will you make a tool to automatic grab d.ts files for JS library?

[jednano]

I built the PostCSS declaration files and somehow managed to convince @ai to accept them into the official repository, but he's recently stressed interest in removing them from the repo, concerned that 99% of the users won't need them and so why burden them with more files, just to support the 1%? It's a valid concern, which is why I wish there were a better way to install declaration files.

How "cool" would it be if the TypeScript team could collaborate with npm in such a way to introduce something like this?

npm install postcss@5.0 --include-typings

This way, the declaration files could still be published with the official repository, complete with versioning; however, they would only be installed if one were to opt-in with --include-typings or some flag.

Can we not do something like this? I realize it would require reaching out the npm team, but can't we at least try? For Node projects at least, this would solve the versioning issues and make project owners that don't necessarily love TypeScript perhaps more willing to incorporate them into their projects and even publish them w/o fear of burdening "the 1%".

[aluanhaddad]

@jedmao I'm sorry to hear that, it's just a single file and even for JavaScript users it allows for better tooling. For example JetBrains have been using DTS files to power JavaScript intellisense for a while, and now we're seeing the same thing in Visual Studio and Visual Studio Code, as a formalized part of the typescript language service! This is going to mean that more and more IDEs are going to support it so it's definitely a good idea for the file to remain.

I would be curious if there has been collaboration with the npm team.

[ai]

@aluanhaddad my dream is to have a special typescript: { typings: URL } property in package.json, so JetBrains or VS will download typings only if user really develop with PostCSS API :).

Most of PostCSS users don’t use JS API, so it is better for us to reduce npm package size.