Proposal: Bundling TS module type definitions

[weswigham]

Relates to #3159, #4068, #2568, this branch, and this tool.

Goals

• Bundle declarations for TS projects to allow a library to be consumed with a single TS file, despite being many modules internally. Because of complicated internal module dependencies which would rather not be exposed to the consumer, this should flatten the exported module types as best as it can. (Ideally, completely.)

  Proposal

When all of --module, --out, and --declarations are specified, the TS compiler should emit a single amalgamated .d.ts (alongside its single output js file). This .d.ts should be flattened compared to a concatenated .d.ts file. It should report collisions caused by scoping issues and import aliasing when flattening declarations into a single declare module. It should respect access modifiers when generating the DTS (only exporting things explicitly exported and types marked as public).

For example, given the following set of sources: tsconfig.json:

{
  "compilerOptions": {
    "module": "commonjs",
    "declarations": true,
    "out": "mylib.js"
  }
}

a.ts:

export * from './b';
export * from './c';

b.ts:

export interface Foo {}

export class Bar {
    constructor() {
        console.log('');
    }

    do(): Foo { throw new Error('Not implemented.'); }
}

c.ts:

export class Baz {}

should create the .d.ts: mylib.d.ts:

declare module "mylib" {
  export interface Foo {}

  export class Bar {
    constructor()
    do(): Foo
  }

  export class Baz {}
}

rather than: mylib.d.ts:

declare module "mylib/a" {
  export * from "mylib/b";
  export * from "mylib/c";
}
declare module "mylib/b" {
  export interface Foo {}

  export class Bar {
    constructor()
    do(): Foo
  }
}
declare module "mylib/c" {
  export class Baz {}
}
declare module "mylib" {
  export * from "mylib/a";
}

and should report a semantic error when the following is done: a.ts:

export * from './b';
export {Bar as Foo} from './b';
export * from './c';

as there will be multiple members named Foo (an interface and a class), since b.ts has exported interface Foo.

We should also have a semantic error when the following is changed from the original: If we change c.ts:

export class Baz {}
export interface Foo {}

it should be an error in a.ts (since it's blanket exporting b and c), and the error should suggest to alias either c.ts's Foo or b.ts's Foo (or both) when reexporting them in a.

Internally, when flattening this aliasing becomes important - we need to track usages of the two original Foo's across the generated .d.ts and rename it to the alias created when it is reexported.

Unfortunately, to maintain ES6 compatability, while we can warn about this behavior with classes (since it's possible that a developer is unaware they're overriding a prior export), we still need to support it (or do we? The spec leads me to believe that attempting to export multiple members with the same name - even via export * - is an early syntax error). So it would be nice to have a compiler flag to mark the same kind of thing with classes (or namespaces) as an error, but also do the following by default:

We can do automatic name collision resolution, but that can result in unpredictable (or convention-based) public member names... but it must be done, I suppose. We could ignore reexported types since it's appropriate to do so in ES6 (following export * declarations can override previously defined members? maybe? system works this way at present - but that may just be system relying on transpiler implementers to maintain ES6 semantics), then we would need to create "shadowed" types at the appropriate level in the .d.ts - types whose original public access are overridden by later exports but whose types are still required to describe public function argument or return types. Naming these "shadowed" types could be difficult, but given that they only exist for type information and not for access information, a common (re)naming convention could be a desirable solution. Something akin to <typename>_n when n is the shadowed type number for that type, and renaming the shadowed type name to something else (<typename>__n and so on so long as the name still exists) if that collides with another exported type. Classes used in this way are rewritten to interfaces in the .d.ts, since a constructor function likely isn't accessible for a shadowed class (at least not at its generated exported type name).

Any feedback? There's a few alternatives to what I've suggested here, which is possibly the most conservative approach in terms of ability to error early but supporting ES6 semantics best. It's possible to silently ignore interface name collisions and rename those automatically as well, but since they're TS constructs and not ES6, I think it's okay to force more discipline in their usage.

Something I've been considering is also rewriting namespaces as interfaces in the generated .d.ts in this way to further flatten/unify the types, but this... might? not strictly be needed. I haven't come up with a strong case for it.

[weswigham]

I realize that I've forgotten to propose a way to define the entrypoint module (as is possible in #4434), and I suppose that warrants discussion, too.

The entrypoint is the location which (in the above example) TS considers the 'top level' for flattening types (in this example, a.ts). Ideally, this is the entrypoint to your library or application. All of the dependent files in a well-formed module-based TS project should usually be accessible from the entrypoint via either imports or triple-slash refs... however In TS's default mode of operation for the --module flag, TS ignores most relationships between included files and compiles them mostly separately, resulting in separate js and dts files for each. Like the proposal in #4434, dts bundling may also make sense to require a --bundleDefinitions [entrypoint] flag, like how bundling module sources could require the --bundle [entrypoint] flag.

On the other hand, rather than add a new compiler flag, we could consider all files specified as 'root' files as entrypoints when compiling with --module and output a definition file for each of them. (Meaning that all other files need to be accessed by those root files and compiled via that relationship.) Conceptually, we could do the same thing with #4434, rather than having the --bundle argument to specify an entrypoint. This does lose the meaning of the --outFile argument, however, since there are suddenly multiple output files (one for each root file) and none of them correlate to the --outFile parameter... So maybe it's best to not try to rely on traversing the dependencies ourselves and require an extra parameter specifying the entrypoint.

[NoelAbrahams]

@weswigham,

I'm generally in favour of this proposal. I think this feature would help to formalise the idea of a project in Visual Studio (i.e. types defined in namespaces compiled into a a single output file, that can then be referenced by other projects).

One of the problems with a single output file is that it makes life difficult when debugging in the browser. See https://github.com/Microsoft/TypeScript/issues/192#issuecomment-53185411. Ideally something can be worked out for that as well.

[weswigham]

3159 implements most of this proposal, though it restricts it to ES6 or commonjs modules, uses slightly different terminology for tsc command line flags. and omits the type flattening aspect of this proposal. At the very least, it's probably an excellent starting point for talking about this.

[alexeagle]

some notes from doing this for Angular:

• should also report a semantic error when a symbol is re-exported to an entry point, but a dependent symbol is not (for example, declared supertypes, parameter types)
• we like to emit a namespace as well as a module, allowing users to use symbols from that global namespace without any import statements (eg. useful for ES5 users in VSCode to get intellisense).
• we might want to be able to use this typings bundle feature without having to use the runtime emit bundling feature from #4434 - for example if there are some errors producing a working emit because of private APIs, it would be nice if we could still use this to produce the public API doc.
• We like to preserve some comments so that tools can show inline doc when showing eg. a completion tooltip. Should probably explicitly mention comment handling in the proposal.

[alexeagle]

We can't drop our current .d.ts bundler without constructor visibility: https://github.com/Microsoft/TypeScript/issues/2341

[ffMathy]

What is the current state of this issue?

[mhegazy]

It is still in discussion. we have a PR for it, but there are still some open questions.

[ffMathy]

Can you point to this pull request?

[mhegazy]

I should clarify, the concatenation is already done in: https://github.com/Microsoft/TypeScript/pull/5090 The remaining piece is the flatting, here is the PR: https://github.com/Microsoft/TypeScript/pull/5332

[PavelPZ]

I think that d.ts bundle works very nice, see angular#5796.

Thanks a lot for it.

[heruan]

I jumped to/from many issues regarding this "theme". Is this the right one where discuss?

In my opinion the compiler should be able to produce also separate bundles, like this:

{
    "...": "...",
    "moduleDeclarationOutput": "./modules",
    "modules": {
        "module-a": {
            "from": "./src/module-a/index.ts"
        },
        "module-b": {
            "from": "./src/module-b/index.ts"
        }
    }
}

having then in ./modules/module-a.d.ts the definitions of types exported by ./src/module-a/index.ts and respectively for "module-b".

[davismj]

requesting an update on this please

[mhegazy]

nothing done since last update. no plans to do anything for TS 2.0 at this point.

[davismj]

@weswigham What do you think would be a best strategy in light of this? Would you recommend building a custom TypeScript with your PR?

[weswigham]

I wouldn't recommend it, no. TBH, with recent changes to allow module augmentation and even proposals for relative paths in module augmentation, we may have a better way to do this now which can even preserve secondary entry points.

On Mon, May 16, 2016, 6:24 PM Matthew James Davis notifications@github.com wrote:

@weswigham https://github.com/weswigham What do you think would be a best strategy in light of this? Would you recommend building a custom TypeScript with your PR?

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub https://github.com/Microsoft/TypeScript/issues/4433#issuecomment-219566811

[davismj]

Any other options, then?

[kostrse]

As a workaround solution I'm trying to achieve the same using Rollup.

I build my TypeScript modules to ES2015 and then bundle/tree shake them into a single UMD file with Rollup. Works great for JS, I can even bundle third-party libraries into.

The only problem I haven't solved yet is transformation of type definitions, they become unusable after transformation of JS structure.

[adidahiya]

@kostrse we've hacked together a flimsy solution to the global typings bundling problem here: https://github.com/palantir/global-typings-bundler. There are multiple caveats which restrict the kinds of ES2015 module syntax you can use. Happy to accept contributions.

(to be honest I haven't investigated the new UMD module definition syntax closely yet; it's on my to do list!)

[joelday]

I have a similar project that I completed some time ago, but I haven't been able release it yet.

[unional]

declare module "mylib" {
...
}

would limit remapping as described here (https://github.com/typings/typings/blob/master/docs/external-modules.md) IMO this should be performed on the consuming side, maybe there can be another command to "bundle" the declaration for distribution purpose? e.g.:

// for commonjs
declare namespace mylib {
  export interface Foo {}

  export class Bar {
    constructor()
    do(): Foo
  }

  export class Baz {}
}
export = mylib

// for ES6
export interface Foo {}

export class Bar {
  constructor()
  do(): Foo
}

export class Baz {}

[davismj]

@unional okay, but the problem is still that right now if you use outFile and declarations, you can't even do import * as littleOldJ from 'oldJ'. You can do import * as littleOldJ from 'oldJ/moduleA'. And that's the problem to solve here.

[nomaed]

I needed a solution for creating a single d.ts file as a library for a project I'm working on. I ended up writing a library that grabs all of the generated d.ts files, processes them and generates a single library d.ts file.

However, it's nothing fancy. The processing is mostly regex eliminations of un-needed internal imports and exports and then some removal of duplicate alias imports. The output contains exports of all classes, vars, types, enums and etc. on top level, wrapped by a module declaration.

Also, I was trying to create a solution for my need, so I don't know how well it can work with anything that was built other than ES6 modules.

If anyone will be interested, let me know, so I will clean the code up a little bit, make it work stand-alone and will upload it to github.

[jukibom]

@nomaed yes, please! I was about to embark on something similar myself, the imports are causing us issues.

[nomaed]

@jukibom I did upload that module to github/npm and I am using it for our build system, but I haven't written any tests or proper documentation of how to use it.

You can find it here: https://github.com/nomaed/dts-builder I will add some text about how to use it later on today.

You can check and see if it works for you.

[nomaed]

@jukibom I added some info to the README. If you have any issues/questions/suggestions, feel free to open an issue.

[davismj]

@mhegazy Any update?

[ca0v]

@nomaed How does your solution compare with https://github.com/TypeStrong/dts-bundle and https://github.com/SitePen/dts-generator?

[nomaed]

@ca0v From a quick glace, both dts-budle and dts-generator generate a bunch of declare module scopes for each file in the .d.ts, even if the sources weren't structured in this way so that each file is a named module.

That's not the case with (my) dts-builder. It exports everything as a single library under a single declare module name which mimics the .js file that's being generated. I haven't tried it on sources that use namespace/module declarations inside since I am using ES6 import/export and each file is its own module but the result is a single library (file contents separated by webpack). That is what made me write it, since having multiple modules for files is not the way I needed my .d.ts.

Also, I don't have any unit tests right now and etc., it's just this tool that I made for my use case.

[ca0v]

@nomaed sounds like you are describing tsc -outFile="dist/api.js" -d but dts-bundle has outputAsModuleFolder which might cover your scenario. I wish this magic just happened!

[nomaed]

@ca0v Yupp. That's the idea behind it, as soon as I realized that there is no built-in way to generate a single d.ts file, and after seeing that simply concatenating the multiple d.tses [dee-tee-asses?] doesn't work due to conflicts of duplications and etc.

I see that outputAsModuleFolder was added somewhere in the last month, so it wasn't available when I was looking for this capability. I will take a look at it though, and if it does the same job, I will simply switch to dts-bundle and will remove mine :)

[Roam-Cooper]

This feature is important to me as we are starting to isolate our code at our company by putting them into separate npm modules. We've come across the issue of needing a single declaration file for both the shape of the code and the "ambient types" that we need access to, like a C style header.

We've been thinking a lot about Typescript's declarations and now realise that we (and probably others) need to use them in two ways:

• Defining the shape of the code itself
• Exposing types that exist within the code

The second point is very important to us (and afaik relates to this issue) as we are using dependency injection in large parts of our code, which means that since a class is defined but never used, types for that class will never be exposed in the normal way.

It's difficult because we're using commonjs, and there seems to be some different ideas about how commonjs should be used. In the meantime, like a few others have done, we'll be writing a script to grab the type declaration of anything that's exported from a ts file that uses commonjs and concatenate them all into one nice library declaration file.

[blakeembrey]

@Roam-Cooper What's wrong with just publishing the .d.ts files as-is? I've been using that fine for a couple of years now (since 1.5 come out) - all you need to do is https://github.com/blakeembrey/free-style/blob/80253357ba0ce1cba5709b641e8aeca73e0f0cc6/package.json#L6. I haven't run into anywhere you need to put them all into a single declaration file, that was mostly a way to achieve what become NPM @types.

[Roam-Cooper]

@blakeembrey Because it's annoying to deal with a couple dozen .d.ts files rather than a single bundled .d.ts "header" file.

But also because I might not export a class so that I can prevent a developer from instantiating the class themselves, but still want a developer to be able to reference the type/interface of the class.

For example ExpressJS' (handwritten) declaration file exports interfaces that extend existing classes like so:

import * as core from "express-serve-static-core";
function e(): core.Express;
namespace e {
interface Application extends core.Application { }
interface CookieOptions extends core.CookieOptions { }
interface Errback extends core.Errback { }
}

This results in the import being callable to instantiate an Express Application, but also exposes at the top level, uninstantiatable interfaces relevant to particular types where those types might not be exported themselves.

So again, the function declaration in the Express declaration is declaring the shape of the code, but the interfaces contained within the namespace are declaring types within the code.

We could be going down the wrong path here, so let me know if there's a better solution for accomplishing this sort of thing.

[atrauzzi]

I think you can accomplish what you want. Take a look at my current TS project here, it might offer some insights: https://github.com/atrauzzi/protoculture/blob/master/src/index.ts

[davismj]

@mhegazy Is this on the radar for the TypeScript team?

[marlon-tucker]

@blakeembrey

What's wrong with just publishing the .d.ts files as-is? I've been using that fine for a couple of years now (since 1.5 come out) - all you need to do is https://github.com/blakeembrey/free-style/blob/80253357ba0ce1cba5709b641e8aeca73e0f0cc6/package.json#L6. I haven't run into anywhere you need to put them all into a single declaration file, that was mostly a way to achieve what become NPM @types.

In that example, you have a project which consists of one typescript file and so it generates a single declaration file. For large Typescript projects which consist of 100s of ts files, that is an entirely different story.

Looking at the majority of the d.ts files which are on the DefinitelyTyped repo, they are all defined as single declaration files. Granted most of these are from normal javascript libraries and as such had to have their declaration files manually written.

The use of outFile is prohibited when using the built in UMD mechanism, so you have to use commonjs and something like webpack to then generate the UMD structure - but webpack has no concept of declaration files.

Nearly all javascript libraries I can think of are ultimately distributed as single files. If you have to manually define the declaration files even if you are using typescript to generate the library, it seems a bit of a oversight really.

[blakeembrey]

@marlon-tucker No, it's the same process. If you want to find other open source projects with more files, you can look through my repos. I'm guessing you haven't used many package managers or bundlers if you feel all the libraries are distributed as single files. That case, I believe, is pretty rare but definitely extremely common for "getting started" with big libraries on the browser (they'll bundle it to work on window for you). In that case, it's completely reasonable to want this feature, but there's still nothing stopping you from having multiple .d.ts files in both cases either.

[aluanhaddad]

I feel like the key is structuring the package surface correctly from the JavaScript/ES Module point of view. That may well naturally lead to a clean API such that, regardless of how many actual .d.ts files are present, the user will not care.

This would mean that so called "internal" modules would naturally not be re-exported.

@Roam-Cooper

This results in the import being callable to instantiate an Express Application, but also exposes at the top level, uninstantiatable interfaces relevant to particular types where those types might not be exported themselves.

There are several ways to accomplish this. This one is a little hackish but gets the job done simply

export class X { }
export interface Y extends X { }

const x = new X();
const y = new Y(); // error

[Delagen]

For current state I use typings-core package to bundle definitions programmatically, but it bundle it as multiple declare module constructions which may be not suitable for someone.

[timocov]

For our project we have implemented dts-bundle-generator (npm). Maybe it will be useful for someone.

[eps1lon]

Assuming we could bundle all declarations into a single file: How do I tell typescript where this file is located? If I have @material-ui/styles/index.d.ts which declares declare module "@material-ui/styles/withStyles" {} then I can't use import '@material-ui/styles/withStyles' because typescript never aquired index.d.ts. Adding a types field to package.json does not help.

Edit: Fixable by adding "types": ["@material-ui/styles"] to your compilerOptions.

[IgorMinar]

Bundling d.ts files makes sense only if you also bundle your JavaScript.

On Mon, Feb 4, 2019 at 2:14 AM Sebastian Silbermann < notifications@github.com> wrote:

Assuming we could bundle all declarations into a single file: How do I tell typescript where this file is located? If I have @material-ui/styles/index.d.ts which declares declare module "@material-ui/styles/withStyles" {} then I can't use import '@material-ui/styles/withStyles' because typescript never aquired index.d.ts. Adding a types field to package.json does not help.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/Microsoft/TypeScript/issues/4433#issuecomment-460193729, or mute the thread https://github.com/notifications/unsubscribe-auth/AANM6JpB6XBu2ZLxNmterj6mU1P6olzQks5vKAf0gaJpZM4FxZnO .

[eps1lon]

Bundling d.ts files makes sense only if you also bundle your JavaScript. .

I would agree with you if this is indeed only targeted at single output files. However I got the impression from the linked dts-generator that it also supports multiple js files and a single .d.ts file.

I'm honestly confused that typescript would even create multiple .d.ts files for a single out file. Wouldn't that throw at runtime if I import mylib/a when only mylib/out exists (and has the exports from a.ts)?

[Swatinem]

Just for reference, I implemented this as an experimental rollup plugin: https://github.com/Swatinem/rollup-plugin-dts and it works surprisingly well. Feel free to try that and give feedback.

[octogonz]

Bundling d.ts files makes sense only if you also bundle your JavaScript.

We regularly publish packages with bundled .d.ts files, but unbundled .js files. It's useful because the .d.ts file (1) provides a concise, readable summary of the public APIs, and (2) it makes it difficult for consumers to accidentally access internal types that are not meant for consumption. For a commercially supported product, that makes it very clear what is/isn't supported.

We're using API Extractor's .d.ts rollup feature to generate the bundles. It also enables "trimming" of individual class members. For example, members marked as @beta will be stripped from the .d.ts files for an official release, whereas they will be included for a preview release.

[shrinktofit]

Hi all, I also made a such tools use in our company's project. https://github.com/shrinktofit/gift

[kitsonk]

I had forgotten this issue existed and opened #34531. The proposal the Wesley makes would meet the use case we have a need for in @denoland. From my other issue:

Use Cases

Specifically in Deno, we have to generate our runtime type library, based upon the TypeScript code that identifies that runtime environment. We have a single module which defines the global runtime scope, imports all the other modules that define the runtime module and hoists that into the global scope. Previously, we utilised a TypeScript AST manipulation tool (ts-morph) to generate the runtime type library, but removed it when we removed Node.js as part of our build pipeline. We attempted to utilise the outFile approach, but it generated a correct but non-sensical type library. While we wouldn't expect the full generation of the runtime type library, having a single .d.ts file that contained all the exported symbols from the "main" module, plus re-writting all the other imported dependency types as a single flat file, which we can lightly modify to be the default library file for the Deno runtime.

[andyfleming]

Also still looking for support for this.

In addition to outFile, it might be nice to specify outModule if you want the name to be different.

{
  "compilerOptions": {
    "module": "commonjs",
    "declarations": true,
    "out": "./types/lib.js",
    "outModule": "@my-org/example-package"
  }
}

One other alternative to consider would be to declare the module based on the name field in package.json.

[millsp]

I wrote a small script to generate a bundle of your project's declarations:

#!/bin/bash

mkdir -p out

npx tsc ./src/index.ts --declaration --emitDeclarationOnly --out out/index.js --module amd &&

echo "
declare module 'YOUR_PACKAGE_NAME' {
    import main = require('index');
    export = main;
}
" >> out/index.d.ts

It outputs to out, change it as needed, and replace YOUR_PACKAGE_NAME.

What this script does NOT do is to bundle other projects declarations.

[jeremyben]

Here's my take on it : bundle option with tsc-prog.

Faster than others because it bundles in memory, by intercepting emitted ".d.ts" files during the initial build.

Nevertheless, this is a complicated issue. Corner cases everywhere: name conflict (with global symbols and local ones), namespaces import and re-export, external libraries, json files, import type, etc. Just to name a few I tried to handle properly.