Allow "Compiler Plugins"

[mohsen1]

From wiki:

TypeScript Language Service Plugins ("plugins") are for changing the editing experience only

TypeScript plugins are very limited. Plugins should be able to:

• Apply Transformers
• Provide type definitions
• Override module resolver (LanguageServiceHost#resolveModuleNames)
• Emit code beside TypeScript standard output

[DanielRosenwasser]

@rbuckton and I have some offhand thoughts about this

1. The scope of this is fairly broad - does this include generating new nodes for type-checking? If so, that makes this a much larger item.
2. When it comes to module resolution, there are quite a few subtleties.
   1. Custom module resolution also means providing some special module resolution "host" which can provide this behavior in the editing scenarios.
   2. It's not clear how custom module resolution works between dependencies & dependents.

In general this isn't simple but we're open to at least hearing ideas.

[mohsen1]

I'm not familiar with TypeScript well enough to write a proposal. Instead I can list a few plugins that can be useful and exist in the wild in other forms (Webpack plugin, Babel transforms) to make a case for having such extensibility:

Custom module resolution

• Typed CSS Modules
• JSON-loader
• YAML-loader
• Loading any type of serializable file with types

Code transformers

• babel-plugin-jest-hoist
  • issue
• Minifiers
• Removing debug code

Emitting other code

• A Swagger code generator embedded in TypeScript plugin
  • gist about this idea
• Functionality similar to Webpack: emitting image and other asset files

Providing types

• Typed GraphQL query responses
  • This applies to pretty much any database query language. Result of a query can be typed based on query

There are so many other use-cases for compiler plugins that I'm not aware of but I'm sure compiler plugins will make TypeScript ecosystem thrive.

[phra]

i think that this can be a very powerful feature. in particular i'm interested in points 1, 2 and 4 at the moment.

[ryanelian]

Big yes to this feature being supported by TypeScript.

I propose that the plugin system should be implemented using streaming pattern.

// NodeJS stuffs
import * as stream from 'stream';
import * as path from 'path';

export enum TypeScriptModuleEnum {
    CommonJS, AMD, System, UMD, ES6
}

export interface ITypeScriptTransform {
    (filename: string, module: TypeScriptModuleEnum): stream.Transform
}

////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
// As sample plugin to transform file content into string if matches extensions. Handy for templates.
////////////////////////////////////////////////////////////////////////////////////////////////////////////////////

// Plugin creators can use this to extend TypeScript!
import * as through2 from 'through2';

export interface IStringifyOptions {
    extensions: string[]
}

function doStringify(filename: string, extensions: string[]) {
    return extensions.includes(path.extname(filename));
}

export function StringTransform(options: IStringifyOptions): ITypeScriptTransform {
    return function (filename, module) {
        return through2(function (file, encoding, next) {

            // Determines whether we should stringify the file.
            // For example, the file name = 'test.txt' and extensions list = ['.txt', '.html']
            if (!doStringify(filename, options.extensions)) {
                return next(null, file);
            }

            let s = JSON.stringify(file);

            if (module === TypeScriptModuleEnum.CommonJS) {
                s = 'module.exports = ' + s + ';\n';
                return next(null, s);
            } else {
                return next({
                    message: 'Module not supported!'
                });
            }
        });
    };
}

////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
// Later...
////////////////////////////////////////////////////////////////////////////////////////////////////////////////////

import * as ts from 'typescript';
import { StringTransform } from 'StringTransform';

let tsString = StringTransform({
    extensions: ['.html', '.txt']
});

ts.useTransforms([tsString]);

////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
// Later in application source code...
////////////////////////////////////////////////////////////////////////////////////////////////////////////////////

import s = require('./test.txt');

This system has the following benefits:

• The vast majority of browserify and gulp plugins (which are mostly written using through2) can be ported to TypeScript easily!
• Transforms can be executed sequentially by TypeScript during compilation, in-memory, using the order defined in the passed transform array.
• Allows file manipulation freedom by plugin developers.

[cameron-martin]

Something similar to browserify transforms or webpack loaders would be very powerful and cover most of these use cases.

[phra]

looking forward to see this feature implemented! :raised_hands:

[data-ux]

Currently program.emit() already has customTransformers as possible parameter, but it isn't exposed to consumers of the tsc command line program. It would be great to be able to give transformers in compilerOptions as proposed in #14419. Currently to use customTransformers you have to use the Compiler API and re-implement all the functionality in tsc like watching files etc.

[mohsen1]

Transformers do not allow custom module resolution or extra file emit

[data-ux]

@mohsen1 Yes, you're right. I was suggesting it as an approach for your first point "Apply transformers". For a plugin to do all the things suggested in your issue description is too broad of a scope, as @DanielRosenwasser noted. I think exposing custom transformers is the highest value feature of the suggested and it's also the most simple to implement taking into account the way the TS compiler currently works.

[WiseBird]

@Jack-Works Isn't Language Service doing it already?

[geovanisouza92]

Any news on this? I'm writing a plugin, and would like to just plug it on my current setup (tsc & webpack + awesome-typescript-loader).

[fis-cz]

I am sorry. I didn't know it is possible to write language service plugins... I should read before I write. So, for the compiler it can be implemented in the similar way as for ls.

Regarding this topic I have some tips.

It would be great if loading of plugins is configurable through the tsconfig.json file. Thats because i.e. VS code syntax highlighter / lens / intellisense will use the same plugins as the compiler during the regular build. Plugin can be standard Node module and can be resolved in the standard CommonJS way.

When the plugin for the typescript compiler will be defined in the tsconfig.json file it should be loaded during the tsc startup and tsc should provide access to all currently available tsc APIs (it would be also great if extended tsconfig can be read through API too as when I was playing around the API about year ago i had to write custom config reader / extender, what is not good as with next release of ts you can remove or add some options and its hard to maintain the code afterwards ;).

During the init phase, the plugin can replace various stages of the compiler API with custom implementations (such as file reader) or bind event listeners to events occurring during the compilation process (file load, resolve, parse, transform, emit...). Events with "pre" and "post" would be also great in order to pre-process or post-process the stage data while original components are still in use. I.e. preParse is great time to run text preprocessor which can implement #ifdefs and replace them with empty lines to keep it possible to generate source maps properly, or postParse when AST can be searched for dead code and the dead code can be removed from furthermore processing.

From my perspective, it would be much easier to implement call puigin.init(...) with references to all available tsc components and let plugin developer to choose if he will replace it or not what would be specified in the return object. I'll update this later once I'll check tsc sources.

If this would be possible we can simply use various plugins for code preprocessing, death code elimination, output minification or whatever else we can imagine directly under the hood of the compiler "executable" but without touching the compiler code itself. Currently, we have to write everything as a new compiler using the tsc API. Unfortunately, this later means we have to implement the "new" compiler to our development tools (such as VS code or full VS, what is almost impossible ;).

[pcan]

Events with "pre" and "post" would be also great in order to pre-process or post-process stage data (i.e. preParse is great time to run text preprocessor which can implement #ifdefs and replace them with empty lines to keep it possible to generate source maps properly, or postParse when AST can be searched for dead code and remove it from the furthermore processing.)

I would really need this one. The ideal spot for synthetic code injection is after the parsing phase: here you have the AST ready, you can do some enhancements, and they are already available to language service!

[fis-cz]

I would really need this one. The ideal spot for synthetic code injection is after the parsing phase: here you have the AST ready, you can do some enhancements, and they are already available to language service!

I think it is enough if you can replace the parser with custom one and inside of it you would do your pre, call original parser with modified input and your post where you would modify the AST.

[AzzkiyOne]

Please make it happen. There is just too much need in this. Whenever i find myself in need of extending the typescript compiler with some feature (typed css modules, #10866) i always think "This is definetly can/must be done with a plugin". Babel has a plugin system, and people have an opportunity to do anything they need to do to get a work done, without bothering the core team with requests of new features. Besides, all burden of support lies on the plugin author. I mean that the community can add a ton of requested features to the typescript without bloating the core codebase (which will make it hard to support).

[xtuc]

Now that Babel 7 has the support for TypeScript, would it be possible to achieve that trough Babel?

[krzkaczor]

@xtuc i think this would break all existing tooling etc.

Personally i belive that type providers should be a way to extend typescript (only type system).

[mrmckeb]

I've been working on this plugin, and I would love that it would work during build - and not just when working with files in the IDE.

https://github.com/mrmckeb/typescript-plugin-css-modules

[danielkcz]

I would like to invite everyone to the discussion I've started today that tries to find a way how to make a universal plugin/transform that would work in a similar way as babel-plugin-macros, ideally alongside it to avoid the need for multiple plugins that are hard to configure and use.

Please join and bring your ideas and experience.

[ghost]

My usecase for a plugin, would involve emitting a json file with reflection-like metadata for the API in a SDK description in typescript. Right now we need to manually maintain a json file AND a typescript file for our API. All the information in the json file can be derived from typescript. With a plugin, would like to be able to write code that emitted the json file automatically.

[3cL1p5e7]

I want normal function overloading as C# :D

[justinfagnani]

@DanielRosenwasser is there any chance of starting small and just running existing language service plugins and emitting their additions warnings and errors?

Right now we have a really unfortunate situation where a nice plugin can offer additional type checking during editing, but not fail a build if there are errors.

[DanielRosenwasser]

@justinfagnani I personally think that's the place that LS plugins need to move to over time - it definitely feels strange to have language service errors that aren't build errors.

[twop]

@orta suggested to cross link my tweet: https://twitter.com/twopSK/status/1194117666052567043

I would love to create my own subset of ts but with smaller api surface area and ideally have a custom extension: example *.tss. Things I want to remove: fetch, dom etc.

Context: want to build custom DSL for defining css styles in ts.

Example:

// my_style.tss
export const myStyle = defineStyle({
  backColor: "red"
});

// this function is available in *.tss files
declare function defineStyle(s: StyleDef): string;

// some APIs are not allowed
export const noFetch = fetch(...); // TSError, fetch is not defined

and then seamlessly import it in ts(x) files.

import { myStyle } from "./my_style.tss";
console.log(myStyle); // ASJKH$#Q42  <--- some value produced by 'defineStyle'

Essentially I would love to be able to build a custom ts dialect akin to tsx but with no extra syntax.

code snippet: https://codesandbox.io/s/sketch-of-typed-css-5hlyn

Thanks!

[Peeja]

Similarly, this would enable the type checking of MDX files, which is important for folks using the newer Docs features of Storybook.

[fis-cz]

Just for those who need it I wrote simple AST preprocessor for conditional compilation. Before plugins are implemented it must be run as a complete separate compiler.

See #33800 for details.

[axetroy]

Hi.

All operations of the plugin are now synchronized

I want to know if the Typescript team plans to support asynchronous operations

Eg.

interface LanguageServiceHost extends ModuleSpecifierResolutionHost {
-  getScriptFileNames(): string[]
+. getScriptFileNames(): string[] | Promise<string[]>
}

All operations of the plugin are synchronized

It is possible that some plugins will block the entire Typescript Service until make it crash

[nadavwix]

would love to be able to create plugins that transform non typescript files into d.ts files. these can be used while type checking and will provide a better work experience with non js/ts files.

let all our css files be typed

[glen-84]

@DanielRosenwasser,

The TypeScript 3.8 Iteration Plan included "Plugin Investigation" under the Expected Work Items – was any progress made in that regard?

Will this issue be considered for the next roadmap (July - December 2020)?

[oychao]

Here I need to transform some IDL files into type definition files, by creating a custom module loader (if available maybe)? It's January 2021 now, any progress?

[blaumeise20]

I really love TypeScript but I think that this is a feature that is really missing. I have a few thoughts about it:

Naming - How would the naming structure be like?

How would we publish plugins for others to use them? Of course everyone could use its own name but I think that is not good. Then we will see names like typescript-plugin-css or svelte-for-typescript-plugin (I'm using Svelte) on npm and that is not beautiful. There has to be something like a naming convention like the BabelJS plugins have. How would that look like? A few options:

• typescript-plugin-xyz
• plugin-typescript-xyz
• xyz-plugin-typescript
• @typescript/plugin-xyz (like BabelJS)
• @typescript/xyz-plugin

We would have to choose one of them or just let people choose one for their plugin, but that doesn't seem to be so great to me.

Another thing also included below is local plugins (no npm modules). I don't think we should add a naming convention to them, because people have to choose themselves what they do in their projects.

Loading - How do we tell the compiler to load a specific plugin?

There is currently a plugins compiler option but it is only for IDEs. To not get confused with compiler plugins I would recommend to add a new config option called compilerPlugins. It would take an array of plugin configurations. I imagine it like this:

{
    "compilerOptions": {
        // some other options...

        // array of plugins
        "compilerPlugins": [
            // version 1 - string including the path or module name
            "@typescript/plugin-cool-features", // npm modules
            "./typescript-cool-plugin.ts", // for local ones

            // version 2 - configuration with an object
            {
                "name": "@typescript/plugin-cool-features-with-options",
                "options": { // add type checking for this
                    "coolOption": "cool value"
                }
            }
        ]
    }
}

Implementing - How do we write the actual plugin implementation?

Plugins should always be written in TypeScript. All functionallity should be exported in one single file, the file registered in compilerPlugins. I don't know how that should be done best, if the file should export an init function or a class. I would prefer a class. I'm unsure about the actual structure, but something like this would be cool:

import { Plugin, FileType } from "typescript/plugins";
// notice this, there should be a base class and some helpers in "typescript/plugins"

// enough to allow resolves of ".svelte" files like `import App from "./App"` with file "App.svelte"
class SvelteFileType extends FileType {
    name = ".svelte";
}

interface Config {}
export default class CoolPlugin extends Plugin {
    static info = {
        // don't know if this is needed
    }

    init(config: Config) { // can request config
        this.context // current context (can add things like providers and listeners and is used to remove all added things when disabling this plugin)

        this.context.parser.on("preparse", () => {});
        // i don't know what things should be possible here

        this.context.registerImportResolver(data => {
            return "resolvedFile.ts";
        });

        this.context.registerFileType(SvelteFileType);
    }

    remove() {
        // optional, if you have any sub-objects to destroy do this, all listeners are removed automatically
    }
}

Example - Implicit this

Because I don't know how the implementation would be done, I can't provide a code example here, but basically it would allow you to access properties of the class without using this. Look at this example:

class Example {
    constructor(readonly name: string) {}

    sayHello() {
        console.log(`${name} says hello`);
    }
}

Like in Java or C# this would be helpful if you want to store a lot of data used in various methods.

---

What do you think about these ideas? If you have other ideas please tell me!

[justinfagnani]

@DanielRosenwasser following up on this:

@justinfagnani I personally think that's the place that LS plugins need to move to over time - it definitely feels strange to have language service errors that aren't build errors.

Would it be helpful to open an issue focusing specifically on just running the LS plugins that can already be configured in tsconfig? Is that something that's actionable before designing a more full-featured compiler plugin API?

We have a lot of customers who would benefit from more build-time checking, not just editing-time checking, and they're sometimes and understandably wary to replace tsc with ttypescript.

Also, it does seem confusing to me that the "plugins" config option is documented on the TypeScript site, but tsc doesn't run them.

[Rush]

Transformers would be a huge fin in simplifying our toolchains.

[tea-lover-418]

This would be huge for a project I'm working on where the goal would be to generate an application based on typings. The issue that I can't really work with interfaces at runtime would be solved if I could inject code in the compile step.

[Jack-Works]

Is it possible to extends this possibility to a set of public APIs that allow custom file types (e.g. CSS Modules, GraphQL, ...) to be able to generate in-memory declarations and join the incremental building / watch mode building?

(copied from https://github.com/microsoft/TypeScript/pull/39784#issuecomment-922238282)

[matthew-dean]

It would be great if TypeScript natively supported pre-processing plugins / transformers so that someone in the community (or TypeScript) could create a system like Flotate, and solve the problem of JSDoc being hard to work with to define types (and still avoid compilation).

[matthew-dean]

Funnily enough, even though I left the last comment, I had another use case for this pop up this week, which was I think I should be able to use a project like typescript-rtti (provides runtime type reflection) without requiring a fork of TypeScript.

[yanniser]

Not sure to understand well but is this topic related to the fact that TypeScript plugins (like typescript-strict-plugin) don't work at compile-time ?

[mrmckeb]

With allowArbitraryExtensions, I can imagine that a lot of engineers would like to see virtualised declaration files (.d.ts) for those extensions - rather than having watchers/generators, and having to decide if they should or shouldn't commit those files.

This is a perfect example of where plugins could add a lot of value outside of the IDE.

[jakebailey]

I might be wrong here, but I think that with allowArbitraryExtensions, plugins within the current API can proxy the host given to create and provide the right d.ts files that way.

[mrmckeb]

Would that be during compile/build, or only in the IDE @jakebailey?

[jakebailey]

Just IDE, as there are no compiler plugins (that's this thread). e.g. getScriptSnapshot or via the FS.

[mrmckeb]

Do you think the TypeScript team would be more open to compile-time plugins if they were limited to providing type information, perhaps only running with noEmit or a new flag? TypeScript could also report on plugin performance, so users know why commands have slowed down.

This would solve some high value use cases, like importing classes from CSS modules, nodes/queries from GraphQL, etc.

[ericmasiello]

Piling on, I would love tsc to provide compiler plugin support. Similar to others in this thread, in order for me to compile my TypeScript React library, I have to use Rollup + a number of CSS plug-ins to process my CSS module imports. The only "tsc-only" solution is to use a CSS in JS library like emotion. However, that comes with its own trade offs.

[KostyaTretyak]

Since the proposal from this issue has not yet been implemented since 2017, it may require too much change from the TypeScript team.

How do you like the most simplified version: feat(tsc): allow the --appendCommand option to be added in --watch mode. It seems to me that this simplified proposal can be implemented very easily. What do you think about this?

[matthew-dean]

@KostyaTretyak I think, at this point, ts-patch is the short-term, user-land way to go. It just patches TypeScript after install to allow plugins, vs distributed a forked version of TypeScript.

Although I feel like TypeScript should allow this, I agree it might just never happen.