Plugin Support for Custom Transformers

[MichaReiser]

Since #13764 has landed, it's easy to write custom transformers. However, if I understand the API correctly, it is needed to duplicate the whole tsc command line just to add a single transformer. This leads to incompatibilities since these, on typescript depending, applications do not support the same features as the tsc command line tool.

It, therefore, would be favored to have a plugin system that allows loading Custom Transformers from third party node modules as this is the case for the language service proxies #12231. These transformers then easily integrate into existing build tools / build workflows.

If someone experienced has inputs on how to implement the changes, I'm willing to create a PR as it would simplify my project tremendously.

[mhegazy]

We do not plan on exposing compiler plugin system in the short term. The transforms are exposed as part of the public API as you noted, and we are in the process of writing documentation and samples for these.

[MichaReiser]

I don't want to have an exposed plugin API. Instead, I would prefere way to register my custom transforms by just specifying them in the tsconfig.json instead of having to use the TS-Compiler API. Because using the TS-Compiler API implies that I can no longer use the tsc command line tool which further can imply that it does no longer integrate nicely into existing build tools and workflows.

[WanderWang]

anything update about documentation and samples ? @mhegazy

[longlho]

@MichaReiser you can write a simple compiler based on the API (sample here). We actually use TS very heavily within Yahoo Finance and the transformer public API have been awesome.

Those are some of the transformers we wrote after it became public: https://github.com/longlho/ts-transform-system-import https://github.com/longlho/ts-transform-img https://github.com/longlho/ts-transform-react-intl https://github.com/longlho/ts-transform-css-modules-transform

@mhegazy lmk if you need help documenting/gathering samples

[MichaReiser]

@longlho Thanks for your sample.

That is what I'm currently doing. However, it makes it impossible to use other build tools created for typescript, e.g. web pack loaders, jest loaders. Besides, how can I use one of your plugins together with one of mine when each of us uses a different frontend?

Therefore, I do believe that the current approach is a good start but not sufficient for a plugin ecosystem. Because using a plugin is too much effort for the user and requires more than just writing the transformation code for the author.

I believe a plugin system like the one of Babel is essential for typescript if the goal is to encourage the community to create and use custom transformers.

[longlho]

@MichaReiser yup. I'm not saying it's sufficient for the ecosystem, just good enough for the 1st step towards it.

[ivogabe]

I'd like to add support for transforms in gulp-typescript, but I want to prevent that all TypeScript plugins (for gulp, webpack etc) propose a different API. And TypeScript might add even a different way for configurating this later on. So do you currently have plans to add this in the near or far future?

[RubaXa]

Hi all, I try to write a preprocessor, but nothing comes out :[

In fact, I have two questions:

• How to create AST-fragment from string?
• How to add import?

// 1. Input
class Foo {
    templateString = 'some value';
}

// 2. After transformation
import __LIB__ from '@external/lib';

class Foo {
    templateString = (function compiledTemplate(deps) {
        // ...
        return result;
    })({lib: __LIB__});
}

// 3. Expected result
var lib_1 = require("@external/lib");
var Foo = (function () {
    function Foo() {
        this.templateString = (function compiledTemplate(deps) {
            // ...
            return result;
        })({ lib: lib_1 }); 
    }
    return Foo;
}());

A Simplified Example: https://github.com/RubaXa/typescript-api-questions/tree/master/import-add

[RubaXa]

⬆️ ⬆️ ⬆️
@longlho, @mhegazy Can you give any hint?

[rbuckton]

@RubaXa Since you added the import declaration in a transform, it isn't bound or type checked. Since it wasn't bound or type checked, we cannot resolve __LIB__ in your expression to the __LIB__ in the declaration.

One option is to use a namepace import instead of a default import, so that your emit is something like:

import * as __LIB__ from "@external/lib"

As there is no aliasing that can occur.

The other hold onto a generated identifier for the import declaration, as per the attached zip

[rbuckton]

@RubaXa if your goal is to pass in the entire module object, you probably want to use the import * as __LIB__ from "@external/lib" syntax (which does not require aliasing) and not import __LIB__ from "@external/lib" as the latter imports the default export rather than the entire module object.

[longlho]

yeah we're doing import * as foo in our CSS modules transformer as well to prevent aliasing. Although it'd be nice to tap into that to inline certain named exports

[RubaXa]

@rbuckton O, thanks a lot! Still interested in how to create an arbitrary AST-fragment from string?

function createFragmentFromString(code: string) {
  // ????
}

function visitPropertyDeclaration(node) {
    if (ts.isIdentifier(node.name) && node.name.text === "templateString") {
        // ...
        return ts.updateProperty(
            node,
            ts.visitNodes(node.decorators, visitor),
            ts.visitNodes(node.modifiers, visitor),
            ts.visitNode(node.name, visitor),
            ts.visitNode(node.type, visitor),
            createFragmentFromString('(function compiledTemplate() { /* ... */ })()')
        );
    }
    return node;
}

[rightisleft]

This is what i was hoping the transformer support and workflow would behave like in TypeScript.

https://www.dartlang.org/tools/pub/transformers

[chandu0101]

I am also very much interested in this feature.

as @MichaReiser mentioned If someone experienced has inputs on how to implement the changes, I'm willing to create a PR as it would simplify my project tremendously.

love to see inputs from experts/typescript compiler DEV .

[MeirionHughes]

Looks like someone has wrapped typescript to expose this functionality. https://github.com/cevek/ttypescript

Also looks like ts-loader has it: https://www.npmjs.com/package/ts-loader#getcustomtransformers-----before-transformerfactory--after-transformerfactory----

[jcimoch]

I think this is something that could land in typescript 2.8 release or at least in roadmap at some point, @ahejlsberg @mhegazy @DanielRosenwasser what do you think? In this way custom transformers might be more popular and therefore more powerfull. Having option to plugin in transfromer from tsconfig.json perspective would simplify life a lot.

[mhegazy]

We have no plans to expose plugins in the short term, as noted earlier.

[raveclassic]

@mhegazy Is it a considered decision or is it out of scope just because of low community interest?

[mhegazy]

I did not say that. we would like to maintain a small public API/maintainability cost as well as flexibility in changing how the build is driven moving forward, and these both are affected by a plugin model.

[zerkalica]

Please, stop transformer plugin api fragmentation. Stabilize plugin api on top of transfomer api and expose them in tsconfig.json.

ts-loader, parcel-plugin-typescript, rollup-plugin-typescript2, ts-transformer-keys, ttypescript and others.

Each of them provides custom plugin registration method and custom plugin entry point format. It's a way to ts plugin hell.

[zerkalica]

Now cevec/ttypescript supports all common transformer plugin formats. It's a drop-in wrapper on top of all typescript modules and tsc, tsserver commands (just small runtime ts.createProgram patch). Webpack ts-loader and rollup-plugin-typescript2 can be easily configured with ttypescript.

TTypescript suggest universal transformer plugin format, but existing transformers supported too.

{
    "compilerOptions": {
        "plugins": [
            { "transform": "ts-transform-graphql-tag" },
            { "transform": "ts-transform-css-modules", "type": "config" },
            { "transform": "ts-nameof", "type": "raw", "after": true}
            { "transform": "./transformers/my-transformer.ts", "someOption1": 123, "someOption2": 321  }
        ]
    },
    "exclude": ["node_modules", "transformers/**/*"]
}

[therealmikz]

Any news on this one? Seems to me like TypeScript authors somehow don't want to allow custom transformers on vanilla TypeScript - that's a shame, it would be an awesome feature.

[MeirionHughes]

they don't want another api/surface to maintain for fear of leading to difficulty breaking things in the internals. Ironically, all these 3rd-party plugin solutions are, because there is no standard by the ts-team, all somewhat different and will break eventually.

[pcan]

they don't want another api/surface to maintain for fear of leading to difficulty breaking things in the internals.

After 3 years working with typescript and watching its evolution, I have come to a simple conclusion. Microsoft has open-sourced Typescript in the sense of the code, but not in the sense of the process. In the majority of the cases, open-source projects are more or less community-driven both in the decisions and in the coding, and this should be the natural way to evolve. Do you remember the IO.js fork of node.js? The community realized that the company interests were not so aligned with the common open source governance model, and they simply forked. I hope this won't happen for TypeScript, but Microsoft should take into account this possibility. To be clear, I'm not blaming the devs, anyway, they do a great job, really. Just my 2c, sorry for the OT.

[kitsonk]

Microsoft has open-sourced Typescript in the sense of the code, but not in the sense of the process.

I wholeheartedly disagree with this. I have never worked for Microsoft, but I have had direct influence on several features of TypeScript over the past 4 years. I was representing an open source library built in TypeScript, and we arranged to have some direct conversations, but all of the "debate" of any features was done open, in public, here. The core team publishes their design meeting notes. The only "insider" I got, representing an open source library, was an opportunity to argue my case for some features in person and get to meet the core team. It made me realise that the team acts as a team. One feature we talked about, Anders basically said the problem was too hard and that it would take a lot of time to chip away at it and solve the "real" problems people faced. That was two years ago and in TypeScript 3.0 we finally get it. But the community has a voice in the design process, but like any community, we have factured and various voices, and no team anywhere could please everyone, and if they did, TypeScript would be a really bad tool.

You also want to label the core team "Microsoft". The core team was the least Microsoft team I have ever encountered. They fought battles to get total control of their release cadence, after they were founded on full control of the their release content. They also fought for openness and transparency. The fought to move to GitHub from CodePlex. Admittedly they aren't that unique anymore in Microsoft as several other teams have adopted their model, but they were, as far as I am aware, the ones who started it all, with the VSCode team following quickly after.

So just because the core team pick and choose their battles, stick to their Design Principles doesn't mean that the community doesn't have a voice, but as a voice we suffer from psychotic multiple personality disorder. We are a difficult customer to serve.

[mrmckeb]

My understanding is that this is the same as #16607, or would this achieve a different goal? As a very recent plugin-author myself, I'd also like to see this exposed - including when working with TypeScript using the Babel plugin.

[MeirionHughes]

@mrmckeb this issue is for standardizing and exposing typescript AST transformation, which already exists, while the linked issue seems to be discussing whole file transformation in a very broad (undefined) scope.

[danielkcz]

I want to add my 2 cents here. Some of you may have heard about the brilliant project babel-plugin-macros. Basically, instead of having thousands of different Babel plugins, there can be the only one which is then powered by a user code to do a transformation. An awesomely transparent system without a need to mess with configuration all the time. This is especially great for projects like CRA which supports macros but forbids additional configuration.

Today I've opened the discussion on how to possibly get this to TypeScript world. If you have some insight, please do come and share.

Ultimately, I hope that if we somehow succeed, there won't be a need for this as only one transform will be necessary.

[dsherret]

@FredyC that's a cool project, but we'd need this issue to be resolved in order to easily inject something like that into the compiler when only using tsc. For example, with babel-plugin-macros it still needs to be specified in .babelrc and it would be nice to specify something like that in typescript in tsconfig.json.

Actually, are you suggesting that it would be nice to have babel-plugin-macro-like behaviour built into TypeScript and no configuration would be necessary? That might be nice, but personally I would prefer a configurable solution that allows specifying custom transformers in tsconfig.json and then a version of babel-plugin-macros that supports typescript ast nodes could be specified.

[danielkcz]

@dsherret Yea, I don't expect it would be a part of TypeScript itself. However, I can imagine that once the transform is ready, it could be easy to prepare a tsc-like tiny wrapper that would be injecting only that transform for macros and rest could be done from a user code.

Edit: Actually, the tiny wrapper is already there, with above mentioned https://github.com/cevek/ttypescript it would be easy to instruct to use that along with the universal macros plugin and it's a win-win.

I just need to find people who are well versed in Typescript transforms to figure some basic prototype. I can discuss and talk, but implementing it is above my current skillset.

[mrmckeb]

I think this is unfair criticism @pietrovismara. I agree that this is a must-have feature, but this isn't "Microsoft" blocking this feature. The team, as I understand it, are saying that this is not easy to support and that's why they haven't added such a feature yet.

Again, I want this feature as much as anyone else - but we need to discuss this issue using facts. The facts are that the team aren't ready to support it yet. So let's keep voicing interest, but also keep the conversation on the right track.

[mrmckeb]

It's OK, it's frustrating that the support for this doesn't exist - as I said, I feel it, and we're asked for some kind of solution for CSS modules all the time at CRA now that we have TypeScript support. I hope it'll come soon, but I also understand that it opens up a huge area of risk for the team.

[SalvatorePreviti]

... still waiting :(

[danielpza]

yep, still waiting ..., but waiting for what, is something been done ?

[Kirill89]

Is there any better solution than ttypescript now? Are TypeScript developers eventually planing to support plugins?

[tanguyantoine]

You can create you own compiler ;-)

[inad9300]

Any news on the topic? It seems to me that the community has made use of this feature extensively enough to sit, collect feedback, and finally add support for transformers to the config file. There are even plenty of implementations to use as reference.

[samhuk]

It has been over 2 years (wow!)*. Are people open to a rethink of the pros and cons to opening up plugins for transformers?

I feel like the initial concerns that existed all that time ago, such as a lack of documentation about the API, and having to support the meshing of usercode <-> transformers <-> API, are not as relevant anymore. Open plugin support has worked wonders for packages like babel: progression through collaboration. I can personally attest to Babel specifically, having used it extensively.

Easier "just works" integration of transformers like typescript-is would be nice. This package in particular I see as having quiet the revolutionary influence on typescript as a whole. True strong contracts! 😃

I think a conversation about the pros and cons would help dispel any dejection or differences on this. I get a feeling that this zombie-issue will keep on going until such happens - to be honest.

* And in the meantime someone creating the feature for TS instead (ttypescript). I've used this to success.

[Janpot]

@DanielRosenwasser I saw that there was "Investigate TypeScript plugin APIs" on the 3.5 iteration plan (and that work has already started on it). Does that point handle about what's discussed in this issue? The lack of a link to an issue makes me assume it's too much WIP to share anything about it yet?

[safizn]

I think work should be put on adding full support for Babel parser, that includes adding necessary features for Babel to support Typescript static analysis (e.g. multiple file transforms or dependency on multiple files during transformation). Rather than having to duplicate syntax/transform Babel plugins to Typescript equivalents.

[zpdDG4gta8XKpMCd]

in case you have any trouble with the following instruction, let me know, i will be glad to help

given the sheer negligence of the design team, here is a workaround

we are going to take advantage of tslint and its code fixing capabilities

in the following example we are going to do a type-to-code transformation, in its most basic form

we are going to use decorators as markers of places in code that need to be rewritten

in our case, a decorator is a fake function whose only goal is

• to mark a place of the transformation
• and to capture the type that is going to be the source of information for the generator

for the demonstration purposes, we are going to dump the names and types of properties of an interface as plain strings, into an underlying class

the following steps are for windows users only, if you are not one of them, may god help you:

1. start your VSCode

2. install the following extension: https://github.com/Microsoft/vscode-typescript-tslint-plugin

3. close your VSCode

4. go to a folder of your choice

5. run git clone https://github.com/zpdDG4gta8XKpMCd/code-gen.git

6. go to the folder code-gen: cd ./code-gen

7. run

   • 1-install.bat
   • 2-build.bat
   • 3-install-some-more.bat
   • 4-try.bat

8. observe an instance of VSCode opened

9. go to test.ts (Ctrl + P -> test.ts)

10. take a note of the following interface, it's going to be the source for the code generator

    interface Data {
    one: string;
    another: number;
    }

11. take a note of the following function, which comprises a phantom decorator which we are going to use as a marker

    function gen<_T>() {
    return function (meh: any) {};
    }

12. take a note of the site of rewriting, where we want to shove some auto-generate code based on the interface and the marker decorator

    @gen<Data>()
    export class Gen {
    }

13. observe a squiggly line under @gen<Data>()

14. pick Quick fix... -> Needs a rewrite, wanna fix?

15. observe the auto-generated code:

---

for the reference here is the source code of the generator that you can find in codeGenRule.ts

import * as Lint from 'tslint';
import * as ts from 'typescript';

export class Rule extends Lint.Rules.TypedRule {
    public applyWithProgram(sourceFile: ts.SourceFile, program: ts.Program): Lint.RuleFailure[] {
        return this.applyWithWalker(new Walker(sourceFile, this.getOptions(), program.getTypeChecker()));
    }
}

class Walker extends Lint.RuleWalker {
    constructor(sourceFile: ts.SourceFile, options: Lint.IOptions, private checker: ts.TypeChecker) {
        super(sourceFile, options);
    }
    public visitNode(node: ts.Node) {
        if (ts.isDecorator(node)) {
            const checked = check(node, this.checker);
            if (checked !== undefined) {
                const [node, message, replacement] = checked;
                this.addFailureAtNode(node, message, replacement);
            }
        }
        super.visitNode(node);
    }
}

function check(node: ts.Decorator, checker: ts.TypeChecker) {
    const { expression, parent } = node;
    if (!ts.isClassDeclaration(parent)) return;
    if (!ts.isCallExpression(expression)) return;
    const { expression: identifier, typeArguments: typeArgs } = expression;
    if (!ts.isIdentifier(identifier)) return;
    const { text: name } = identifier;
    if (name !== 'gen') return;
    if (typeArgs === undefined) return;
    if (typeArgs.length > 1) return;
    if (typeArgs.length < 1) return;
    const [only] = typeArgs;
    const type = checker.getTypeFromTypeNode(only);

    // if you got to here, you are at the right place and you have a type

    // working on a fix
    const properties = checker.getPropertiesOfType(type);
    const allNameTypes = properties.map(p => {
        const { name } = p
        const type = checker.getTypeOfSymbolAtLocation(p, node);
        return name + ': \'' + checker.typeToString(type) + '\';'
    })
    const { newLine } = ts.sys;
    const body = newLine + allNameTypes.join(newLine);
    const { pos: start, end } = parent.members;
    return [
        node,
        'Needs a rewrite, wanna fix?',
        Lint.Replacement.replaceFromTo(start, end, body)
    ] as const;
}

[mrmckeb]

@zpdDG4gta8XKpMCd, thanks for sharing your concept - but please be civil.

given the sheer negligence of the design team, here is a workaround

This is not the appropriate way to address the TypeScript for not implementing a feature that you (and many others, including myself) would like to have. This isn't a major defect, and is not reflective of "negligence". Please be respectful.

[zpdDG4gta8XKpMCd]

oh please do not take it personal, this is just the way i always start my complaints, i have a very sour personality (medical condition) and this is the best i can squeeze out of myself, i am very sorry

[zpdDG4gta8XKpMCd]

you people are not going to stop surprising me

• what you see is a piece of code that seems capable of solving a major today's problem with comfort (no tricks, no hacking)
• this workaround is a wake up call to the design team, and as we saw it happening before when the community was about to take a turn in a wrong direction, the design team was there to promptly addressed it: #4212,
• so if they care, they can do it right rather sooner until it's too late, or if they don't, we are free to dig in this direction and it won't be a problem for them down the road
• so the tone of my feedback was something inappropriate, but all the appropriate tone of yours has got you nothing since March 2, 2017 (over 2 years), and yes, you can wait another 2 years

[jhpratt]

@zpdDG4gta8XKpMCd What have you contributed towards an actual implementation? TypeScript is open source; I have no doubt they would seriously consider a pull request if you created one (even partially).

Blaming your rudeness on a medical condition is unacceptable; they are words that you typed and voluntarily chose to click "comment". No medical condition makes you do that. Speak, maybe, not type. You also have had the chance to edit your comment to remove it, yet you have not done so.

If you need custom transformers now, you can use Babel. It can remove type annotations from almost all TypeScript, while allowing syntax transformations as readily as you please. As a bonus, it's also faster.

I suggest you stop commenting before you get kicked from Microsoft's repository permanently. I know if it was my repo, you'd be on your last straw.

[zpdDG4gta8XKpMCd]

ok, you kicked me out, then what can i ever come back? can i create a new account? do you think i care about this account too much?

anyway, yes, i tried pull requests, unfortunately it's not feasible for the following reasons:

1. there is a certain category of problems you are welcome to solve, that are rather trivial and of less interest, anything serious like this - and your are not

2. since you cannot solve it on your own, a problem like this (even having 224 thumbs up) can wait for years if ever considered at all

3. luckily you can do something today using whatever means you have and start making a difference that can be seen, hence the suggestion (don't blame me for doing nothing)

[DanielRosenwasser]

@zpdDG4gta8XKpMCd as much as you know I've been a fan of the cynical takes over the years, I've gotta agree with others here - we need to keep the conversation civil and respectful. It's always good reminder that everyone tracking this issue wants to make the project better.

---

@Janpot Yup, it's very much a WIP, though when @rbuckton gets back, he may be able to give some more details on status.

As we investigate plugin points (like I alluded to in the 3.5 iteration plan), I think exposing easier hooks for custom transformers is a major scenario we want to consider. The reason we wouldn't necessarily just do this outright is that we might have broader work that subsumes this. In other words, maybe there's something broader than just pre- and post- transformers.

[zpdDG4gta8XKpMCd]

@DanielRosenwasser there is not that much cynical in it, i only said 2 words and then i said i am deeply sorry about that

anyway, point is i think we all agree that there is a good part of healthy frustration over how things go, and we do not blame it on you, we understand what drives this project

but if we look at it from our angle there must be somebody to blame, because letting issues like this sit over years just collecting thumbs up, is what stops from being more productive

i am sure there are reasons for putting it off and focusing on some other things first, so i think it would be better if you let your audience know why certain highly wanted features cannot be fulfilled, it could lower the degree of frustration, say:

• we cannot implement this feature because of A, B C and those need D, E, and F done first

so i guess i am referring to this conversation: https://github.com/Microsoft/TypeScript/issues/30696#issuecomment-478799258

[RyanCavanaugh]

Having thumbs ups doesn't create new resources out of thin air, make complex and difficult decisions simple and easy, change existing business priorities, stop in-flight projects, deprioritize other work, cause overly-ambitious proposals to become well-scoped, or alter a suggestion with wide-reaching and permanent impacts to become focused and maintainable.

I have no apologies for us doing (or allowing PRs for, which is what you were complaining about in #30696) upfront well-understood good and simple work ahead of extremely complex and high-maintenance work like this. It should not be hard to guess why we're willing to let someone come in and fix a leaky faucet while the plans to remodel the basement and add another story on top of the house are still being worked out.