Plugin system

[Michael-F-Bryan]

I'd be keen to help out with this! We can probably reuse a lot of the logic from the Dynamic Loading and Plugins chapter from my FFI guide.

The main thing we'd need is a stable-ish interface for doing transformations (i.e. the Fold<T> trait) and then figure out a set of conventions for how users register their plugin(s).

[kdy1]

Note: All core transformations ported from babel and closure compiler will be statically compiled.

[kdy1]

@Michael-F-Bryan I'm sorry about my mistake. I meant ecmascript module by 'Module'. Module loader meant bundler like browserify

I created another issue instead of modifying this issue as plugin system is required anyway.

[stefanpenner]

I have been doing some exploration re: work to get swc powering ember applications. So far the performance seems really excellent, which motivates me to open discussion in-regards to the remaining non-bug blockers.

Currently, we rely heavily on babel plugins. Some of these plugins may make good core functionality, others maybe should remain as part of a plugin ecosystem.

The core babel plugins are:

• babel-plugin-ember-modules-api-polyfill rewrite imports to globals <-- candidate for core swc feature
• babel-plugin-debug-macros a macro system for build time code stripping <-- candidate for core swc feature
• ember-compatibility-helpers a plugin to aide transitioning consumers from 1 system to another <-- may be better as a plugin
• babel-plugin-htmlbars-inline-precompile a plugin which rewrites tagged template literals containing HBS templates into their compiled output <-- may be better as a plugin

My tracking issue: https://github.com/stefanpenner/broccoli-swc/issues/10

[kdy1]

@stefanpenner I have some questions

• babel-plugin-debug-macros

swc currently has inline_globals pass and simplifier pass which performs dead code elimination.

if (__DEBUG__) {
    console.log('Foo')
}

Isn't this enough?

(See https://github.com/swc-project/node-swc/blob/06845c25c64ccfc626cfdac53987f17b94ef95d8/__tests__/optimizer_test.js#L4-L22 for usage)

[stefanpenner]

@kdy1 it is very close, the exception is our API accomplishes this not via globals but rather via module imports.

so where you currently have __DEBUG__ global, we also have:

import { DEBUG } from '@ember/env-flags';

and

import { FEATURE_A, FEATURE_B } from '@ember/features';

Honestly, it seems like inline_globals having inline_modules as a sibling feature would be generic enough to accomplish this.

[kdy1]

Neon does not support invoking javascript function from worker thread. It seems like node's limitation as node requires all javascript code to run in event loop thread. So we don't have many option.

Options are

• moving transformAsync to event loop thread This will reduce performance gap between swc and babel. Even though, it's also an option because swc is 16x - 20x faster than babel even with transfomSync, which works on event loop.

• implementing plugin layer in javascript It can be done by

  • making transform* to return ast instead of stringified code (with hidden emit: 'ast' flag)
  • handling plugins from javascript
  • exporting parse / toCode method

I'll try second option.

@stefanpenner Okay, I got it and I'll implement it as a core module.

[stefanpenner]

@kdy1 honestly for us, we absolutely wouldn't mind authoring/re-implementing our plugins in rust. (In-fact this is what I assumed we would need to do).

The performance difference is something we wouldn't want to compromise on at all.

My expectation was that swc would provide a mechanism to providing plugins in rust, and we would author to that API.

[kdy1]

@stefanpenner Oh.. I'm bit surprised by it. But either option do not affect performance gap caused by core transforms. I will do some investigation and add some benchmarks.

[stefanpenner]

Keep me posted!

But yes, absolutely down to author rust plugins. All we need is a good plugin API. Especially if this allows concurrency provided by transform over transformSync.

[kdy1]

@stefanpenner javascript plugin cannot run in worker threads. It runs only on event loop thread. To make javascript code work concurrently, we should spawn child process. (Parcel do this).

[stefanpenner]

@kdy1 ya, that was my assumption.

[stefanpenner]

@kdy1 is there a way to load rust plugins? That way we get the best of both worlds.

[stefanpenner]

To make javascript code work concurrently, we should spawn child process.

Ya, we currently do this via node-workerpool for our babel builds.

[kdy1]

@kdy1 is there a way to load rust plugins? That way we get the best of both worlds. @stefanpenner Yes, but I'm not sure about how rust plugins would be published and loaded

[stefanpenner]

Fair fair. I’ll do some inquiry this week myself :)

[stefanpenner]

@kdy1 I’m curious what issues the above https://github.com/swc-project/swc/issues/18#issuecomment-435336787 idea poses?

[kdy1]

@kdy1 I’m curious what issues the above #18 (comment) idea poses?

It was about plugin system based on dynamically loaded library (.dll on windows, .so on linux). I didn't merged it because such plugin system would make deploying plugin too hard.

[kdy1]

Plugin system is almost ready.

TODOs

• [x] parse(code)

• [x] print(module)

• [x] Visitor (for javascript) (See https://github.com/swc-project/plugin-utils)

• [ ] hook in transform() / transformAsync()

Sidenotes:

• AST types are not compatible with babel's one. Even though, visitor based plugin will be easy to port for swc.

[stefanpenner]

@kdy1 very cool. I'll give it a whirl once the the last step is out (even if its in PR form).

I'll be curious how performance characteristics are. I have some PTSD from another project which mixes js -> native -> js -> native -> js ;P But i'll keep an open mind

[lifeart]

Just in case - nginx has very fast subset of js

• https://github.com/nginx/njs
• http://nginx.org/en/docs/njs/
• http://nginx.org/en/docs/njs/compatibility.html
• https://youtu.be/Jc_L6UffFOs

[izelnakri]

transformAsync() JS plugin hook sounds very exciting, despite that being in JS, it would still be a very useful first step.

Later for rust plugins we can probably define the rust swc plugins in package.json and make @swc/cli swc compile command to read them and initiate several steps to create a new neon module with @swc/core + rust plugins in it compiled and exposed, it could also be a local npm linked module if needed. Probably hacky but shouldnt be impossible :)

just subscribed to this thread. @kdy1 please let us know when there is a PR.

[hh9527]

There will be many choices to make a plugin system. A. dll/so based B. crate based: swc as a dependency, leave the final combined binary to the other project C. crate based: all plugins are a dependency crates, and build into one single binary (swc) together by feature gates D. swc binary + ipc based plugin protocol E. swc binary + js based plugin interface F. swc as a wasm module (loaded by nodejs) + js based plugin interface

I prefer C (B is a pre step of C). F is also interesting.

[kdy1]

@hh9527 I almost implemented E, but I think I should reimplement a javascript code generator in javascript because overhead of serialization / deserialization makes swc too slow.

[hh9527]

@kdy1 , I do think the performance is most important, if we can not beat babel family in performance, very few people will use swc.

In the old days, plugin system is very important to babel, because ECMAScript was evolving very fast. people always need to use plugin-transform-xxx to use latest syntax. But these has been changed nowadays. In most cases, for a web/js developer, these are enough:

• preset-env
• preset-typescript
• preset-react
• css module (id/classname mapping)
• a bundler

So, maybe it is not necessary to make a full dynamic plugin system to external developer. A crate based static plugin system is enough. That is why I prefer C. And if some user do have a requirement to make a customized extension, he can build his own customized swc by B (combine plugins and swc-core with his own plugin crate)

[kdy1]

@hh9527

A: seems quite hard, Deployment process would be incredibly complex. B: Has same issue as B. Deploying binary based stuff is too hard. C: I also prefer C, but it requires user to write plugin in rust and would make .swcrc very complex. D: Too hard, and serialization is too expensive. E. swc binary + js based plugin interface F: Serialization is too slow.

So I'm currently reimplementing code generator in javascript. (to implement E)

[kdy1]

There's a new issue for plugin system

Please see: https://github.com/swc-project/swc/issues/470#issuecomment-559329450

[kdy1]

Closing in favor of #471.

[kdy1]

@hh9527 I'll go with js-based plugin and ipc-based plugin.

[kdy1]

Javascript based plugin is almost done, and there's a basic example of using plugin. See https://github.com/swc-project/node-swc/blob/d51369e43e590fa66ddc2db6ca39976cdd185ac9/__tests__/transform/plugin.js

cc @Michael-F-Bryan, @stefanpenner, @lifeart, @izelnakri, @hh9527

[stefanpenner]

@kdy1 very cool, any early benchmarks / perf intuitions?

[kdy1]

@stefanpenner https://github.com/swc-project/swc/pull/475#issue-347195806

[plugin]
  parse x 3,156 ops/sec ±0.43% (89 runs sampled)
  parse + print x 740 ops/sec ±0.35% (88 runs sampled)
  parse + transform x 734 ops/sec ±0.22% (88 runs sampled)
  plugin x 720 ops/sec ±0.32% (87 runs sampled)
[transform]
  swc (es3) x 761 ops/sec ±0.23% (89 runs sampled)
  swc (es2015) x 800 ops/sec ±1.02% (87 runs sampled)
  swc (es2016) x 2,123 ops/sec ±0.84% (88 runs sampled)
  swc (es2017) x 2,131 ops/sec ±1.13% (90 runs sampled)
  swc (es2018) x 2,981 ops/sec ±0.25% (90 runs sampled)
  swc-optimize (es3) x 712 ops/sec ±0.21% (86 runs sampled)
Browserslist: caniuse-lite is outdated. Please run next command `npm update caniuse-lite browserslist`
  babel (es5) x 41.75 ops/sec ±8.07% (56 runs sampled)
[typescript]
  swc (es3) x 646 ops/sec ±2.25% (87 runs sampled)
  swc (es5) x 703 ops/sec ±0.55% (89 runs sampled)
  swc (es2015) x 708 ops/sec ±0.26% (87 runs sampled)
  swc (es2016) x 1,656 ops/sec ±0.17% (90 runs sampled)
  swc (es2017) x 1,661 ops/sec ±0.33% (91 runs sampled)
  swc (es2018) x 2,135 ops/sec ±0.25% (88 runs sampled)
  swc-optimize (es3) x 631 ops/sec ±0.13% (88 runs sampled)
  babel (es5) x 41.89 ops/sec ±4.62% (54 runs sampled)

[hh9527]

@hh9527 I'll go with js-based plugin and ipc-based plugin.

congratulations! thank you!

[swc-bot]

This closed issue has been automatically locked because it had no new activity for a month. If you are running into a similar issue, please create a new issue with the steps to reproduce. Thank you.