Add support to transpile modules inside node_modules

[arunoda]

Now some of us ships NPM packages (specially components) written in ES2015 without transpiling them.

That's a pretty good thing specially if they are gonna used in a project like Next.js or CRA (which does transpiling). They offer benefits like:

• No need to transpile before shipping to NPM
• Get the benefit of Webpack 2's tree-shaking

But we can't do this now we exclude everything inside node_modules from babel transpiling.

So, here's the proposed solution.

We have an entry in next.config.js to include modules which needs to go through babel. See:

module.exports = {
   transpileModules: [
     "my-component",
     "redux/src"
   ]
}

[rauchg]

• Why is this not doable with webpack() extension?
• transpileModules sounds better
• Will we accept regexps?

[arunoda]

Why is this not doable with webpack() extension?

I hope you mean custom webpack config. That's doable. But this sits in a existing loader. Getting that is somewhat harder.

transpileModules sounds better

Awesome. I'll update.

Will we accept regexps? Yep. sure.

[slorber]

Hey,

As I'm starting my own website I'm trying new techs like Lerna/Next/Styled... and would be happy to provide early feedback on this.

I've opened a duplicate issue where I tried to import/transpile a CRA-based module in my NextJs module but didn't know how to make the transpilation happen (note that I'd like to keep my module runnable as a standalone)

I've also noticed that Babel, also based on Lerna, is transpiling each modules before exposing them, but it looks to me better to do like @arunoda suggest and let the client app do the transpiling.

I'd like to have a single babel config for my client and share that config with all my decoupled modules. That's probably not so easy if I want to keep ability to run my modules as standalone, outside of Next runner

My current test project is here: https://github.com/slorber/playground/ I'll try to upgrade it as soon as there's a fork/PR. @arunoda are you working on it?

[arunoda]

@slorber currently we are focusing on 2.0 release and we are fine tuning stuff and finding bugs as possible as we can.

I haven't started work on this but we can do this just after 2.0.

[slorber]

Ok so I'll make a fork. I'm already running against 2.0.0 beta because I'm not building a critical website and I don't think webpack 1.13 resolve jsnext:main/module field.

I'm not a bundler expert but I think I'd rather use "module" field of package.json no? "main" seems for already transpiled code as far as I know. But as the webpack config allows to include/exclude transpilation I'm not sure it's relevant. Any recommendation on which of the 3 fields I'd rather use?

[arunoda]

@slorber I think webpack only supports main just like NPM. You can use that. We can check for the filepath in the exclude function in our next.conf.js

[slorber]

Hmm according to what I've seen in practice against Next 2.x I've seen module works (but fails later at runtime because not transpiled) while jsnext:main did not work (as far as I remember). But it's supposed to be supported.

Anyway, jsnext:main or module does not seem to be the solution to this problem so for company-internal modules just enabling transpilation is probably enough

[rauchg]

The community has not agreed on one approach right? For example, I was able to use react-youtube the other out of the box with no problems. I'm assuming a big number of modules transpile before publish?

Ref: https://github.com/rauchg/blog/blob/master/components/post/youtube.js

[slorber]

Yes that makes sense to always transpile before publish because you don't know who/how the module will be consumed and you don't want to force the client to setup appropriate babel settings for your lib. That's what Rollup suggest: to publish the module transpiled in different ways so that bundler can decide which to use.

But for company internal packages, the transpilation settings might be the same across several project (like a babel preset) and it makes sense to me to let the client bundler to transpile all the company dependencies

[philcockfield]

Very much agree @slorber - this would be very handy for internal modules if you're breaking your project up and isolating things as much as possible.

And @rauchg / @arunoda supporting RegExp's would be really nice, so you could have one entry that catches all the company internal modules, using say the NPM org namespace:

// next.config.js
module.exports = {
  transpileModules: [
    /^\@my-npm-org\/.*/
  ]
}

[rauchg]

Beautiful suggestion @philcockfield

[slorber]

Hey maybe it could be worth offering some presets. It looks to me most tools (Lerna/npm link...) rely on symlinks so why not something as simple as:

module.exports = {
  transpileModules: ["symlinks"]
}

[philcockfield]

The more I use next.js in earnest, and build out a rich library of modules around it, the more this feature becomes important. It's becoming a real PITA replicating the babel compilation step in my internal modules.

🚀🤖

[arunoda]

I'm working on this today :)

[arunoda]

@philcockfield give this a try: https://github.com/zeit/next.js/pull/749

[slorber]

thanks @arunoda

So as commented on your PR if this does not support symlinks the feature will be a bit limited because it won't work with npm link or Lerna, but only for npm modules that are not transpiled (right? I don't see any other usecase unless you commit modules inside /node_modules)

Why not supporting symlinks? is it harder to support?

Also I wanted to test your branch on my app, but I'm not sure what's the best way to do that. Is there any known procedure so we can easily test a branch and it's not too painful for the tester? I've tried some stuff like:

• npm install https://github.com/arunoda/next.js.git#add-706 : fails because the next /bin folder is empty on github repo
• git clone of the fork inside /node_modules: not much success (but it might be because of my specific Lerna settings)

What's the best way to test a fork currently?

[lorensr]

If you're looking at doing this with next.config.js: module.exports = { webpack: (config, then config.module.rules has a few things, looks like you need to change one of these rules, or add one?:

  { loader: 'babel-loader',
    include: '/Users/me/gh/guide/node_modules/next/dist/pages',
    options: 
     { babelrc: false,
       cacheDirectory: true,
       sourceMaps: 'both',
       plugins: [Object] } },
  { test: /\.js(\?[^?]*)?$/,
    loader: 'babel-loader',
    include: 
     [ '/Users/me/gh/guide',
       '/Users/me/gh/guide/node_modules/next/dist/pages' ],
    exclude: [Function: exclude],
    query: 
     { babelrc: true,
       cacheDirectory: true,
       sourceMaps: 'both',
       presets: [] } } ]

Looking forward to the simpler syntax suggested.

[andrewmclagan]

Sorry for my ignorance I cant see what the resolution of this issue is? We would love to be importing es6 into our codebase, we need the tree-shaking .

Is there a PR on this?

[slorber]

@andrewmclagan This issue is still open and has a related PR that probably won't satisfy all (like LernaJS users)

[Yuripetusko]

What's the status of this? Are there any other ways to make next's webpack to transpile files imported from node_modules ?

[andrewmclagan]

@slorber i will take a look at the PR. Contribute our use-case.

[muhajirdev]

I am facing kind of similiar problem. Trying to use get-urls package. Works find with dev but when i compile it. I got error from uglify

...
{ Error: commons.js from UglifyJs
...

Is there any workaround for this please?

[timneutkens]

Arunoda will work on it sometime here. He has done before in #749

[eliot-akira]

I'd love to see this feature implemented. As @philcockfield mentioned, it's a common scenario to build up a library of modules that depend on Next.js transpilation, and it'd be great to be able to share components among projects.

[ernestofreyreg]

This is not only needed for tree shaking. Also for babel plugins like styled-jsx. So if you have a module (like a library) that uses a babel plugin the best solution is to include the ES6 source code and allow your app to transpile it from node_modules. Of course next already includes styled-jsx by default.

[thealjey]

Here's what I did

// next.config.js
exports.webpack = config => (config.module.rules = config.module.rules.map(({exclude, ...rest}) => ({
  exclude: Object.prototype.toString.call(exclude) === '[object Function]' ? (str => !/mycomponents/.test(str) && exclude(str)) : exclude,
  ...rest
})), config);

I basically replaced each exclude with a custom function. I do not know what I am doing wrong, but I just cannot make it work. I need the contents of node_modules/mycomponents to also be transpiled by Next.js

It doesn't even work if I completely override all excludes with an empty array

exports.webpack = config => (config.module.rules = config.module.rules.map(({exclude, ...rest}) => ({
  exclude: [],
  ...rest
})), config);

Please help me :) Thanks

[damianobarbati]

Hey guys ( @thealjey ) I've been transpiling jsnext:main powered modules for months by now.

• use the following resolver: https://github.com/damianobarbati/react-app/blob/master/config/webpack.config.js#L66 https://github.com/damianobarbati/react-app/blob/master/config/webpack.config.js#L203 Basically it says: if jsnext:main is in package.json then transpile the specified entrypoint, otherwise import what's in main
• the following will help when using npm-linked modules: https://github.com/damianobarbati/react-app/blob/master/config/webpack.config.js#L168

I'm not using next.js but I hope it helps.

[thealjey]

@damianobarbati no, unfortunately it does not Configuring webpack directly to transpile whatever is not difficult, but I am struggling to make this work in the context of Next.js

[revolunet]

hey all, anyone have found a solution?

i have a local ES6 linked node module i need to import in my project but i cant get the webpack voodoo right !

[gwintrob]

I'm sure there's a better way, but we run everything through babel on build:

next build && babel .next/*.js --out-dir . --presets=es2015,react

[mattfelten]

Did this die? I'm looking for a way to transpile a custom module and seems like it's still not possible.

[timneutkens]

@mattfelten it's on the roadmap for v5 👍

[makis-spy]

Does anyone have an example of a workaround for this?

[chrisui]

@timneutkens Is there any timeline for this? Appreciate that's often an impossible question but we're trying to determine our stack at work as of now and this is a pretty big blocker for us! :)

Workaround suggestions also valid.

[chrisui]

@thealjey realise this is an old comment but your solution probably didn't work because there is an include specified as well which would need to be overridden.

Update: looked into this strategy but it's just not sane given all the different loaders for different module directories within next.js' internal configuration. This will need to be first-class.

[thealjey]

@chrisui my (temporary) solution was to use babel-plugin-module-resolver, configured like this - "plugins": [["module-resolver", {"root": ["./"]}]] it is by no means a perfect solution, but, since everything else failed, it works for now thanks to that I do not need to write a bunch of ../ with every import this might actually be a better solution to some, though it certainly doesn't help with reusability

[jamesgorrie]

@thealjey could you give an example? I have a project setup where I try this to no avail... https://github.com/jamesgorrie/nappy

I'd happily try get a PR in as this would make our life so much easier, but there are a few questions like: Should next.js support transpiling of certain modules or should this be up to the transpiler, but the next.js follows the module resolution more strictly. Not sure who to ask or where to start here as new to next.js.

[oliverjam]

it's on the roadmap for v5 👍

@timneutkens did this make it into v5?

[blackbing]

wondering to know.

[andrewmclagan]

Just look at where its merged. this was merged 8 days ago. 5.0.0 was released 2 days ago. from the canary branch where this is merged into...

[brianyingling]

Does anyone have an example on how this can be implemented? Is it supposed to work like this, as mentioned above?

module.exports = {
    transpileModules: ['my-npm-module']
}

or does it look different?

[timneutkens]

https://github.com/zeit/next.js/pull/3732

[brianyingling]

I'm confused. @timneutkens The abovementioned link is to a PR that is still open. Has this NOT been merged in v5 yet?

EDIT: Never mind, this is just a PR to highlight the example.

[timneutkens]

It's an example, you can use the example without it being merged. transpileModules is something we'll tackle later on.

As a general rule of thumb: When an issue is open it's not released.

[jamesgorrie]

@brianyingling I've turned the example into a plugin for an interim solution (v5 only). It's working for us at the moment, until the more robust, official solution is put in place.

https://www.npmjs.com/package/@weco/next-plugin-transpile-modules

[statico]

I would love to see an example of this for a single module in the application codebase.

For example, I have a module that I use both server and client side. I haven't been able to get any of the above examples to work, either by factoring the module into its own separate module and yarn linking it, or by hacking the webpack rules.

Here's a simple repro: https://github.com/statico/nextjs-with-async-lib

// pages/index.js
import { foo } from '../lib/test'
export default () => <div>hello {String(typeof foo)}</div>

// lib/test.js
async function foo () {}
module.exports = { foo }

@timneutkens tells me this is because I'm using async/await in the shared module. I guess my solution might be to remove async/await and change everything to .then()-style callbacks.

[statico]

OK, I found a fix that works for me.

First, I added the config.resolve.symlinks = false setting to my next.config.js configuration per https://github.com/zeit/next.js/issues/3018#issuecomment-380879576

// next.config.js
webpack: (config, { dev }) => {
  config.resolve.symlinks = false
  return config
}

Then I put my shared library — a .js file that uses CommonJS exports and async/await keywords — in a subdir of the application called shared:

// shared/index.js
async function foo () {}
module.exports = { foo }

// shared/package.json
{
  "name": "@myapp/shared",
  "version": "1.0.0",
  "main": "index.js",
  "license": "UNLICENSED",
  "private": true,
  "dependencies": { ... }
}

And finally I added a postinstall script to link it all together when anyone does a yarn install in the main application:

// package.json
{
  ...
  "scripts": {
    "postinstall": "cd shared ; yarn -s unlink ; yarn link && yarn && cd .. && yarn link @myapp/shared",
   ...

Now my Mocha tests pass on the server side, my custom Koa server starts up fine, and there's no more crazy Cannot assign to read only property 'exports' of object '#<Object>' in my NextJS pages.

[stugoo]

I had this same issue when upgrading to NextJs 5.1.0. One or two of the node modules in next were not transpiling fat arrow functions and throwing errors in IE11. I had previously been through setting up individual polyfills and in the end I opted to target those modules files with babel-polyfill in my next.config.js with this:

module.exports = {
  webpack: (config, { dev }) => {
    const polyfill = new Promise((resolve, reject) => {
      const originalEntry = config.entry

      originalEntry().then(entries => {
        if (entries['main.js']) {
          entries['main.js'].unshift('./client/polyfills.js')
          entries['main.js'].unshift('babel-polyfill')
        }
        config.entry = entries

        resolve()
      })
    })

    config.module.rules.push(
     {
        test: path.resolve('./node_modules/next/node_modules/'),
        loader: 'babel-loader',
        options: {
          babelrc: false,
          cacheDirectory: false,
          presets: ['es2015']  
        }
      }
    )

    return polyfill.then(() => { return config })
  }
}

Hope this helps someone.

[curran]

ESM works like a charm.

With a custom Next.js server in index.js, I can run this command to start the server and esm kicks in perfectly, resolving ES modules in Lerna-symlinked project packages.

node -r esm index.js

[blackbing]

@curran cool, can it replace babel-node ?