Discuss .babelrc detection, engines field, etc.

devongovett commented 6 years ago

Based on twitter convo:

https://twitter.com/dan_abramov/status/938090483166924800

I’d suggest to reconsider this (or at least limit it to compiling with -env preset only). This encourages authors to publish packages with stage 0 proposals, and ultimately makes builds slower for everyone. Also .babelrc won’t be compatible between Babel versions.

mswiecicki commented 6 years ago

As i understand the fundamental issue here is to detect how much transpilation (if any) is needed for each third party dependency, correct?

Stop me if it's stupid but couldn't we somehow check package.jsons of dependencies? I mean is there really any other way to programatically check what is needed by given project? Though i imagine it could be potentially costly... :/

benjamn commented 6 years ago

Whether or not to honor .babelrc files in nested node_modules directories is a hot topic over at the Babel repo, too: https://github.com/babel/babel/issues/6766

In short, most npm packages are fully compiled before they are published, so it is wasteful/inappropriate to recompile them (even using their own .babelrc files). It would be great if those packages put .babelrc in their .npmignore files, but that doesn't always happen.

For the minority of npm packages that do need to be recompiled for the browser, the application developer needs to be able to intervene somehow, to enable recompilation selectively. I can imagine various ways of configuring that behavior, but it may be hard/impossible to make the right choices without increasing the configuration burden somewhat.

Taking a step back: Parcel is competing with Webpack, which is known for using configuration to solve any and all problems. While "zero-configuration" may be more of a goal than a reality (you still have to configure packagers, e.g.), I think it's a really valuable goal, so it's worth thinking hard about how to solve this problem with minimal additional configuration.

mswiecicki commented 6 years ago

@benjamn So do I understand correctly that the issue lies in whether it is necessary to recompile given dependency and if so using which .babelrc? Because if that's the case maybe we could choose the default behaviour, and allow the other via some CLI flag?

It does seem like a band-aid of some kind though...

jamiebuilds commented 6 years ago

I'm generally against doing any kind of transforms on node_modules.

Basing things off of package.json#engines#node and only using babel-preset-env would be an improvement, but we'd still have the problem of unsupported APIs in older versions of Node.

I suppose if someone shipped:

{
  "engines": {
    "ecmascript": "2017"
  }
}

We could deal with that a bit easier since we can polyfill and transform all the relevant bits.

I'm still not convinced that's a good idea.

davidnagli commented 6 years ago

Does Babel currently transpile node_modules?

jamiebuilds commented 6 years ago

@davidnagli Babel itself defaults to ignoring node_modules. Parcel (currently) undoes that behavior and will run it on node_modules if it finds a .babelrc

chipit24 commented 6 years ago

This is currently causing some issues for me. All the 3rd party modules I import (from node_modules) for my application are already compiled/transformed; I don't want Parcel to run any transofrms even if it finds a .babelrc file there.

davidnagli commented 6 years ago

@chipit24 I guess we’re gonna have to work on resolving this issue. In the meantime, if you’re looking for a workaround use the --no-minify flag

tjenkinson commented 6 years ago

I've always understood that the 'main' property in package.json should point to an es5 build, so transpilation should not be necessary. Is this not the correct assumption?

chipit24 commented 6 years ago

There's no real requirement for what the main field should point to. There's some interesting info in the rollup.js docs:

Yes. If you're using things like arrow functions and classes (or more exotic features like decorators and object spread), you should transpile them if you want your library to work in environments that don't support those features. Otherwise, whoever uses your library will have to transpile it themselves, including configuring their transpiler to handle your code, which might involve esoteric transforms and conflicting versions of Babel.

It's a frequent source of frustration. Be a responsible library author and ship code that actually runs in the environments you support!

tjenkinson commented 6 years ago

I see. My opinion is that parcel should see packages as read only and not run any transformations on content in them unless this is explicit in some way.

govizlora commented 6 years ago

@davidnagli Could you specify how to use the --no-minify flag? Maybe I'm missing something but I just want to find a workaround for babel to ignore node_modules. Thanks a lot!

rj254 commented 6 years ago

Also, a common problem is that people accidentally publish their node_modules with .babelrc even if it is already transpiled.

Additionally, let us say we have package X which uses babel-preset-stage-1, which includes a .babelrc for some reason. Our project includes package X, but uses babel-preset-env. The current behavior of attempting to transpile package X again results in a Couldn't find preset stage-1 relative to directory "PATH_TO_X". Because in most cases, package X will install babel-preset-stage-1 as a devDependency, it will not be installed when running yarn (probably npm install) on our project.

I think that a flag to disable this behavior would be very helpful (or have this behavior prevented by default and allow it to be toggled on). Also, I'm not quite sure what the --no-minify would solve this issue.

Cinamonas commented 6 years ago

It would be great to have an option to bypass .babelrc detection in node_modules that are being imported into the project.

I've created a minimal example of it breaking the build: https://github.com/Cinamonas/parcel-babelrc-issue

Is there currently any workaround? Because if you do import a transpiled module that also includes .babelrc (probably accidentally), you hit a wall.

codyhatch commented 6 years ago

Any progress on a workaround? This seems like a pretty critical issue which makes parcel more-or-less unusable in many situations.

To be clear, my issue is:

I have a small project that uses a few npm libraries
Some of them have included .babelrc files despite already having properly transpiled output in the npm package, which is unfortunate but common
Parcel is trying to recompile them using Babel plugins which are not installed, and which should not be installed, and which really shouldn't be used even if they were installed.

One (of many!) examples is the react-slick package, which yields an error like this for a project that uses it:

foo/node_modules/react-slick/lib/index.js: Unknown plugin "transform-object-assign" specified in "foo/x/node_modules/react-slick/.babelrc" at 0, attempted to resolve relative to "foo/node_modules/react-slick"

It has a .babelrc file, but it also includes the already compiled output; no transpiling is needed.

shawwn commented 6 years ago

The solution is to submit a PR to those projects asking them to remove their .babelrc file from their npm deployment.

Just kidding.

There are two issues to consider:

When we load config files, we try to load the one closest to a given file. Parcel walks up the filesystem from the source file until locating the config.
This is the correct behavior in all cases except for transpiled code. Ideally, we would detect that we're importing code that has already been run through a compiler, and therefore shouldn't compile it again.

It would be a bad idea to do a simplistic, hacky solution such as "just ignore all node_modules/*/.babelrc files". For example, source code is shipped along with transpiled code, and right now you can just point the package.json to the source folder instead of the transpiled folder and everything will just work. Parcel picks up on their babelrc correctly.

On the other hand, I'm not sure whether there's a reliable way to detect whether a js file has already been transpiled. Does babel insert a magic string into output files we could look for?

This becomes especially tricky in certain other situations: for example, you could imagine wanting to legitimately transpile code twice for various reasons, e.g. chaining plugins together.

The key here is to make things as painless as possible for end users by default, and give them ways to override the defaults tastefully.

I'm not sure what a good solution is yet, but we should think carefully before shipping changes related to this. Parcel is getting picked up across the world now, and our decision here will affect everyone later. The worst situation to end up in is to ship a quick fix and then realize later that it's a bad approach, since users will upgrade their parcel and discover their codebase breaks on the new version. Problems like that can doom traction, and parcel has a lot of momentum now.

A dumb solution would be to just try a stock babel fallback config if we get an unknown plugin error. That should transpile transpiled code just fine. But I'm skeptical that's a good idea.

Another idea is to add a .parcelignore file, so that users can add individual .babelrc paths that should be ignored. But that just punts the problem to users.

Can anyone think of a default behavior that works in all cases but doesn't produce unexpected or hidden side effects?

chipit24 commented 6 years ago

Even if we can determine whether or not the library has been transpiled, it may not be desirable to simply apply the transforms described in said library's babel config. Aside from the clear problem of conflicting babel versions and missing babel plugins, there's other things to consider.

For example, let's say parcel can detect transpiled code and conditionally run transforms if the code isn't transpiled. I'm building a package that pulls in package X which parcel knows has not been transpiled, but the babel config for this package only transpiles code to ES2015 (ES6). I want my code to support the ES5 spec - so this solution won't suffice.

There appears to be a solution for the above issue with webpack, via https://github.com/andersdjohnson/webpack-babel-env-deps (which I discovered through the discussion at https://github.com/babel/babel-loader/issues/171); so perhaps we can take some ideas from there.

gaearon commented 6 years ago

IMO it is fundamentally wrong to try to respect .babelrc in other projects. This means that if I publish a library using Babel 6 but Parcel updates to Babel 7, my library is suddenly not useable anymore. As a publisher I don’t want my consumers to depend on Babel at all, much less to be confined to a single major because I accidentally included a config file in the package.

devongovett commented 6 years ago

My comments from twitter:

I totally agree that there should be a more standard way of publishing non-ES5 code to npm, but I also think it’s important to let modules specify their own custom transforms as well which only apply locally.

We should not distinguish between app code and module code - they should build the exact same way. If you imported another app from node_modules, it should build the same way as if it were an entry point.

We should use the version of Babel specified in the module not the one in the app. That way it will always build as the module intended.

devongovett commented 6 years ago

To elaborate:

We already do not transpile with babel unless a .babelrc file is found. We definitely do not want to run babel over all node_modules.
node_modules should build exactly the same way if they are the entry point or a dependency of another app. e.g. parcel build module.js should produce the same output for module.js as parcel build app.js which depends on node_modules/module/module.js. If module.js needs to compile some things with babel, that needs to be part of the build process.
Transpiling things pre-publish to npm is not a good solution. Ideally, the source code would be transpiled based on a browser support matrix defined in the app, which may not be all the way down to ES5. Also, it is very useful to be able to npm link a module locally and just edit the source files, without needing to have a manual build step. This is only possible if Parcel handles the transpilation for you.
We should not treat JS differently from other languages. If you have a TypeScript file in your node_modules, we transpile it with the locally specified version of the typescript compiler. Same for SASS, CoffeeScript, etc. JS should be no different. If we use the locally specified babel version, this should work as desired.
The main issue here is that modules are published to npm with .babelrc by mistake, but are already pre-transpiled. This could be solved in a number of ways: we could require an additional flag to enable babel, we could detect if the babel is actually specified as a dependency (not devDependency), etc. Not sure any of those are really ideal, but we do need to do something to work around this issue.

I think we should go for a two pronged approach:

We support some way to specify the version of JavaScript that a module is written against. This could be something like the engines field in package.json: "engines": { "ecmaVersion": "ES2015" }. This would enable transpilation. The target would be specified by the app rather than an individual module via a .browserslistrc file or similar.
We support an explicit way to turn on babel for node_modules, to enable custom transforms. Browserify does this via transform field in package.json. Perhaps we can come up with some similar mechanism. This is not ideal as it breaks the "zero-configuration" promise, so if there is a better way to detect when NOT to run babel when there is a .babelrc, I'm all ears.

In any case, we need a lot more community feedback on how do implement this, along with support from other bundlers, before we jump into this. I don't want to come up with something Parcel specific here.

youknowriad commented 6 years ago

In any case, we need a lot more community feedback on how do implement this, along with support from other bundlers, before we jump into this. I don't want to come up with something Parcel specific here.

I understand the reasons behind both positions (transpiling or not) and I like this ⬆️, trying to come with a "standard" across all npm modules.

That said, this will likely take a lot of time to achieve something decent and in the meantime refrain people from using parcel broadly because packages they're using with their current setup (webpack) can't be used with parcel. Do you think coming up with a temporary workaround (a .parcelignore or similar) could be an option?

benedyktdryl commented 6 years ago

@devongovett I got lost in this conversation - sorry if question is unrelated but it does mean that Parcel should respect .babelrc in project root directory or not?

ianstormtaylor commented 6 years ago

@devongovett I think your proposal is missing the fundamental piece of @gaearon's argument, which is that for a goal of "Parcel does all the transpiling" to succeed, you'd have to know the specific version of every transpiler that every piece of source code was written against. Otherwise, there will end up being incompatibilities as the transpilers evolve.

It might not be obvious or quick to happen, but as time passes transpilers will definitely end up making breaking changes that are incompatible with previous versions. When that happens, Parcel will break because it will be using a newer (or even just different) version than the source code was written against.

Most good transpilers will try to make as much syntax backwards compatible as possible, but in stage-* proposals (to name one example) that can't be guaranteed. (Not to mention that even the target of stage-0 is variable over time.) (Not to mention too that that is purely considering the syntax layer—there is also the tooling layer of Babel/etc. themselves, which also evolves.)

One (unrealistic I think) solution would be for packages to somehow expose their complete transpiler configuration, which would then theoretically allow Parcel to use the proper versions. But this seems super hard to actually implement, because (1) it requires packages to be updated, which is not going to happen, and (2) it requires somehow defining and being aware of all the transpiler configurations, and (3) it puts Parcel in the position of trying to apply a zero-configuration abstraction to an infinitely complex space.

Instead, I think the only reasonable solution is to use the built-in contract of the package system: that package.main points to a file that exposes valid JavaScript (ECMAScript)—at the time of writing.

If someone is writing a package with a syntax that is not (current) JavaScript (be it TypeScript, or CoffeeScript, or stage-*, or any other transpiled syntax), they should never be exporting that directly as package.main if they intend it to be relied upon publicly.

And Parcel should not do any transpilation of packages by default, because it shouldn't need to, because they should be exported with valid JavaScript to begin with.

Since "JavaScript" (ECMAScript) is a moving target, there's always going to be variation in the "JavaScript" exported by these packages, with people migrating to the newest spec over time. And there is friction there, that may be a solution to try to solve with engines—just like how node versions are handled. But this is a secondary concern, and shouldn't impact Parcel's default configuration IMO.

The engines field is more of a warning system for developers as they integrate things, and could maybe be automagically handled. (I believe @gaearon suggestion of limiting the transpiling to only -env presets is essentially this "automagical" fix, although I think even that has issues?) For example, try to imagine how a transpiler might automagically handle abstracting away the idea of node versions. Doing the same thing for JavaScript would be slightly easier, since it attempts to guarantee "forever backwards compatibility", but still hard.

I'm pretty sure that this is the only realistic way to solve this if Parcel wants to be long-lived. Trying to transpile everything seems like a recipe for complexity in the short-term, and impossibility in long run, once you factor in a lifespan that spans across many major version bumps of the ecosystem.

ianstormtaylor commented 6 years ago

FWIW, in the short term, I think this functionality should be switched off ASAP, because it seems like it effectively makes Parcel unusable for packages that happen to ship with .babelrc. Today I needed to switch off Browserify, and wanted to use Parcel in its place, but it seems like this issue forces me to use Webpack instead, as much as I would have liked not to.

ianstormtaylor commented 6 years ago

FWIW too, regarding JavaScript being a moving target. I think that is causing a confusion between two different uses of transpiling, and right now they're getting thought of as one as far as Parcel is concerned...

1. Old Browser Support

Someone might decide they need to support older JavaScript environments than whatever the current commonly-exposed version is. (Right now "commonly-exposed" is probably ES2015, soon it might be 2016+) And to do that they might decide to use something like babel-preset-env to reach into node_modules and transpile all of the JavaScript in those modules to support an older environment.

This use case is (I think) the one @gaearon is talking about when he mentions restricting it to -env. I think this is a totally valid use case, but also that is not the primary use case, and thus shouldn't be enabled by default.

But the key here is that this entire use case lives within the realms of the current ECMAScript spec, so it's not infinitely growing in complexity. It's also not actually tied to Babel in any way, that just happens to be probably the easiest way to solve it.

2. Supporting Transpiling

Someone might decide they don't like writing JavaScript, for whatever valid reasons, and they want to write their code in TypeScript, or CoffeeScript, or ClojureScript, or UnstableFutureJavaScript, or who knows what.

This is okay to do in development, but shouldn't be exposed as package.main by modules. They should use their own transpile step before publishing to generate valid ECMAScript (at time of writing) and expose that instead. If they also want to expose their own thing, they could choose to do that, but it should be as package.somethingElseHere instead.

Trying to automagically infer these transpiler versions and configurations is super non-trivial as far as I can tell, and is a problem space that is much larger than just the scope of ECMAScript.

I think we're conflating the two.

(1) is a non-infinite problem space to solve, and requires keeping up with ECMAScript, which dedicates itself to backwards-compatible futures.

(2) is basically infinite, and requires keeping up with many different transpilers across potentially non-backwards compatible versions.

If Parcel wanted to solve (1) that would be really, really interesting I think. It would allow people to stop caring about JavaScript versions of their dependencies completely, and hasten the rise of people exporting ES2017 un-transpiled from packages, which I think is a great thing. All the other bundlers would start doing the same thing out of necessity. And it could do all of that with extremely minimal configuration, the way babel-preset-env does it. But its primary requirement is that people must be exporting valid ECMAScript (not future versions, and not other languages) in the first place, otherwise the problem space's complexity explodes.

And solving (1) wouldn't involve reading any of the .babelrc logic from node_modules, because it would only be applying the app's required configuration onto the whatever was exposed by the modules.

But trying to solve (2) is going to lead to lots of complexity and lots of incompatibility over time, which seems like exactly the opposite of the goal of zero-configuration to me.

ianstormtaylor commented 6 years ago

Here's an example of solving (1) with Webpack that @gaearon just added to Create React App, which sounds like something Parcel could turn on by default actually given it didn't have any negative impacts — https://github.com/facebookincubator/create-react-app/pull/3776 (with interesting discussion from https://github.com/facebookincubator/create-react-app/issues/1125)

(But note that this is still different from trying to respect the Babel configuration of those modules' original source code.)

gaearon commented 6 years ago

We’re going to cut an alpha with https://github.com/facebookincubator/create-react-app/pull/3776 soon and see if our users run into any issues. Please track the discussion in https://github.com/facebookincubator/create-react-app/issues/1125 if you’re interested in how that goes. Hope this is helpful!

devongovett commented 6 years ago

Ok, I think I've heard from enough people that the current behavior is problematic. We will do something about it. I like the proposal of using the engines field to determine the source language, and a .browserslistrc in the app to determine the target. We should be able to do a diff and see whether we need to transpile based on that information.

The main issue I have is with some projects I work on, which are developed as a bunch of local modules put together with npm link. I would like to avoid needing to run a manual build step each time one of those modules is changed. The modules include things like JSX, so don't really fit into any of the proposals here which all want standard JS features to be transpiled only. How do we go about solving this case?

gaearon commented 6 years ago

On our end, we're probably going to add support for Yarn Workspaces or similar. This doesn't solve the "multiple workspaces" problem though but I'm not sure we'd be aiming to solve it for our users at all ("why not a bigger monorepo then?")

ianstormtaylor commented 6 years ago

@devongovett I totally feel the pain of that issue. I have similar process annoyances currently, but I think it's the scope of something like Yarn Workspaces like @gaearon mentioned, and shouldn't be solved at the bundler layer, which should act on exported packages as is, instead of trying to dip into their source code and infer things.

(I could imagine other watching-based tools that might impose some sort of standard like reading package.scripts.build, watching across a user's entire HD, and executing builds when it sees development packages changing. Maybe. No idea. But I think, at least with the current level of tooling, it would be separate from the bundling process.)

devongovett commented 6 years ago

Made some progress on automatically transpiling to target app level in #559. Please provide feedback on the proposed approach there!

Doesn't solve the linking issue though. Workspaces are nice but not the solution to all problems and are hard to migrate to if you're not already using a monorepo. I think something to explicitly enable babel compilation of modules is necessary. Perhaps a {"parcel": {"babelrc": true}} field in package.json might work. We'd also want to require a babel-core version to be explicitly provided in package.json dependencies, along with all plugins needed. This is close to how browserify works, with its {"browserify": {"transform": ["babelify"]}} option (babelify must be an explicit dependency).

gaearon commented 6 years ago

I think something to explicitly enable babel compilation of modules is necessary. Perhaps a {"parcel": {"babelrc": true}} field in package.json

We have a similar problem, but in monorepos. It's not clear which packages should be treated as app "source" and some just happen to be shared but shouldn't be compiled. https://github.com/facebookincubator/create-react-app/pull/3741#issuecomment-357564101

It would be nice if we could land on some similar approach.

pindlebot commented 6 years ago

Is it absurd that I'm using this as a temporary workaround?

const path = require('path')
const fs = require('fs')

function rmBabelRc() {
  let nodeModules = path.join(__dirname, '../node_modules')
  let deps = fs.readdirSync(
    nodeModules
  )

  deps.forEach(dep => {
    let babelRc = path.join(nodeModules, dep, '.babelrc')
    if(fs.existsSync(babelRc)) {
      fs.unlinkSync(babelRc)
    }  
  })  
}

rmBabelRc()

bradfordlemley commented 6 years ago

I suggest splitting this into 2 separate things:

Standards part
- Package describes itself in terms of standards, e.g., "engines: {ecmascript: 2017}" as suggested above in https://github.com/parcel-bundler/parcel/issues/13#issuecomment-349808685.
- Importantly, only the project that is including the package knows the desired output, so it should transpile what is necessary based on the desired output target. The desired output could be described by browserlist or by node version or whatever, but that's in the realm of the including project and its tools.
- The description is standards-based and independent of any transpiler/tool.
Custom part
- Package describes how to build itself into standard (not just ES5), and describes its output, e.g. "use babel w/ my .bablerc to make ecmascript-2017".
- Tools like parcel could perform this custom build step if needed, then # 1 (which would presumably be very standard).

Ok, # 2 seems inefficient and difficult to support, so, actually, do that part before you publish.

A couple other comments:

JSX seems standard enough to include as a standard, e.g., "engines: {JSX: 1.0.0}". Well, things would be a lot cooler if it was considered a standard.
Handling existing packages that are ambiguously-described seems difficult.
engines:node doesn't seem like a concise way for the package to describe itself. It's ambiguous as to whether it applies to ES version or node apis (or both), and basically makes it impossible to ship a package that can be efficiently utilized for multiple output targets. So, deprecate it, and make another thing that specifies node apis that the package relies on. That way, you could ship an NPM package in ES6+ and let the including project's tools transpile it according the desired node target, and fail at build time if it requires newer node apis when attempting to transpile to older node version.

spion commented 6 years ago

The one shared thing seems to be that packages in source form are symlinked, right? Perhaps that can be used as an indicator (although I'm sure that would break pnpm, but pnpm always points to a X/v/node_modules/X directory so perhaps that could be used as an indicator too).

That wont work for source packages deliberately published on npm, but should be good enough to fix the burning dev issue.

edit: For a package containing sources, a field in package.json named source could point to the source directory or entrypoing just like main points to a main field. If the source entry file exists, the package is compiled and the "source" entry is treated as a replacement for "main". Otherwise its assumed that the source is missing (the package was published without it) and the "main" filed is used to resolve the compiled package as per normal.

ghost commented 6 years ago

What is the current workaround for this issue, other than @unshift's solution? Just stumbled on this after installing redux-devtools with redux-devtools-dock-monitor and redux-devtools-log-monitor.

🚨  /Users/samvv/Projects/engage/app2/node_modules/redux-devtools-themes/src/index.js: [BABEL] /Users/samvv/Projects/engage/app2/node_modules/redux-devtools-themes/src/index.js: Using removed Babel 5 option: /Users/samvv/Projects/engage/app2/node_modules/redux-devtools-themes/.babelrc.stage - Check out the corresponding stage-x presets http://babeljs.io/    at Logger.error (/Users/samvv/.config/yarn/global/node_modules/babel-core/lib/transformation/file/logger.js:41:11)
    at OptionManager.mergeOptions (/Users/samvv/.config/yarn/global/node_modules/babel-core/lib/transformation/file/options/option-manager.js:220:20)
    at OptionManager.init (/Users/samvv/.config/yarn/global/node_modules/babel-core/lib/transformation/file/options/option-manager.js:368:12)
    at File.initOptions (/Users/samvv/.config/yarn/global/node_modules/babel-core/lib/transformation/file/index.js:212:65)
    at new File (/Users/samvv/.config/yarn/global/node_modules/babel-core/lib/transformation/file/index.js:135:24)
    at JSAsset.getParserOptions (/Users/samvv/.config/yarn/global/node_modules/parcel-bundler/src/assets/JSAsset.js:60:20)
    at <anonymous>

Vanuan commented 6 years ago

I see the following considerations:

What is npm/node_modules/package.json package: is it only a channel of distribution, disregarding the contents, or is it also a standard which defines how to consume the contents?
Will we ever see truly universal JavaScript or it'll always be transpiled?
Will Web Assembly emerge? If it will, would it be a standard form to distribute code in the Web?

As I see it, zero configuration implies that there would be some standard, convention or minimal preset to author code on the web. But Parcel is not even compatible with what create-react-app produce. How it's going to be considered by developers?

andybarron commented 6 years ago

This is a major stumbling block for Parcel. IMO, "ignore .babelrc under node_modules" is the most sensible "zero config" solution, because almost all modules that use Babel are published to NPM precompiled.

andybarron commented 6 years ago

(At the very least, if I have a .babelrc in my project root with "ignore": ["node_modules/**/*"], I would expect Parcel to respect that!!)

Vanuan commented 6 years ago

You might want to consider how jspm is doing that. It also tries to transpile everything. Bit instead of treating all packages the same, it treats all of them as unique. And it has an extensive API to configure each and every package: https://jspm.io/0.17-beta-guide/installing-the-jsx-babel-plugin.html

Vanuan commented 6 years ago

I.e. consider copying babelrc files from packages into some directory where developer can tweak/override them. E.g. .parcel-overrides/some-es6-package/.babelrc-ignore or .parcel-overrides/some-es6-package/.babelrc-override

DieterHolvoet commented 6 years ago

Not every package stores their babel config in a .babelrc file, it can even be embedded into packages.json. Only handling .babelrc files is no solution.

devongovett commented 6 years ago

@gaearon

We have a similar problem, but in monorepos. It's not clear which packages should be treated as app "source" and some just happen to be shared but shouldn't be compiled. facebookincubator/create-react-app#3741 (comment)

So far the best approach I've come up with is some additional configuration in package.json. Maybe we could come up with something that works for multiple bundlers here instead of making it parcel specific.

One option is a bundlerOptions field in package.json, to specify options specific to bundlers. For example, babel could be enabled like this:

{
  "bundlerOptions": {
    "babel": true // enable babel, use .babelrc
  }
}

You could also use it to specify bundler specific config for babel:

{
  "bundlerOptions": {
    "babel": { // enable babel, override .babelrc config
      "presets": ["react"]
    }
  }
}

In order to make this work across Babel versions, we would require babel-core and all babel presets and plugins to be installed locally within the package. The bundler would use the locally installed babel to compile rather than the one built into Parcel or even the one installed in the app. This is likely not to cause too big of a problem as npm will already dedup babel when the same version is used across multiple packages.

This config could be read by Parcel, create-react-app, etc. Even babel-loader could theoretically read it, or maybe babel-register/babel-node as well (though maybe a different name than bundlerOptions would make sense then).

We can also combine it with the config we generate based on browserslist targets (as already implemented in this PR). That way you could only enable the react preset in your bundlerOptions babel config, and also specify a browserslist config for e.g. chrome latest. Parcel would look at the app's target browserslist to determine what env plugins to run, and also run the react preset. So depending on the app's target browsers, you could get ES6 or ES5 output, but JSX would always be compiled.

An alternative option to specifying a bundler specific babel config is specifying the language features you want to include, e.g. JSX, class properties, etc. But this is prone to bugs in the future as the spec for those things evolves, or proposals get dropped. It is better to explicitly specify babel plugins (and versions) for non-standard things.

Finally, after much thought, I think it is absolutely necessary to be explicit about this and require additional configuration even though I don't want to. There is no way to reliably infer that we need to use the .babelrc in a module. I thought about checking if babel-core was a dependency, but a quick search on npm shows that people include it as a dep instead of a devDependency by accident. I thought about checking whether all the presets/plugins were installed, but this isn't backward/forward compatible with babelrc formats (e.g. babel5 had a completely different config format). The only option is to be explicit about it via a bundlerOptions or similar flag.

Sorry for rambling on so long. What do you think of this bundlerOptions proposal? Would this work for your monorepo case as well?

Vanuan commented 6 years ago

What about babel plugins? Shouldn't Parcel also install devDependencies?

spion commented 6 years ago

I think there are two things being conflated here.

One is about supporting modules that are published as ES2015 or ES2016 (etc) code, as we move forward with new language features.

This could be handled by adding a field "main-lang" in package.json (specifies the language that main points to). No crazy presets allowed there, only official versions of the language spec.

The other is about compiling modules that have their sources included with the package.

Not recommended by npm, but sometimes you have no other options e.g. you want to distribute SCSS, and additionally this is true in workspaces and with linked modules.

The first thing you want to find out is whether the sources are, in fact included in the package. For that you could use a "source" field, which is the source file equivalent to the "main" field. If that file is present, the package was distributed with its sources. Otherwise, its not.

When sources are present, the bundler would expect that all the plugins and configurations (e.g. babel with .babelrc and any babel plugins and presets) are already present - with the right version - to generate a new "main". Parcel could try to support all that... OR (this is a bit radical, but not entirely unreasonable!) it could look up package.json for a script named "build" or "watch" (depending on mode) and run that, waiting for a new "main" - and its up to the module to deliver.

devongovett commented 6 years ago

@Vanuan If you want to use babel plugins in a node_module, then they must be a normal dependency, not a devDependency. Otherwise, npm won't install them.

@spion You're right, there are two aspects. I think we pretty much have a solution to part 1: use engines or browserslist to determine the engine support matrix for a module, and compile with babel-preset-env if needed. See #559. I am focused on figuring out part 2 at this point, in order to allow compiling modules with custom babel plugins (e.g. JSX) when needed (e.g. locally linked modules, monorepos, etc.).

I don't think a separate "source" field like "main" is necessary. The only thing Parcel needs to know is whether to treat what is in "main" as source (e.g. un-transpiled), or more specifically, whether to apply babel. This is accomplished by bundlerOptions.babel = true.

spion commented 6 years ago

Why would anyone write a module with custom-JSX-requiring code and point "main" to it? AFAIK main always points to pre-built code.

devongovett commented 6 years ago

Local development is a huge reason. It is convenient to be able to npm link a module locally and develop against it without needing to manually compile each time you make a change. See above: https://github.com/parcel-bundler/parcel/issues/13#issuecomment-357536878

spion commented 6 years ago

@devongovett I thought in monorepos packages still point the main field to a prebuilt file. e.g. https://github.com/infernojs/inferno/blob/master/packages/inferno-compat/package.json#L14

edit: would expect the same of linked modules. You wouldn't point main to a TypeScript file, for example, and I don't see why you would point it to other non-transpiled non-standard JS

devongovett commented 6 years ago

Sure it can, depends on your setup. But that sounds incredibly inconvenient. The idea of this feature it to make it easier to develop against linked modules.

Theoretically, the main field can point to whatever you want. Doesn't have to be compiled, doesn't even have to be JavaScript. It's just used as part of module resolution so you don't need to e.g. require("module/main.js"), you can just do require("module") instead.

parcel-bundler / parcel

Discuss .babelrc detection, engines field, etc. #13

1. Old Browser Support

2. Supporting Transpiling