search

parcel-bundler / parcel

The zero configuration build tool for the web. 📦🚀
https://parceljs.org
MIT License
43.37k stars 2.27k forks source link

[RFC] Parcel 2: Programmatic API #2574

Closed devongovett closed 5 years ago

devongovett commented 5 years ago

While Parcel 1 did have a programmatic API, it was mostly focused around the CLI. In Parcel 2, we would like to expose a more comprehensive API to allow other tools to make use of Parcel under the hood. For example, Next.js, Gatsby, create-react-app, Vue CLI, Nuxt, Storybook, and others should be able to use Parcel internally if they want without hacking anything to achieve what they need. This is your opportunity to help us design an API for you! ✨

This issue is to investigate and discuss what external tools using Parcel would need from our API in order to make their jobs as easy as possible. We will break the outcomes of this issue out into separate issues for implementation. Below are some ideas to get started. If you are unfamiliar with Parcel 2's design (e.g. plugin types, config, etc.), please read the RFC.

Parcel options

The programmatic API should allow passing all options that the CLI would provide, in addition to some extras that would normally be resolved from the filesystem.

import {Parcel} from '@parcel/core';

let parcel = new Parcel({
  config: {/* ... */},
  entries: [/* ... */],
  targets: [/* ... */],
  // ...
});

await parcel.run();

ConfigProvider

This would allow providing config to other tools that Parcel runs rather than resolving them from the file system. For example, wrapper tools may want to provide a default config for Babel rather than requiring the user to have a .babelrc in their project.

I would imagine the API for this being a function, where a source file path is provided (e.g. a JS file from which the config should be resolved), along with a config filename that is being searched for (e.g. .babelrc). If no config is returned, the filesystem is searched as it normally would be.

import {Parcel, ConfigProvider} from '@parcel/core';

let configProvider = new ConfigProvider({
  getConfig(sourceFilePath, configFileName) {
    if (configFileName === '.babelrc' && !sourceFilePath.includes('node_modules')) {
      return myFancyBabelConfig;
    }
  }
});

let parcel = new Parcel({
  configProvider,
  // ...
});

This is just one API idea, but it is pretty flexible. If you have other suggestions, please leave a comment! One open question is what happens if there is both a filesystem config and a config provider returns something. Do they get merged or overridden? Perhaps we leave that up to the config provider itself.

Feedback

I am cc'ing this issue to a bunch of people who maintain some tools that might be interested in Parcel 2. This isn't to try to get them to switch to Parcel or anything like that, just to collect feedback and ideas so that we at least make it possible for tools like them to use Parcel in the future.

If there are other people or projects you think would be interested in helping us design Parcel 2, please let me know so I can reach out!

Please tell me what you think about the ideas listed above, and if you have other requirements or suggestions for the Parcel 2 API, please comment below. 🤗

iansu commented 5 years ago

Merging user and provider configs is tricky. You might want to allow users to override certain settings but not others. Maybe the user's settings would override by default but the provider could either specify a list of settings to filter out or just provide a function to do the merging itself.

devongovett commented 5 years ago

Yeah I was thinking of leaving the merging up to the ConfigProvider. Perhaps it could be a separate function overrideConfig(sourceFilePath, configFileName, config) that you could implement instead of getConfig if you wanted to support merging.

ndelangen commented 5 years ago

I'd be down to giving this a try.

Storybook would need APIs for starting in watch-mode and static-build-mode.

We also provide ways for users & presets to override the config. A method to merge config would be great.

devongovett commented 5 years ago

Awesome. I didn't list all of the options that are also available in the CLI above, including the watch option which will definitely be available, along with production for prod builds.

padmaia commented 5 years ago

Would it be desirable to accept an array of ParcelOpts?

jstcki commented 5 years ago

Hi 👋, I maintain Catalog which is a style guide tool similar to Storybook, Styleguidist and Docz. I'm also quite interested in this because maintaining a CLI tool and writing a good Webpack config that works for the majority of people's setups is frankly a pain. Integrating with a code base that has a specific setup is crucial (as well as providing a good zero-conf experience when starting from scratch).

I'd be happy with anything that allows extending configs in a predictable way (as opposed to having to deep-merge tool-specific configurations and hope that they don't change). Concretely:

These are just some of the things off the top of my head right now. I'd be happy to provide more details, test, etc.

P.S. another – maybe silly – idea: a tool like Catalog could work exclusively as a Parcel plugin, i.e. add functionality to an app's dev and build workflow instead of providing its own CLI. Let's say by using { "extends": ["@parcel/config-default", "@catalog/parcel-config-default"] } my plugin could add a separate route to the dev server (e.g. localhost:3000/__catalog) and on build create a separate directory from the main app with the bundled style guide. This way people wouldn't have to run two separate CLIs and I think some of above's issues could be solved pretty neatly.

rtsao commented 5 years ago

I noticed @parcel/core has a dependency on @parcel/config-default, which includes various transformers, etc.

For programmatic usage within other tools, it would be preferable if this was not a dependency in the "true" core package, so that whatever tool is using Parcel can supply its own configuration/plugins/transformers/etc.

For example, a CSS-only build tool using Parcel probably has no need for any JS-specific dependencies (i.e. @parcel/transformer-babel, etc.) so it would be nice if these were not included as dependencies in whatever package is used for programmatic usage of Parcel. For Parcel usage within other build tools, I think the responsibility of "zero-config" shifts to the maintainers of that tool rather than Parcel itself.

padmaia commented 5 years ago

@rtsao Eventually the cli will be pulled out of core and moved into the parcel package, and that package will depend on the default config instead of core.

brillout commented 5 years ago

This is awesome and I'm very much looking forward to integrate Parcel into Reframe (https://github.com/reframejs/reframe).

The API would need to provide:

In code:

const parcel = new Parcel({
  onBuildEnd,
});

function onBuildEnd({entries}) {
  // `entries[0].bundles` holds the built bundles of the first entry point
  assert(entries[0].bundles[0] === '/path/to/the/built/script-bundle.js');
  assert(entries[0].bundles[1] === '/path/to/the/built/style-bundle.css');
}

I'll dig into more details in the next couple of days.

devongovett commented 5 years ago

@brillout sounds like your use case would fit a reporter plugin! Reporters are passed events as they happen and can do whatever they need to do. The events will be: buildStart, buildProgress, buildSuccess, buildFailure, and log. buildSuccess will be passed the full asset graph and bundle graph that we generated as part of the build.

We are using reporters to implement our own CLI interface, and I imagine they'll be used for bundle visualizers, manifest writing tools, and more.

brillout commented 5 years ago

Neat, these events and the full asset graph are the most important things Reframe (and I guess any web framework) would need.

We are using reporters to implement our own CLI interface

Is the code already written? I've looked in parcel v1 and v2 but couldn't find such code

devongovett commented 5 years ago

@brillout it's a WIP on the reporters branch.

brillout commented 5 years ago

For Reframe I'd need the Parcel API to let me:

I believe every SSR framework will require these things as well.

In code:

const parcel = new Parcel({
  entries: {
    browserEntry_page1: '/path/to/browser/entry/of/page1.js',
    browserEntry_page2: '/path/to/browser/entry/of/page2.js',
    serverEntry: '/path/to/server.js',
  },

  targets: {
    browserEntry_page1: {
      "browsers": ["> 1%", "not dead"]
    },
    browserEntry_page2: {
      "browsers": ["> 1%", "not dead"]
    },
    serverEntry: {
      "node": ["^8.0.0"]
    },
  },

  buildSuccess: function(assets) {
    assert(assets.browserEntry_page1[0].file==='/path/to/bundle/of/page1.js');
    assert(assets.browserEntry_page2[1].file==='/path/to/style/of/page1.css');
    assert(assets.browserEntry_page2[0].file==='/path/to/bundle/of/page2.js');
    assert(assets.browserEntry_page2[1].file==='/path/to/style/of/page2.css');
    assert(assets.serverEntry[0].file==='/path/to/bundle/of/server.js');
  },

  // For development
  watch: true,
  minify: false,
  /* For production
  watch: false,
  minify: true,
  */
});

parcel.run();

// Needed when `watch: true`
parcel.stopWatching();
brillout commented 5 years ago

I just saw that almost all the things I need are actually already available in Parcel v1's API :+1:. I guess that Parcel v2's API is going to support all of Parcel v1's API options, correct?

As for targets it seems that the plan is to be able to set different targets for different entries, correct? This is crucial as the server entry targets Node.js while the browser entry targets the browser.

Thanks for Parcel btw, I really dig its zero config philosophy.

padmaia commented 5 years ago

I've just opened up a new issue suggesting a different approach for the ConfigProvider idea (#3284). I believe we've pretty much decided to move forward with the rest of this proposal so I was getting ready to close it, but there is an outstanding question in the comments from @brillout. Do we plan to support being able to configure different targets for different entries? I think we've thrown some ideas around like maybe adding them to package.json targets, but don't think we've landed anything. @devongovett should we keep this issue open until we decide on a solution for this or should we maybe create a more narrow issue for that?

brillout commented 5 years ago

SSR requires different targets for different entries.

For example:

// Browser (this code runs in the browser only)

import React from 'react';
import ReactDOM from 'react-dom';
import App from '../common/App';

ReactDOM.hydrate(
  <App />,
  document.getElementsById('react-container')
);
// Node.js (this code runs in Node.js only)

const express = require('express');
const React = require('react');
const ReactDOMServer = require('react-dom/server');
const App = require('../common/App');

const app = express();

app.get('/' , async (req, res) => {
  res.send(`
    <html>
      <body>
        <div id='react-container'>${
          ReactDOMServer.renderToStaticMarkup(
            <App />
          )
        }</div>
      </body>
    </html>
  `);
});

The first code snippet is the browser entry and runs in the browser only. The second code snippet is the server entry and runs in the server only.

If Parcel v2 doesn't support different targets for different entries, then no SSR tool will be able to use Parcel.

I'm almost done with a new SSR tool ssr-coin (github.com/reframejs/ssr-coin). It uses Webpack but I would absolutely love to use Parcel instead of Webpack.

Thanks to Parcel's zero config, the users of ssr-coin would not have to fiddle around with build. This is a boon — please make Parcel SSR compatible and allow different targets for different entries :-).

ndelangen commented 5 years ago

Or support running two configs in parallel?

brillout commented 5 years ago

@ndelangen That's fine with me. Although I expect the Parcel end user to not want to have to configure 2 configs.

What I do care though is that I don't want to have to run Parcel twice. That's what I'm currently doing with Webpack and it's a huge pain because I have to synchronise the browser webpack compilation and the server webpack compalition. That's complex and error prone. It's the most complex part of ssr-coin and, other than that, ssr-coin is a fairly straightforward piece of code.

A single Parcel run that compiles both for the browser and the server would make my life much easier.

devongovett commented 5 years ago

Going to close this issue out for Alpha 1. Everything here is implemented except the config provider, and there's a new issue for that: #3284.

I split out the multi target/entry discussion into a separate issue. Tried to summarize the ideas that have been thrown around here. Would be good to hear everyone's feedback there. #3302

mattdesl commented 4 years ago

I'd love to use Parcel 2 as the engine under future versions of canvas-sketch (I'm also planning on an Electron GUI for it).

Is there a good place to chat about feedback for the upcoming Parcel API? For now I might just ramble a little here...

All of this RFC stuff sounds great, although I'm having a hard time testing right now as I am not sure about the syntax or options to use (say, to achieve the equivalent of bundler.middleware() but with the v2 API). Is there any API documentation yet or copy-paste snippets I could try?

One of the things I noticed with V1 API is the design of "one project folder → one output". In the case of React/Vue/etc that might be true (all the source combines into one app), but in the case of canvas-sketch the project folder acts more like a sketchbook: each .js file in your directory can be run independently.

For my needs I will probably 'hide' the dist/ and .cache folders from the user in a tempdir folder, and only actually emit files in their project folder for final builds (e.g. building their sketch to a static HTML file). Another thing I am doing often is building to an inlined HTML site (inlining all JS+CSS) so that users can just worry about a single output file. Something like that is really hard with the V1 API, hopefully with V2 it becomes easier.

The goal of my own application is to make it dead-simple for beginner programmers who have no concept of things like caches, packages, bundlers, servers, etc. Think like Processing GUI.

Hope that gives a little insight into how others might like to use the API!

swyxio commented 4 years ago

probably better to open a new issue he’s not guranteed to see this. he says he has it implemented so maybe ask specific qtns for docs

mischnic commented 4 years ago

probably better to open a new issue he’s not guranteed to see this. he says he has it implemented so maybe ask specific qtns for docs

👁

All of this RFC stuff sounds great, although I'm having a hard time testing right now as I am not sure about the syntax or options to use (say, to achieve the equivalent of bundler.middleware() but with the v2 API). Is there any API documentation yet or copy-paste snippets I could try?

There is no middleware function comparable to the one from V1 (because the dev server is a plugin that can't really be exposed by @parcel/core with the current architecture). Unfortunately, there is no documentation at the moment. Looking at the cli package in packages/core/parcel is currently your best bet for the public API.

One of the things I noticed with V1 API is the design of "one project folder → one output". In the case of React/Vue/etc that might be true (all the source combines into one app), but in the case of canvas-sketch the project folder acts more like a sketchbook: each .js file in your directory can be run independently.

Compiling a single source into multiple directories is already supported. You might want to take a look at some SSR related RFCs, like this one: https://github.com/parcel-bundler/parcel/issues/3302

For my needs I will probably 'hide' the dist/ and .cache folders from the user in a tempdir folder, and only actually emit files in their project folder for final builds (e.g. building their sketch to a static HTML file).

We currently have dist inside .parcel-cache, and wouldn't that folder by hidden by default anyway?

KaKi87 commented 4 years ago

Hey there,

Everything here is implemented except the config provider

I can't find out how to use Parcel with this version and my IDE doesn't seem to find out either. Any docs, anything ? Thanks !

mischnic commented 4 years ago

@KaKi87 An example:

import defaultConfig from "@parcel/config-default";
import Parcel from "@parcel/core";

let bundler = new Parcel({
  entries: path.join(__dirname, "./index.js"),
  defaultConfig: {
    ...defaultConfig,
    filePath: require.resolve("@parcel/config-default")
  },
  defaultEngines: {
    browsers: ["last 1 Chrome version"],
    node: "8"
  }
});

// ------------------

await bundler.run();

// or

let watcher = await bundler.watch((err, buildEvent) => {
  // ...
});
// ...
await watcher.unsubscribe();
KaKi87 commented 4 years ago

@mischnic Thanks for your answer. Unfortunately, it doesn't work :

Error: Cannot find module './App'

Please note that I didn't changed my project structure nor my code while switching versions.

samvv commented 4 years ago

Just a heads up: when trying the snippet of @mischnic in @parcel/core@next, the following error is thrown:

/home/samvv/Projects/wiki/packages/@swiftly/app/node_modules/@parcel/workers/lib/Handle.js:88
(0, _core.registerSerializableClass)(`${_package.default.version}:Handle`, Handle);
                                    ^

TypeError: (0 , _core.registerSerializableClass) is not a function
    at Object.<anonymous> (/home/samvv/Projects/wiki/packages/@swiftly/app/node_modules/@parcel/workers/lib/Handle.js:88:37)
    at Module._compile (internal/modules/cjs/loader.js:1147:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1167:10)
    at Module.load (internal/modules/cjs/loader.js:996:32)
    at Function.Module._load (internal/modules/cjs/loader.js:896:14)
    at Module.require (internal/modules/cjs/loader.js:1036:19)
    at require (internal/modules/cjs/helpers.js:72:18)
    at Object.<anonymous> (/home/samvv/Projects/wiki/packages/@swiftly/app/node_modules/@parcel/workers/lib/WorkerFarm.js:28:38)
    at Module._compile (internal/modules/cjs/loader.js:1147:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1167:10)

Not sure what is wrong, but I'd thought I'd let you guys know nonetheless. Using @parcel/core does not give this problem.

mischnic commented 4 years ago

@samvv I've just tested my snippet with @parcel/core@next and @parcel/core@nightly both with Yarn and Npm. Doesn't happen for me.

{
  "dependencies": {
    "esm": "^3.2.25",
    "@parcel/core": "^2.0.0-alpha.3.2",
    "@parcel/config-default": "^2.0.0-alpha.3.2"
  }
}

Take a look at your lockfile, I think you got a newer (incompatible) version @parcel/workers than @parcel/core.

samvv commented 4 years ago

D'oh! Of course I forgot about my lockfiles! Will try it again. Unless I post another message assume it is working. Thanks!

samvv commented 4 years ago

Hmm OK so @parcel/core@next still gives me the same error even when doing a clean install. Luckily, @parcel/core@nightly works as expected, so I guess it does not matter much :smile: Maybe I forgot to remove something, so I don't believe this is an issue with Parcel.

adrianmcli commented 4 years ago

Is there an example repo of this anywhere? It's a little difficult to figure out how to use this new programmatic API as there are no docs.

mischnic commented 4 years ago

Two examples are here: https://parcel2-docs.now.sh/features/parcel-api/

brillout commented 4 years ago

Are there docs explaining what workerFarm's are?

mischnic commented 4 years ago

Not yet, I'll add that to my list.

A worker farm makes it possible to not kill & restart your workers on every Parcel invokation,

// would restart workers every time `new Parcel()` is used
await new Parcel(...).build()
await new Parcel(...).build();
await new Parcel(...).build();
await new Parcel(...).build();

// reuse workers
let workerFarm = createWorkerFarm();
await new Parcel({workerFarm}).build()
await new Parcel({workerFarm}).build()
await new Parcel({workerFarm}).build()
await new Parcel({workerFarm}).build()
await workerFarm.end();

(and in this case, it is needed for new MemoryFS(workerFarm); so that the MemoryFS knows how to communicate with the workers to keep the FS in sync)

(The API at https://parcel2-docs.now.sh/plugin-system/api/ only contains @parcel/types at the moment, not @parcel/workers).

brillout commented 4 years ago

https://parcel2-docs.now.sh/plugin-system/api/

Oh wow neat, I've been looking for this for a long time. Maybe it's time for me to retry to implement SSR/SSG on top of Parcel 2 :-). Although there are couple of FIXMEs that I need hehe :).

Thanks for the worker farm explanation. Sounds interesting and maybe it will play an important role in the context of SSR. (One major difficulties of SSR are dynamic entries: a user can create/delete pages while developing his SSR app; in Parcel terms: the Parcel build needs to be able to dynamically add/remove entries.)