[IDEA] Allow for Universal Apps with HMR

[jaredpalmer]

Currently Backpack is focused on providing the best development experience for Node.js server-side applications and libraries.

This was a deliberate decision.

The idea is that people should use tools like Create React App or Next.js for their frontend and then use Backpack to build out API's etc. My goal was to create a complimentary tool.

However, it appears that the community wants a some sort of drop-in solution like CRA's react-scripts but for isomorphic / universal apps. I'm not sure why it isn't more popular, but this is what the New York Times' kyt project aims to do. Personally, I don't care for the aesthetics of kyt (e.g. the emoji's in the console) and for some of the conventions (like extract text/css and the eslint-config airbnb), but those are just my opinions. Regardless, Backpack could technically accommodate universal apps with a few small, yet non-trivial modifications.

In my research, I've come up with two approaches to Universal Apps with Hot Module Replacement and server reloading worth considering:

(Note: these have been extracted from other React server-side rendered projects of mine. However, they are not yet drop-in replacements to backpack dev. They do not currently handle custom webpack modifications like Backpack's currently does. These are just POC's).

1. Chokidar, Nodemon, Webpack-Hot-Middleware

This serves up client-side assets on another port like localhost:3001. It would be up to the user to properly reference where the assets are served from in their apps. This isn't necessarily a bad thing though, as these frontend assets should ideally be served from a CDN in production anyways. We could handle this by providing a Webpack flag to the server for use in the application's HTML template such as BACKPACK_ASSETS_URL.

// Proof of concept #1 dev.js
const nodemon = require('nodemon')
const path = require('path')
const chokidar = require('chokidar')
const express = require('express')
const webpack = require('webpack')
const url = require('url')
const once = require('ramda').once
const devMiddleware = require('webpack-dev-middleware')
const hotMiddleware = require('webpack-hot-middleware')
const serverConfig = require('../config/webpack.dev.server')
const clientConfig = require('../config/webpack.dev.client')
const { clientUrl, serverSrcPath } = require('../config/paths')

process.on('SIGINT', process.exit)
let clientCompiler, serverCompiler

const startServer = () => {
  const serverPaths = Object
    .keys(serverCompiler.options.entry)
    .map(entry => path.join(serverCompiler.options.output.path, `${entry}.js`))
  const mainPath = path.join(serverCompiler.options.output.path, 'main.js')
  nodemon({ script: mainPath, watch: serverPaths, flags: [] })
    .once('start', () => {
      //console.log(`NODEMON: Server running at: ${'http://localhost:3000'}`)
      //console.log('NODEMON: Development started')
    })
    // .on('restart', () => console.log('Server restarted'))
    .on('quit', process.exit)
}

const afterClientCompile = once(() => {
  // console.log('[WEBPACK-CLIENT]: Setup RHL')
  // console.log('[WEBPACK-CLIENT]: Done compiling client')
})

const compileServer = () => serverCompiler.run(() => undefined)

clientCompiler = webpack(clientConfig, (err, stats) => {
  if (err) return
  afterClientCompile()
  compileServer()
})

const startClient = () => {
  const devOptions = clientCompiler.options.devServer
  const app = express()
  const webpackDevMiddleware = devMiddleware(clientCompiler, devOptions)
  app.use(webpackDevMiddleware)
  app.use(hotMiddleware(clientCompiler, {
    log: () => {}
  }))
  app.listen(url.parse(clientUrl).port)
  // console.log('[WEBPACK-CLIENT]: Started asset server on http://localhost:' + url.parse(clientUrl).port)
}

const startServerOnce = once(() => startServer())

const watcher = chokidar.watch([serverSrcPath])

watcher.on('ready', () => {
  watcher
    .on('add', compileServer)
    .on('addDir', compileServer)
    .on('change', compileServer)
    .on('unlink', compileServer)
    .on('unlinkDir', compileServer)
})

serverCompiler = webpack(serverConfig, (err, stats) => {
  if (err) return
  startServerOnce()
})

startClient()

2. BrowserSync, Proxy-Middleware, Webpack-Hot-Middleware

The following is heavily inspired by Ueno's React Starter project.. It uses a http-proxy and browser-sync to work out the ports. My only criticism of this technique is that browser-sync and Docker do not play nicely with each other at all (last time i checked). That is either irrelevant or a dealbreaker for people.

const path = require('path')
const url = require('url')
const bs = require('browser-sync').create();
const webpack = require('webpack')
const proxyMiddleware = require('http-proxy-middleware');
const webpackDevMiddleware = require('webpack-dev-middleware')
const webpackHotMiddleware = require('webpack-hot-middleware')
const color = require('cli-color');
const debug = require('./debug');
const serverConfig = require('./webpack.dev.server')
const clientConfig = require('./webpack.dev.client')
const { clientUrl, serverSrcPath } = require('./buildConfig')

const domain = color.magentaBright('webpack');

// Get ports
const port = (parseInt(process.env.PORT, 10) || 3000) - 1;
const proxyPort = port + 1;

// Create compilers
const clientCompiler = webpack(clientConfig);
const serverCompiler = webpack(serverConfig);

// Logging
const log = (...args) => debug(domain, ...args);

// Build container
const build = {
  failed: false,
  first: true,
  connections: [],
};

const devMiddleware = webpackDevMiddleware(clientCompiler, {
  publicPath: '/',
  noInfo: true,
  quiet: true,
  stats: {
    timings: false,
    version: false,
    hash: false,
    assets: false,
    chunks: false,
    colors: true,
  },
});

serverCompiler.plugin('done', stats => {

  if (stats.hasErrors()) {
    log(color.red.bold('build failed'));
    build.failed = true;
    return;
  }

  if (build.failed) {
    build.failed = false;
    log(color.green('build fixed'));
  }

  log('built %s in %sms', stats.hash, stats.endTime - stats.startTime);

  const opts = serverCompiler.options;
  const outputPath = path.resolve(opts.output.path, `${Object.keys(opts.entry)[0]}.js`);
  // Make sure our newly built server bundles aren't in the module cache.
  Object.keys(require.cache).forEach((modulePath) => {
    if (modulePath.indexOf(opts.output.path || outputPath) !== -1) {
      delete require.cache[modulePath];
    }
  });

  if (build.listener) {
    // Close the last server listener
    build.listener.close();
  }

  // Start the server
  build.listener = require(outputPath).default; // eslint-disable-line

  // Track all connections to our server so that we can close them when needed.
  build.listener.on('connection', (connection) => {
    // Fixes first request to the server when nothing has been hot reloaded
    if (build.first) {
      devMiddleware.invalidate();
      build.first = false;
    }

    build.connections.push(connection);
    connection.on('close', () => {
      build.connections.splice(build.connections.indexOf(connection));
    });
  });
});

log(`started on ${color.blue.underline(`http://localhost:${proxyPort}`)}`);

serverCompiler.watch({
  aggregateTimeout: 300,
  poll: true,
}, () => undefined);

clientCompiler.watch({
  aggregateTimeout: 300,
  poll: true,
}, () => undefined);

// Initialize BrowserSync
bs.init({
  port: proxyPort,
  open: false,
  notify: false,
  logLevel: 'silent',
  server: {
    baseDir: './',
    middleware: [
      devMiddleware,
      webpackHotMiddleware(clientCompiler, {
        log: () => {}
      }),
      proxyMiddleware(p => !p.match('^/browser-sync'), {
        target: `http://localhost:${port}`,
        changeOrigin: true,
        ws: true,
        logLevel: 'warn',
      }),
    ],
  },
});

process.on('SIGTERM', () => {
  if (build.listener) {
    build.listener.close(() => {
      log('closing %s connections', build.connections.length);
      log('shutting down');
      build.connections.forEach(conn => {
        conn.destroy();
      });
      process.exit(0);
    });
  }
});

Anyways, those are some ideas. I'd love to get the discussion going. I've been thinking about this since reading @jlongster 's Backend Apps with Webpack

[jaredpalmer]

cc @ctrlplusb @gajus @koistya

[sompylasar]

There's also a third option, though it cannot be considered an honest "with Webpack" solution. https://github.com/halt-hammerzeit/webpack-isomorphic-tools + nodemon

These webpack-isomorphic-tools implement certain parts of webpack functionality (loaders, require aliases) without actually building through webpack on the server-side, all the magic happens via magic require hacking. This approach adds certain benefits, such as you don't need a build step for the server. But it's not entirely webpack-compatible.

And there is the fourth option of the same author which I haven't used myself: https://github.com/halt-hammerzeit/universal-webpack + webpack target: "node" + nodemon

[jaredpalmer]

@sompylasar yup. also worth exploring solutions with those.

[ericclemmons]

Related, this is about how we did our universal apps ~6 months ago:

https://github.com/ericclemmons/terse-webpack/tree/master/example

Notice how it's a single process with webpack-{dev,hot}-middleware doing the heavy lifting.

[wmertens]

I simply include all the loaders for client on the server side as well and import the client code directly. Inside the server I run webpack for the client bundle. So I have webpack -> universal server with HMR -> webpack -> client bundle with HMR.

I encountered lots of bugs with Babel though, especially if I include things in the Babel config. I should bundle my webpack config first and then run it.

I also wish hot reloading on the server could all be done in memory and not using polling.

[wmertens]

Hey here's an idea: use https://webpack.github.io/docs/node.js-api.html#compile-to-memory and then read the entry into memory, prefix it with something that patches require to use the memoryfs result and eval it.

For HMR, go deep into Webpack code and implement a mode that uses a global EventEmitter for HMR notification?

Maybe all this needs to be implemented in webpack proper, though. Thoughts?

[koistya]

BTW, is there any good reason to use Webpack in data API / backend projects? Isn't it supposed to be used in front-end projects only? See Node.js API Starter Kit (no Webpack)

E.g. the code is transpiled directly with Babel, no need to minimize the code for the server as it won't have performance benefits. Loaders - it's front-end related concept, using them on backend makes things more complex than they should be I believe, HMR on the server is easy without Webpack, just a few lines of code to cear the cache, require Express module(s) again and run app.listen(..). Demo:

[wmertens]

On Sat, Jan 14, 2017 at 3:04 PM Konstantin Tarkus notifications@github.com wrote:

BTW, is there any good reason to use Webpack in data API / backend projects? Isn't it supposed to be used in front-end projects only? Ref Node.js API Starter Kit https://github.com/kriasoft/nodejs-api-starter (no Webpack)

With webpack, you can transpile, and you can optimize => faster production server loading.

You can also use loaders that do special things and use HMR to develop backend endpoints faster (see https://github.com/ericclemmons/start-server-webpack-plugin/) => better development experience.

[krainboltgreene]

Just a quick clarification, does backpack support react by default or not?

[jaredpalmer]

Technically yes, it is possible. However, I don't think you'd consider it a great DX. All defaults and scripts are meant for Node.js-only and not for the browser right now.

Will publish a fork though that implements one of the above solutions either tonight or tomorrow hopefully.

[RaghuChandrasekaran]

Will publish a fork though that implements one of the above solutions either tonight or tomorrow hopefully.

Is there a fork or a sample that has the proposed solution for universal apps?

[ericclemmons]

I've been working on the #39 HMR example, but also forking off to see what changes would be necessary for a great DX that worked with:

• webpack-{dev,hot}-middleware
• create-react-app
• next.js

(In an effort to no longer maintain @terse/webpack :D)

[jaredpalmer]

I have a universal react hmr example in progress here. Would need to make major changes to backpack to integrate, but that's okay.

[jaredpalmer]

See https://github.com/jaredpalmer/razzle