[RFC] Drop websockets in favor of HTTP chunk streaming

[steelbrain]

• [ ] This is a bug
• [x] This is a modification request

Context

A few years ago I started working on a Module bundler from scratch (Pundle / https://github.com/steelbrain/pundle), It worked well for most simple cases and had plugin support and everything, I couldn't focus on it because of my fulltime job. I am hoping to upstream some of the clever fixes to the limitations I found working with Browserify and Webpack.

Why drop websockets?

There are many reasons for why we would want to remove Websockets from the dev server. The simplest explanation through code is that, webpack dev server consumers do this:

const app = express()
app.use('/mycoolwebsite', webpackDevServer({...})
const server = app.listen(9123, ...)

To setup a websocket server on an http server, we require access to the http server instance in NodeJS, which is only available AFTER calling .listen(), we cannot have it when registering the webpackDevServer route. Currently this is handled by setting up the websocket server on a different port inside the middleware and connecting to it instead.

It is complicated to setup webpack dev server with HMR when working with proxies. All of this can be avoided if we do simple HTTP chunking as described below.

The proposed fix

For HMR, the client just needs to listen to the server for updates, so it's a one-way communication, not bi-directional. Using fetch() API with a TextDecoder, we can keep an HTTP connection open for as long as we want and continue to write onto it from the server.

Here's the Client / Server implementation of it in Pundle: https://github.com/steelbrain/pundle/blob/fee7d9dc055591bf002b5a66190d651a02dbdfa4/packages/pundle-dev-middleware/src/client/hmr-client.js#L137-L157 / https://github.com/steelbrain/pundle/blob/4152d29de8b28be69c3e21bd3589a6109726310a/packages/pundle-dev-middleware/src/index.js#L220-L229

As you can see, we no longer need to worry about HMR port or hostname, the client JS can do a request to the server with selfUrl.webpack.hmr, it'll be routed to webpack through the same proxy that loaded the JS in the first place, webpack can then continue to write chunks on it until the client disconnects.

Benefits of proposed approach

• Massively reduce complexity for embedded dev server setups
• Reduced codebase complexity and ease of maintenance

---

Please let me know if there's anything I can explain to make it more understandable. Happy to do a PR implementing this change.

[alexander-akait]

The idea sounds good, but I think we can solve it more cardinally - we should extract hot client from webpack-dev-server into own repo, ideally webpack-dev-server = express + webpack-dev-middleware + webpack-hot-client-middleware + couple features like static watching/etc, also provide interface for communication between these packages, it is allow developers create own clients based on any solution and technologies, I'm afraid it may require quite a bit of work, but I would be happy for any help. How do you like this idea?

[ad-m]

To setup a websocket server on an http server, we require access to the http server instance in NodeJS, which is only available AFTER calling .listen(), we cannot have it when registering the webpackDevServer route. Currently this is handled by setting up the websocket server on a different port inside the middleware and connecting to it instead.

koa-easy-ws is able to lazy create websocket server without listening on any port: https://github.com/b3nsn0w/koa-easy-ws