[Feature proposal] Use MemoryFS instead of disk filesystem

[peter-mouland]

maybe this should be on https://github.com/kossnocorp/assets-webpack-plugin but i see the owner is looking for help to maintain that project, so i thought i'd try here first.

Is there a way to write the webpack-assets.json to memoryFS rather than disk?

or maybe just return the resulting json rather write to disk?

the final step would be to get isoTools server to read from memory if possible?

[catamphetamine]

That's an interesting proposal. That would be a good benefit.

I made some quick research on the internets, and seems that such a feature is quite possible to implement.

First, one needs to not use webpack-dev-server command and instead start webpack dev server using webpack Node.js API. That's an easy task, I've done this many times. Here's the example of how it's done:

import webpack from 'webpack'
import express from 'express'
import webpack_dev_middleware from 'webpack-dev-middleware'
import webpack_hot_middleware from 'webpack-hot-middleware'

const compiler = webpack(configuration)

const development_server = new express()

development_server.use(webpack_dev_middleware(compiler, development_server_options))
development_server.use(webpack_hot_middleware(compiler))

development_server.listen(3001, (error) =>
{
    if (error) 
    {
        console.error(error.stack || error)
        throw error
    }

    console.log('[webpack-dev-server] Running')
})

Now, about memory-fs. This module is written by webpack maintainer, and that's how MemoryFS is enabled in a Webpack compiler:

import webpack from 'webpack'
import MemoryFileSystem from 'memory-fs'

const compiler = webpack(configuration)
const fs = new MemoryFileSystem()
compiler.outputFileSystem = fs

So now we have fs filesystem object from which we can obtain any file

if (fs.existsSync(path))
{
    return fs.readFileSync(path, 'utf-8')
}

So, this way one has already Webpack development server running and using memory-fs.

Proposedly that memory-fs setting will be enough to make write assets plugin (part of webpack-isomorphic-tools) write webpack-assets.json file to that memory-fs filesystem instead of a disk filesystem.

Now, about Node.js application (e.g. React rendering application). The Node.js process starts, require() function is hooked into (using require-hacker), and every time a file is require()d the require function hook checks if the requested file resides in that MemoryFS fs filesystem, and if it does, then require() simply returns that file from fs.

How can Node.js application process get data from fs which exists in a separate Webpack development server process? That can be done through "inter-process communication", e.g. TCP sockets.

The described process of require()ing a file is synchronous by its nature, so communication between Node.js application process and Webpack dev server process must be synchronous too. I looked at Node.js's net module and didn't find any synchronous functions (like readFileSync but for network sockets).

So seems that there's no native Node.js module for synchronous communication over a network socket.

Maybe there are 3rd party modules? The ones that are just wrappers around some synchronous C/C++ socket library.

I found some: https://github.com/JacobFischer/netlinkwrapper

There are also node-fibers on top of which any asynchronous stuff can be made synchronous. Like this: https://github.com/olegp/common-node

But, it should be a well-maintained module because Node.js major versions change quickly and so do native V8 bindings and such. There are usually issues with compiled modules when the next Node.js major version is released.

There's also another route: through Node.js native process module which has synchronous functions. https://github.com/ForbesLindesay/sync-request

The idea behind using process is simple: the application synchronously (through spawnSync) spawns a child Node.js process which communicates over TCP socket, gets response data, and then exits, returning the obtained data from the spawnSync function call.

So, seems that all the required tools are available. So anyone who's maybe exited about this feature can fork the repo and implement this approach, and then send me a Pull Request and I'll merge that functionality into the project. Even though the plan is trivial, that's gonna be a lot of work.

As for me, currently I'm not bothered at all about webpack-isomorphic-tools writing webpack-assets.json to disk every time the developer saves a file: SSD lifetime span has increased greatly during the past years, so seems that no one is worried about wearing them off anymore.

[catamphetamine]

Made a small commit on this matter. The master branch now looks for port configuration parameter, and, if it's found, then it runs an express.js HTTP service serving webpack-assets.json from RAM. http://0000:${port}/ will give application/json response with Webpack assets. So, seems that MemoryFS is not required at all and can be dismissed.

The next step on this feature could be looking for port configuration parameter inside assets() function of source/index.js, and, if it's present, then send a synchronous HTTP request to http://0.0.0.0:${port}/ obtaining Webpack assets that way rather than require()ing them from disk. And no require() hook intervention is required either. So it turns out to be a significantly simpler task overall.

(webpack-assets.json existence check in this case can be substituted with pinging http://0.0.0.0:${port} for a JSON response)

[catamphetamine]

Well, this turned out to be pretty fast. Not "a lot of work" as I estimated earlier.

So, I seem to have made it.

Try installing the latest version (2.5.0), set port configuration parameter to some value, and then tell me if it works (delete your old webpack-assets.json and confirm that it's not created).

[catamphetamine]

@peter-mouland ^

[catamphetamine]

@peter-mouland Oh, and you'll need to run webpack dev server not as webpack-dev-server command but rather as a Node.js application. See an example: https://github.com/halt-hammerzeit/webapp/blob/master/webpack/development%20server.js express package must be installed for this to work.

[catamphetamine]

@peter-mouland Or will you need to run webpack as a Node.js application? Maybe not. If you're already using webpack-dev-server then stick to it for a while. I guess it should work too. But express still must be installed.