Sharing ideas

[christianalfoni]

Okay, so from Olivers talk it seems that we might be able to share some ideas. So let me explain how Webpackbin solves it as I believe that service is most similar, due to server side app bundling as well.

So this is what happens on Webpackbin:

1. User opens a Firebase application. Basically Firebase serves a JS application and it connects directly to Firebase

2. Webpackbin now makes a POST request to the webpack-sandbox service with the files, packages and loaders for the BIN. If no session is active or the packages/loaders has changed a new webpack middleware is created for the session. It does in-memory bundling of your BIN files ONLY

3. If the POST request to webpack-sandbox contains packages it will ask webpack-dll for a manifest.json file though a CDN. It will either get it from the CDN, the DLL service database or the DLL service will find an available webpack-packager service to create the DLL bundle with the manifest

4. When the POST request is resolved Webpacbin loads an iframe pointing to the same sandbox URL, only it is now a GET request of course. The session is picked up and the index.html with injected script tag for bin bundle and script tag pointing to the DLL.js file of the requested bundle

5. Whenever the users saves the middleware on sandbox service just rebundles the BIN files, the DLL is cached in the browser

Codesandbox.io, @CompuIves, does not use webpack-sandbox, but talks directly to webpack-dll instead. Maybe Ives can share how he uses it :-)

When I saw your talk Oliver, and if I understand this correctly, it did occur to me that maybe you would get better performance by separating NPM packages from client code. Cause that would mean you in worst case scenario have to bundle the libraries the client is using once, rather focus on the custom code of the client instead. But maybe I misunderstood something here :)

Anyways, my question to you is... how do you solve the loaders of Webpack with MemoryFS? I had to do quite a few hacks, especially with Babel (which you can see in this repo) to make it understand what file system to use. It seems that though Webpack does support other filesystems, some loaders hardcodes to the actual filesystem... making it difficult to do this kind of stuff :)

[CompuIves]

This is how CodeSandbox does it:

CodeSandbox transpiles and evaluates the project in the browser using a custom require function (in a closure), so we do include the dependencies in the browser itself. This can be done because the dll service returns an umd bundle, which creates a global variable with all dependencies on the window called dll_bundle. These are the steps CodeSandbox takes:

1. Fetch the dll.js and the manifest.json from the dll service
2. Send all files and directories using window.postMessage to the sandbox iframe.
3. Add the dll.js to the head tag and start evaluating the user written code
4. Each time a dependency is encountered it will check in the externals of the manifest.json if the dependency exists, we take the module number, execute window.dll_bundle(/*module_name*/) and return the result.

[oliverwoodings]

Oh man you guys have done some crazy awesome stuff with both these projects, and I'm super impressed you are now both sharing the same core services 👍

When I saw your talk Oliver, and if I understand this correctly, it did occur to me that maybe you would get better performance by separating NPM packages from client code. Cause that would mean you in worst case scenario have to bundle the libraries the client is using once, rather focus on the custom code of the client instead. But maybe I misunderstood something here :)

Unfortunately the dependency list for a client could change between every build. Different users could be working on separate personalisations with different dependency requirements at the same time. We don't actually see many issues from this, however, for a few key reasons:

• The global set of dependencies a bundle could ask for is relatively small and very rarely changes. Users are not able to bundle in any old npm dependency. This is the package cache for the last 7 days:
• Because we have a restricted global set of dependencies, we can afford to leave them written into MemoryFS in a shared cache
• After monkey-patching mtime support into mfs, webpack will only end up compiling each dependency once; after that, it will see that the mtime of the files didn't change, so it will use its in-memory compiled module cache
• The immediately obvious issue with this approach is that if an instance goes down and a new one comes up, all the in-memory caches will be cold. We solve this by running a fake build through every instance when it comes online so that all the caches being acceptably hot. Through some load-balancer wizardry we also make sure an instance doesn't start getting traffic until the warmup has completed.

Anyways, my question to you is... how do you solve the loaders of Webpack with MemoryFS? I had to do quite a few hacks, especially with Babel (which you can see in this repo) to make it understand what file system to use. It seems that though Webpack does support other filesystems, some loaders hardcodes to the actual filesystem... making it difficult to do this kind of stuff :)

Unfortunately you probably aren't going to like my answer: we haven't needed to solve this, because we haven't come across the same issue. All builds use the same loaders, none of which attempt to do anything with the physical FS. Currently we only have 4 loaders:

• buble
• less
• a custom CSS injector
• a custom text loader

I wonder if maybe the webpack API could support your issue more - maybe if they made the input/output FS that webpack uses available to loaders/plugins?

One thing I mentioned in my talk was a shared compiled module cache, which webpack doesn't actually support right now. I've opened an issue on the Webpack repo after talking with @thelarkinn on Twitter so that we can discuss this further.

[bbrzoska]

maybe if they made the input/output FS that webpack uses available to loaders/plugins?

I believe this is already the case @oliverwoodings. Loaders and plugins shouldn't use the fs directly, but instead use the exposed fs (IIRC compiler.inputFileSystem and compiler.fs, which expose most of the same async methods as fs).

[oliverwoodings]

@bazyli-brzoska ah - so then the issues @christianalfoni is having is because certain loaders aren't using that compiler-provided fs correctly?

[christianalfoni]

@oliverwoodings So sorry for my late reply here :-)

Thanks for sharing and yeah it seems like you have gotten a good hold around how to handle NPM packages :-)

Yeah, about loaders. Cause it is possible to define on webpack what filesystem loaders should use, but I think the loader implementators are not referencing that, but rather directly imports "fs" module. Maybe it would be an idea to create PRs for this. For example try babel-loader first. If I find some time I will test that :)

Shared module cache seems very interesting, there is a hook for most things in Webpack it seems, so a shared module cache... why not? :)

I think where our projects differ in needs is that we want to keep NPM packages away from the app (bin) bundles and you need them to be compiled with them. That said, really glad we got a chance to share this as I am sure it inspires!

Really great talk btw ;-)