Question about feature set

[rosenfeld]

Hi, I'm currently evaluating a switch from the Rails Assets Pipeline (sprockets) to a better resources build system. I was able to find everything I needed feature-wise in Webpack but I didn't settled on it because it doesn't support incremental builds so that I could deploy new changes quickly.

While searching for alternatives I found that Sigh does support incremental building but I'm not sure about the full set of features. So, could you please let me know if Sigh.js supports those features?

I think I've read it already supports those:

• languages: ECO templates, SASS, CoffeeScript, ES6 transpiler (Babel);
• support for source maps and uglifier (or any other minifier);
• support for a watch mode;

Those I'm not sure about the current status:

• dependencies management supporting AMD and Common JS styles so that it would be compatible out of the box with most libraries;
• ability to transform other browser libraries into an AMD or Common JS module without having to change the original sources (so that I could use the vanilla npm packages or submodules);
• allow generated resources to have a checksum hash for permanent caching purposes;
• allow images to be either embedded or externally referenced (with hashes in those cases) depending on some criteria like regexp or size and format;
• supports for IE8/IE9 (won't generate several style tags as they only support up to 31 stylesheets);
• supports the concept of initial and extra chunks, optimized for quickly initial loading;

Let me better explain the last item. I want to serve an initial "loading..." page as soon as possible. To do that, I want to be able to serve the smallest CSS file in the header which is just enough to render the initial loading page layout properly while the additional resources (JS) are loading. The CSS link tag will block the initial page rendering but it should be pretty small and would hardly change so it would be already cached in most cases.

But I don't want the JS resources to block this initial rendering process so I want to be able to load them from the header (so that they would start downloading as soon as possible) with the async (and defer) attributes so that they don't block the initial rendered page. Also, it should be possible to ensure the vendors chunk will be evaluated before the app code even if the app chunk is likely to be smaller and finish downloading before the vendors chunk.

Also, I want to have a vendors chunk with libraries which barely change so that they would be cached for a longer time. The other chunk would only contain the required code for the initial page full load. Then, as the user interacts with it, the missing code would be downloaded and evaled (everything minified and with source maps support) as needed. This is achieved in webpack using the async syntax for AMD or Common JS pattern:

require(['extra-code.js', 'extra-style.css'], function(c, s) {
  ...
});

Webpack supports lazy loading and commons chunks and even grouping lazily loaded chunks to reduce the amount of requests as well as embedding CSS into the served JS (although it's possible to use extract the CSS rules for the initial chunk into a separate stylesheet), which is what I'm looking for.

I've been doing some experiences with webpack here in case it's not much clear what I'm looking for:

https://github.com/rosenfeld/webpack-sample-project

But if Sigh.js can provide most of this while still supporting incremental build, then I'd take some time to learn how to achieve the same with Sigh.js.

Additionally, it would be awesome if it was possible to set up some dynamic rules for the watch mode so that it creates new chunks as new test files are created after the watch mode was started. I like to run each test file in a separate iframe, loading only their dependencies to make sure all dependencies are explicitly required. I couldn't get this working on webpack yet, so it would be a plus if sigh.js would also support something like that. :)

Could you please confirm what Sigh currently supports and what it doesn't?

Thanks!

[insidewhy]

Hi :)

sigh is just a build system, in my opinion one of the best ones and definitely my favourite one, whereas webpack is a module bundler that happens to have many aspects of a build system built into it.

To me this isn't so great, I prefer less monolithic code bases composed of many node modules, it's more flexible if your module bundler is separated from the build system so you can chose the best of both.

So some of your questions don't make sense when directed at sigh itself, however sigh (as your build system) + jspm (as your client side package manager instead of bower) + systemjs (as your module loader) I believe is a really great and modern combination to use and can be a better solution than webpack depending on what you need to do. jspm and systemjs are already integrated so essentially when you combine sigh+jspm you get systemjs for free. The only instances I'd personally chose webpack over sigh would be if there are a tonne of webpack modules I want to re-use, there are no equivalents as sigh plugins, and I don't have the time to write the sigh plugins myself.

dependencies management supporting AMD and Common JS styles so that it would be compatible out of the box with most libraries;

systemjs supports every single module style, then you can use the sigh-babel plugin via sigh to compile your input source files, or sigh-coffeescript or whatever if you want to avoid having the client compile the code. Maybe sigh-coffeescript doesn't exist but you get the point.

ability to transform other browser libraries into an AMD or Common JS module without having to change the original sources (so that I could use the vanilla npm packages or submodules);

systemjs is a universal module loader, you don't need to convert libraries as it is capable of reading every module type. Of course you can bundle it all into an optimised format though via jspm bundle.

allow generated resources to have a checksum hash for permanent caching purposes;

You'd write a sigh plugin to handle this, it would be a small number of lines of code. I don't think anybody has done it yet.

allow images to be either embedded or externally referenced (with hashes in those cases) depending on some criteria like regexp or size and format;

Would need to write a new sigh plugin for this.

supports for IE8/IE9 (won't generate several style tags as they only support up to 31 stylesheets);

Combining stylesheets is easily handled via sigh-concat etc. You can use different environments sigh build -e production to influence the build system, it might concat all the stylesheets together for production and not for the default (development).

supports the concept of initial and extra chunks, optimized for quickly initial loading;

That's not something I'm really familiar with, would you care to explain a little more thoroughly?

There is a bunch more information in the README, in this presentation and in the plugin writing guide.

The presentation currently needs chrome, you can use the arrow keys to progress through it and hit left/right to go forwards and backwards. It's kinda funny or something.

[insidewhy]

Oh about the initial and extra chunks thing... I did some googling. jspm bundle can easily handle this for you, it can create multiple output bundles and it only requests a bundle when one of the components is lazy-loaded.

[insidewhy]

Oh and I updated my first response to show that sigh-babel etc. aren't necessary given that systemjs will compile the assets client-side for you, just that you can use them to precompile assets for greater speed (sigh can use many processes, the node compilers are probably faster than the client-side implementations etc.).

[rosenfeld]

Thanks a lot for your answers, @ohjames. I've since then reading about jspm and systemjs, but there are a few things that are not completely clear to me.

Before I talk about them, let me first clear up a few things:

To me this isn't so great, I prefer less monolithic code bases composed of many node modules, it's more flexible if your module bundler is separated from the build system so you can chose the best of both.

I do also prefer more modular code bases but when you target some browsers, benchmarks will often show that for big apps loading a full bundle as a single file will often be faster for most browsers, even when using HTTP/2. So, bundling is a key feature for a build system, specially if you are still supporting IE8.

Now, let me explain how things seem to work in webpack and in jspm with regards to bundling as I understood from what I have researched so far.

In webpack, when you the sync style of require (require('./some-dep.js')) it will include the dependency in the bundle (unless it belongs to the common vendors bundle already). Then, if you write the async style require which relies on some dependencies which are not yet part of the entry or vendors bundle webpack will try to group the common dependencies in non-entry bundles that will be also minified and hashed and served lazily and all of this is transparent. You can set up how many dependencies to bundle together for lazy loading if you want to, or you can let it manage it the best possible way if you want to reduce the amount of requests as much as possible.

Now, one has to be more explicit with regards to jspm bundle as you'd have to manage those non-entry chunks yourself by excluding the vendors, app and their dependencies from the extra chunks of your app that will be lazily loaded.

Also, webpack is smart enough to include in the initial CSS only the rules required by the initial (entry) chunk. All other CSS rules would be loaded by JavaScript, when lazy loaded. I don't know how to define this behavior using jspm and systemjs.

Now, the slowest operation on Webpack is running uglify in the final output. I'm curious on how minification is performed with Sigh. Does sigh concatenates all sources before minifying or does it minify before concatenating? Even though the resulting bundle could be a bit bigger if minification is performed before concatenating, this strategy allows for better caching for incremental builds, which could reduce dramatically the total deploy time, which I'm interested in. So, how does that work in Sigh?

From what I was able to understand so far, there's a trade-off I have to put on balance. I can use a smarter bundling system, like webpack, and suffer with slow deploy times (deploying usually takes about 1 or 2 minutes to complete currently and we deploy a few times a week usually), or I can be more explicit about the bundles and perform this lazy loading extraction manually, which is a bit error prone and not optimal, and possibly get much faster deploy times by using incremental builds hopefully generating the production resources in just a few seconds rather than about 30 or 40s which I estimate it would take with webpack...

Overall, webpack looks cleaner to me while jspm is not that much. For example, while I can use @import "jquery-ui/themes/dialog.css" with SASS in webpack, the sass documentation for jspm mentions using @import "jspm:jquery-ui/themes/dialog". Also, I was able to find plugins for basically everything I needed in webpack but I find it very surprising that no one has written a plugin for generating a hashed filename for caching purposes in jspm or sigh.

Also, webpack documentation, even though far from ideal, is much more complete than sigh/jspm/systemjs combined for what I searched for so far...

There are some questions I have no idea on where to look at for an answer. For example, the Knockout.js build assumes window.jQuery exists if jQuery is available. I was able to replace jQuery with require('jquery') in the proper place by using a webpack plugin without breaking source-maps. How would I perform such replacements while keeping the source-maps with Sigh?

[rosenfeld]

For the commons bundling I was able to find this last example here which is very helpful:

https://github.com/systemjs/builder

[insidewhy]

I do also prefer more modular code bases but when you target some browsers, benchmarks will often show that for big apps loading a full bundle as a single file will often be faster for most browsers, even when using HTTP/2. So, bundling is a key feature for a build system, specially if you are still supporting IE8.

I wasn't talking about bundling. Someone who doesn't understand that bundling is necessary (until HTTP2) is someone who shouldn't be writing a build system. I'm talking about your tools... two single purpose tools working together are better than one monolithic tool combing two features.

[insidewhy]

Now, one has to be more explicit with regards to jspm bundle as you'd have to manage those non-entry chunks yourself by excluding the vendors, app and their dependencies from the extra chunks of your app that will be lazily loaded.

Actually jspm is more flexible than that... you can do like:

Bundle C = Bundle A - Bundle B Bundle D = Bundle C + Bundle E Bundle F = the module file.js and all of its dependencies

It may be explicit but it's so terse that you don't lose anything from having to be ever so slightly explicit.

[insidewhy]

Also, webpack is smart enough to include in the initial CSS only the rules required by the initial (entry) chunk. All other CSS rules would be loaded by JavaScript, when lazy loaded. I don't know how to define this behavior using jspm and systemjs.

With systemjs you can just lazily require('file.js') and if file.js is part of a bundle, it will fetch it from that bundle (fetching the bundle if it isn't already present) otherwise it'll retrieve the individual file. The config.js which is read by the browser (and jspm itself) describes a mapping of files to bundles (or no mappings if you aren't bundling anything).

[insidewhy]

Now, the slowest operation on Webpack is running uglify in the final output. I'm curious on how minification is performed with Sigh. Does sigh concatenates all sources before minifying or does it minify before concatenating?

Whichever you want, sigh is a stream based build tool that provides simple ways to do either of these things based on streams and plugins. It isn't a prescription on how to do things, you can't compare it directly to web pack in this way.

[insidewhy]

the sass documentation for jspm mentions using @import "jspm:jquery-ui/themes/dialog"

You could do that but in most cases you'll be assigning aliases to things and requiring those instead based on your config.js and stuff.

[insidewhy]

I find it very surprising that no one has written a plugin for generating a hashed filename for caching purposes in jspm or sigh.

These tools are only a few months old, web pack is several years old. Hopefully more powerful non-monolithic tools will win out, but currently more people are looking at existing state of the art rather than the future. Eventually hopefully things like jspm, sigh and aurelia will become very popular in time.

[insidewhy]

in sigh translating from ES2015 to ES5 then minifying after concatenation:

[ glob('*.js'), babel(), concat('all.js'), uglify() ]

Or minifying before concatenating:

[ glob('*.js'), babel(), uglify(), concat('all.js') ]

[rosenfeld]

I'll try to set up some time to make some experiments with sigh, jspm and systemjs and see how things work and I'll keep you updated of my progress and ask any remaining questions on how to achieve what I need for my application.

But I need to understand how to transform some source by doing some regexp replacements while keeping the source map, since I can't find how to do this with sigh anywhere...

Actually jspm is more flexible than that... you can do like:

Bundle C = Bundle A - Bundle B Bundle D = Bundle C + Bundle E Bundle F = the module file.js and all of its dependencies

It may be explicit but it's so terse that you don't lose anything from having to be ever so slightly explicit.

This could become really complex in an SPA with several separate modules. In my application, for example, it would load the common libraries bundle and the main app layout and Search features in an "app" bundle. When clicking in a "Saved Searches" tab, it would then load code and styles required to load the tab on demand. Same for the glossary menu entry, or for opening a deal in a new tab or editing a tab and so on. While I wouldn't have to think about the interdependencies between each of those modules when using the webpack approach, I'd have to carefully think about what they share and what they don't share when deciding which bundles to create. webpack will simply detect those and write chunks like 2.2-hash.min.js and so on. This is something less I'd have to worry about.

Thanks for your help!

[insidewhy]

While I wouldn't have to think about the interdependencies between each of those modules when using the webpack approach, I'd have to carefully think about what they share and what they don't share when deciding which bundles to create.

You can use [] around a statement when constructing a bundle to exclude dependencies...

Bundle Libs = [ 'lib/**/*' ]
Bundle A = [ 'module1/**/*' ]
Bundle B = [ 'module2/**/*' ]
// Bundle All = Libs + A + B

I'm pretty sure you can do everything webpack can do and more in just a few lines of code. In my most recent aurelia project I generate a few bundles using some javascript as the glob expressions it supports don't include negation, other than that I've found it works great.

[rosenfeld]

A path based approach wouldn't work in my case as there's some of the app code which is shared among a few modules while the dependencies are not quite obvious and will depend on the implementation itself, which could change from time to time. With the webpack approach I don't have to worry about things like that when the implementation changes and suddenly a module no longer requires another helper which used to be the case. I'm pretty sure the webpack approach is much better in terms of usability for what I need, but I'm really looking for fast incremental building, otherwise I would be already decided for webpack... The main reason why I'm looking at sigh is incremental build performance to speed up my deploys. This is the first thing I'll play with and after that I'll see what I can do with it once I'm convinced I can do fast incremental builds with sigh.

[insidewhy]

Shared modules belong in repositories, systemjs+jspm have ways of managing fetching/bundling of these also, I don't think the problem you're explaining exists.

[insidewhy]

My typical case for bundling is:

• Production: Everything bundled into as few files as possible. My users will be using the entire site anyway and won't gain anything from a finer granularity of bundling but extra latency.
• Development: Bundle all external dependencies in one bundle, don't need the overhead of requesting all the files for these when they rarely change, I rebundle them manually whenever I upgrade. Then just load source files for the project directly rather than from a bundle. Any shared code gets factored out into a new node module and ends up in the development bundle (with temporary exclusions using - and using npm link during testing if you want to use the live code as you edit it).

[insidewhy]

sigh is incremental by design, (event payload is an array of events denoting changes, helper code is provided in sigh-core for cases where incremental rebuilds are not appropriate). It also splits work over multiple CPUs/CPU-cores, this is extra important when loading say, the sigh-babel plugin. Just require('babel') alone takes node 700ms of (sync) time. It's great to be able to delegate this work to other CPUs to avoid blocking sigh's startup.

At the moment there is no bridge between jspm and sigh that supports incrmental bundling though. I have committed some code into systemjs/builder that's part of the current release aimed towards supporting this, but I need to add some more code to it before I can write a fully incremental sigh-jspm-bundler plugin. Until then sigh has to call jspm via the sigh-process plugin.

[insidewhy]

So essentially all of your preprocessing will happen incrementally and using multiple CPUs, your translating from ES2015/ES2016/TypeScript to ES5 etc. but the bundle part of the process will always be fully synchronous until sigh-jspm-bundler gets built.

[rosenfeld]

Let me give you concrete examples then of my specific application. There are a few models in our application like User, DealType, Analytics and several others. They are used by the other UI modules like Search, SavedSearchesManager, FieldsManager, DealViewer and so on. It's not always trivial to be sure about which modules rely on the other ones and many of those modules are so tiny that extracting them in another repository would only take more time from us for maintaining them. The DealType is currently probably used by all of them, but it doesn't have to be this way. Another implementation could be made so that it would no longer be directly used by all of them for example. This is just a detail on how it's currently implemented. With webpack's approach this is something I don't have to spend time thinking about and I would be able to focus in the code architecture only and stop thinking about the build system as it would detect how to optimize the bundles based on the lazy styles extracted from the code itself...

[insidewhy]

I know what you're saying but I'm rubbish at explaining to you why this problem doesn't actually exist, sorry English isn't my strong point! I highly suggest you take a thorough look at systemjs/builder, systemjs and jspm's documentation though if you're interested, then maybe also sigh!

[rosenfeld]

Production: Everything bundled into as few files as possible. My users will be using the entire site anyway and won't gain anything from a finer granularity of bundling but extra latency.

We used to think this way and if I thought this was still the case I would simply keep using the Rails Assets Pipeline since this is the current strategy we adopt.

But it will generate a 1MB (~ 200KB gzipped) JS bundle and another big CSS. We have signed an SLA of 5s at most for the initial loading time but our analytics tools is reporting about 10-15% of the users are loading the page in over 5s and most of the time is used downloading the JS and CSS resources from our Amazon Cloudfront CDN. This means we should serve less code to be able to render the initial complete layout more quickly. We can start downloading the extra code just after rendering the initial basic app instead of waiting for the next iteraction, which I indeed intend to implement, but I'm already backed by many data proving that the size of the resources is the reason why some clients take so long to load the application, while others are able to load the full application without using cached resources in about half a second.

[insidewhy]

We used to think this way

It's not the way I think, it's just the way the majority of the sites I've worked on have been, I tend to work with smaller companies. I've also worked on massive projects with many bundles though, it's not that I don't get the need, I'm just saying that no matter how you want to do your bundling it's not really any more difficult with webpack than it is with jspm and systemjs, you just sorta require/import/whatever things and it works basically, no matter how you set up your bundling... but I think the fact I can manually take control and do things like union operations when I want to makes systemjs+jspm more powerful than webpack rather than less so.

[rosenfeld]

Let me try to illustrate what I'm saying with regards the differences between webpack and jspm/systemjs so that you better get the picture on why webpack makes it transparent and easier.

// main.js
$ = require('jquery');
dep1 = require('./dep1.js')
dep2 = require('./dep2.js')

$(document).on('click', '.my-btn', function() {
  // lazily loads mod1
  require(['./mod1.js'], function(mod1) {
    // ...
  });
});

// mod1.js

// when this module is loaded through main, $ and dep1 are already loaded, so no additional request
// is performed in this case:
require(['jquery', './dep1.js'], function($, dep1) {
  $(document).on('click', '.other-feature', function() {
    require(['./dep2.js', './dep3.js', './dep4.js'], function(d2, d3, d4) {
       // d3 and d4 are detected as missing by webpack and processed in a single requested chunk
    });
  });
});

As you can see, I'm free to think about dependencies only when writing my modules when using webpack. This is a very contrieved example and it's not always trivial to find out all possible work-flows manually. In that case, webpack would extract jquery to the commons chunk as by the configuration and main.js, dep1.js and dep2.js into app.js. It would then create some dynamic chunk names, like 2.2-hash.min.js containing dep3.js and dep4.js which would be then loaded at demand when .other-feature is clicked. I find this feature very powerful, but I may consider writing the chunk rules manually if it results in a faster deploy.

[insidewhy]

I see, the bundles are created dynamically, they must be scanning the source files and parsing for require statements.

This might be even easier with ES2015 + jspm as you can distinguish between require, import and System.import easily. I reckon I could write a sigh plugin for that with not too much effort.

Thanks for the idea!

[rosenfeld]

Exactly, the source is parsed by Webpack and seems to be even smart about it, by supporting require('templates/' + templateName + '.js') if I understood it correctly, even though I can live without that feature:

https://webpack.github.io/docs/motivation.html

It also supports the import from ES6 syntax in addition to AMD and CommonJS.

[insidewhy]

That's pretty cool. If you assume templateName is a constant then it wouldn't be that hard to support it, could use esprima or babel APIs.

[rosenfeld]

or maybe you could treat it as an * and build for all combinations of 'templates/*.js'.

[insidewhy]

I guess it would have to be templates/**/*.js

[rosenfeld]

yep

[devinus]

Is sigh still maintained? Is there a roadmap? I'm looking for good n:1 build tools because I have VERY specific needs and it seems my only good options are sigh and brocolli.

[insidewhy]

@devinus I'm still maintaining it yes, I don't receive many pull requests but I tend to the ones I get promptly.

My future plans are:

• Port from bacon to rx.js
• Work on better integration with jspm.
• Grouping of lazily loaded chunks as @rosenfeld explains in a comment above.

I won't be starting any of this work in the next month but will still be maintaining the project until that work starts. If you choose to use it and have any queries I'll be here to help also!

[unlight]

@ohjames Switching to rx.js is it final decision? Do you have any intentions to switch to kefir.js which has a better performance and less usage of memory, and better documentation?

[insidewhy]

I was previously considering Kefir but the majority of sigh users and potential sigh users ask for Rx. Rx is more powerful than Kefir but I wasn't aware it performed significantly worse. It's also being used by Angular 2 and is closer to the official observables spec. Seems to have a lot going for it.

[insidewhy]

@unlight I think the memory usage/speed of the FRP implementation is also insignificant. It probably accounts for a tiny fraction of total memory/CPU usage in the build system.

[insidewhy]

@rosenfeld I'd really like to think you for bringing this excellent code splitting technique to my attention: I'm trying to convince the jspm authors now of the merits of this technique.

You can see that even very small aurelia apps (a very awesome modern web framework built on top of jspm/systemjs) pause for a long time on startup, I think the only way they'll be able to solve this issue is via the technique you explained.

[rosenfeld]

@ohjames I'm the one who should thank you for carefully answer my questions and consider improving the situation of sigh :)

I've recently written 3 articles about loading performance of web applications in case you'd be curious about:

http://rosenfeld.herokuapp.com/en/articles/2016-02-29-getting-an-spa-to-load-the-fastest-possible-way-and-how-webpack-can-help-you http://rosenfeld.herokuapp.com/en/articles/2016-02-29-scripts-loading-trade-offs-a-performance-analysis http://rosenfeld.herokuapp.com/en/articles/2016-02-26-improving-spa-loading-time-with-webpack-and-why-sprockets-is-in-your-way

The last one will probably only be relevant to Rails users, but I've provided the link anyway just in case :)