Speed up development workflow (browser refresh downloads are too slow)

[lancejpollard]

One of the other big things that's been keeping us short of an ideal workflow is how long it takes to rebuild whenever you change a file. Even if you have every component you need downloaded, and everything is built from the duo cache, the browser has to potentially download a few megabytes of JavaScript during development. Right now in one of our apps it's downloading 2.3MB of JavaScript in development.

Note, when this gets minified, it gets down to a fraction of the size, around 500KB, so cutting down on the amount of JavaScript we're requiring, while an ideal, won't really help too much here. That and in production the browser cache speeds up all subsequent loads.

So basically, every time you make a small change to the JavaScript, the browser has to download 2.3MB again and again, which takes anywhere from 2-5 seconds.

I don't really see a way to improve the development speed if the build is always as a single file. But maybe there is a way to have the browser download each component separately during development.

What if it was sort of like how Sprockets was in Rails, where in development, it creates a script tag for each individual component. This would mean that the browser caches everything, except for the small amount of code you changed (maybe just a few KB), so reloading would be super fast!

Obviously duo isn't setup for this atm, but wanted to open up the conversation and see what you guys think. Something like this would speed up the workflow a lot, from taking 2-5 seconds per refresh to a fraction of a second, which would make it so you can get into the flow. Right now I can't really get into the flow because it's so slow making each little change.

What this would take is:

• Making require globally accessible (maybe not the default behavior, but possible). This way, any subsequent script can use it.
• Adding some sort of way to generate code so each component can load separately, and be cached by the browser. I have some ideas how this could be done, but there are many other ways that would work too.

One way to do this for node-land is to add plugins for express/koa. These plugins would take the duo.json as input, and output an array of <script> tags that you can pass along in your HTML rendered from the server. They would point to individual components, with some sort of cache invalidation in the path like an md5 hash or whatever. The middleware would also, when it gets a request for the component, read the duo.json cache (or just the file perhaps) if it changed, and send that to the browser. The browser would cache it.

Another way to do it would be to write out each component individually to some build folder, along with the main build.js that is just a document.write thing like this:

document.write(''
  + '<script src="/build/component-emitter@1.0.0.js?bust=1847198y9c9a98uca">'
  + '<script src="/build/another-component...">'
  + ...
);

That would mean, whenever you change a file in development, duo just regenerates that simple document.write(scriptTags), which you can load. And then the browser will only fetch the changed components.

Something like this would be awesome. It would mean that development doesn't slow down as the project grows. Put in sort of Big-O notation, it would be O(1) rather than O(n).

Thoughts?

[matthewmueller]

yah this is a good idea. I think we'll just need to tell duo-pack to use a different algorithm when process.env.NODE_ENV != "production".

As for the require stuff, I think this would help us support bundles a little bit better, but I do really like just require'ing a <script> tag and everything loading, so that should remain the default imo.

[inca]

+1 on this one, is it what I intuitively expected to get from Duo straight out-of-the-box :)

[micky2be]

2 to 5 seconds to download 2.3Mb?? What kind of connection you have? It takes me 1s or less to download my 5Mb app. You could easily build a cache system yourself btw

[inca]

Sounds pretty much like an overkill to me, if you send-receive 2.3Mb on changing one character in a single source file. Even if it's .1 s, it hurts me to know it is happening :grinning:

Could save a lot of battery life, too.

[dominicbarnes]

I think external source-maps could be a good idea here too, since that won't block the page like a double-sized JS file would.

[stephenmathieson]

yep, external sourcemaps would help

ref: duojs/pack/issues/30