Improve build toolchain

[jsilvestre]

I think the current way to build / dev the app can be drastically improved. Brunch is very nice and all, but I think we should replace it from something like npm+browserify.

[jsilvestre]

So I have finally put something together that is working well.

@bnjbvr and @frankrousseau I think you will want to have a look at it. Tell me more about how easy it is to get started with Tasky: https://github.com/jsilvestre/tasky#development

Magic happens here and here.

I wasn't very comfortable with moving from JS/Coffee stuff to Bash, but it appears, once you move the big commands into their own file, that it's nice, easy to understand for the most part (the JS watching stuff is very WTF, it comes from substack himself). When I write tests, I will move the tests execution to a npm script too.

This new build system allowed me to remove many, many, many files which is my favorite coding activity, so I'm super happy (2,739 additions and 77,350 deletions). Also, all dependencies are explicitly declared in both JS (thank you browserify!) and CSS. That makes me super happy too.

The major drawback is that assets aren't automatically updated when you add a font or a picture. Also, in dev mode, changing JS vendors requires you to edit this and this. It doesn't occur often, so it's barely a drawback.

My work can still be improved, mostly in reusing paths, arguments, etc. so let's consider it a very good first working draft.

Oh, and btw: rebuild is < 500ms, I wouldn't dare showing my work otherwise :)

[jsilvestre]

And I improved the cache invalidation thing a lot, no more bullshit double template files "index.jade" and "index_build.jade", just a modification of the index.js (heck, that jade compilation kills first render speed and memory consumption).

[frankrousseau]

The final API looks very nice. But it makes a lot of scripts. I don't know when I will have time to check all your build process, but it looks like there is a lot of good stuff.

[bnjbvr]

Definitely good ideas and stuff, but indeed that makes a hell lot of scripts. Also, the ultimate goal of such a build toolchain is to avoid having to commit the build itself, how about compiling it on start, if the whole rebuild process takes less than 500ms?

[jsilvestre]

Yes, I followed substack's philosophy which is basically UNIX philosophy: small modules that perform one task each. The obvious downside is that you have to write (and thus maintain) more code. I will look forward to improve maintainability (help needed here :D).

I'm not completely convinced it's better or not than what we have in other Cozy apps. I'm still trying to get around my head, looking for pros and cons, etc. My excitement comes from the fact that I spend a day on it, and I finally got results (after many failures, during other days spent on that matter :p) o/

About the rebuild time: the build time is around 8-10s. What I call rebuild is the action of rebuilding the coffee source into a bundle.js during development. It's just a very small part of the build. This is the difference:

Build

• bundle all coffee client's code and dependencies into a bundle.js file, minified, with source maps
• build all stylus code into an application.css file, adds the vendor's stylesheets, minified, with source maps
• pre-build jade template into javascript. Replace some stuff and append the git version to assets so there is an cache invalidation mechanism.
• convert server's code into JS
• move assets to the proper directory

Watch (dev mode)

• starts the server with nodemon
• copy once assets into client/public
• build once only JS client's dependencies
• watch stylus changes, update accordingly (with source maps)
• watch coffee client's code changes, update accordingly (with source maps) This last step runs under 500ms, often 300-400ms. This is still around 2x brunch performances btw, but it's acceptable.

You can also note that build mode (production) and watch mode (dev) are similar yet different. I had to split client's code and its dependencies to lower the rebuild time, and get decent performance, whereas in production, I want only one minified file.

Also, the ultimate goal of such a build toolchain is to avoid having to commit the build itself

What could be done about the build, is to add a postinstall script which would make Cozy build the app during installation. But I'd rather see Cozy being better at handling builds, because suck trick would involve moving all dependencies into production's dependencies.

I tried to get a pros/cons "this" vs Brunch+Cakefile: Advantages

• it's fully customizable
• my dependencies are clearly declared, so they are more easily manageable
• as a result, I can switch them easily too
• this is "one thing", Brunch+Cakefile is two (we could hide Brunch+Cakefile behind npm scripts too, but that would be 3 things)
• you can't use Browserify with Brunch (I couldn't make the plugin work, and that's awkward anyway).

Downsides

• more customizability always means that you have to be more careful to be coherent between apps
• this is really bound to Tasky's case, but there is more differences between dev template and build template (vendors and app JS are concatenated into one file). So if the build has a bug somewhere, the "prod" version will be broken, but you won't see anything in dev mode
• maintainability of bash scripts can is a PITA, especially since what I did is far from perfect.

[bnjbvr]

What could be done about the build, is to add a postinstall script which would make Cozy build the app during installation. But I'd rather see Cozy being better at handling builds, because suck trick would involve moving all dependencies into production's dependencies.

I feel the opposite way. Having Cozy be better at handling and sharing dependencies would even be better. Plus, having commits that reconstruct an entire file just because of building doesn't really make sense: it adds commits to authors, which aren't their own work, giving them more credit than they're due. Adding dependencies to apps will just increase install / update time, which shouldn't be an issue in most cases (in the future, when web servers and IRC bouncers will be hosted in cozy, that is going to be an issue).

See also https://github.com/bnjbvr/kresus/commit/010e96e4aab9ed209e3764bad48b3faf01c18f02

[jsilvestre]

I think you misunderstood me, because I didn't express myself correctly: I completely agree that there shouldn't be "build" commits. And that the platform should handle those shared dependencies better. But putting dev dependencies in production dependencies is a bad practice, even if it's convenient. About build commits, you don't even have to go as far as the credit attribution, even if you are definitely right with that. Managing built files in Git is a non-sense, it's error-prone (i.e. we sometimes forget it), it causes un-necessary conflicts during merge, it's just bad practice, but it's convenient (too).

All in all I think it's better to have all dependencies in "prod" than to have build commits, so your solution is probably the better one (I'm considering using postinstall for that in Tasky too, your post forced me to make the objective comparison and I reconsidered my position which was "both are bad practice so I won't bother to change"), but neither solution is perfect. That's why I'm willing to see something in Cozy that wouldn't be a bad practice. So my "I'd rather see Cozy being better at handling builds" comment can be anything. It'd be interesting that we have that in mind to seek a very good solution for that. Thanks for the input, as usual it's good stuff that feed my thoughts! I'm going to make the change and we'll see how it goes!

[jsilvestre]

I did it successfully, but I had to remove it because "npm install" was erroring in Cozy's prod, probably due to our npm cache or something. I will investigate that with someone from the IT team.

[frankrousseau]

Some remarks:

• Including build result in the versioned repository doesn't make sense for most of web app deployments. But we don't do tradionial SaaS softwares. In our case, running a build on every machine that runs cozy is a bad practice too.
• The real question is: should we use git as a package manager ?
• Someone who performed a build, performed an action that makes sense for us. It's good to see this person appearing in the commit log.
• About the credit part, a build often comes from someone who at least tested the code, who contributed to it in a more significant way than just running a build command.
• I think @substack is right. Using programming tools instead of simple script tools to perform build commands is overkill.
• Bash does the job well and a lot of people understand it. So the combo npm + bash looks good to solve our problem.
• Brunch fits well with our needs. Despite the fact it doesn't fit with the two previous points, we shouldn't throw it too easily.

[jsilvestre]

So it works well, but it's still slower than Brunch. I'm really looking forward Brunch 1.8, that will bring browserify support, because I can't code without browserify anymore. I'm closing!