search

h5bp / html5-boilerplate

A professional front-end template for building fast, robust, and adaptable web apps or sites.
https://html5boilerplate.com/
MIT License
56.25k stars 12.19k forks source link

A build script is necessary #16

Closed paulirish closed 13 years ago

paulirish commented 14 years ago

others: http://daverupert.com/2010/06/web-performant-wordpress/

I'm unsure on how this could be added to the project in a general way.. something php/ruby/py/html folks could all use it.

paulirish commented 13 years ago

http://github.com/Schepp/CSS-JS-Booster php http://github.com/cjohansen/juicer/ ruby, no ie6-7 support. http://github.com/mintdigital/asset_hat

Garowetz commented 13 years ago

How about SASS (http://sass-lang.com/) for css, since you're already using it.

Or maybe a post development compressor like the YUI Compressor (http://developer.yahoo.com/yui/compressor/). This will do both js and css

paulirish commented 13 years ago

on filename revving .. query params and caching; http://www.stevesouders.com/blog/2008/08/23/revving-filenames-dont-use-querystring/ http://wiki.squid-cache.org/ConfigExamples/DynamicContent

vvo commented 13 years ago

I can tell by experience that schepp cssjsbooster is the best option if dealing with dynamic scripts merge and minification. It handles data uris, works with either closure compiler, yui compressor, jsmin. Support filename reviving without query params. It now has support for concurrency calls for generating cache. Easy integration either dedicated server or hosted sites. Works only with php.

Garowetz commented 13 years ago

http://github.com/Schepp/CSS-JS-Booster

roblarsen commented 13 years ago

I'm partial to ant (or similar) for this task because it never needs a server or related infrastructure so it's more portable across projects. You can test with and deploy static files even. I've got a roughed out ant script that does the CSS/JS/file revving part. It should be pretty easy to add image optimization to it.

Garowetz commented 13 years ago

Does Ant need to be installed on most web hosts? Is there a stand alone binary/file that can travel with the boilerplate? Or is it generally already there on most good hosts?

roblarsen commented 13 years ago

An Ant based solution takes the power out of the host's hands and puts it in the hands of developers. Ant would need to be installed on any machine that was going to execute the build script. It's a Java app so that can be basically anything. The important part is the build script itself and that's a natural part of a project like this.

I don't think Ant is the only solution for this (it's basically an impossible problem to solve one-size-fits-all,) but it's definitely A solution. For a project with an inclusive scope like this there will likely need to be multiple options for these tasks,

I'll tweak this sample build script I put together a few months ago to work with this project. Hopefully that'll get the ball rolling on this issue.

louisremi commented 13 years ago

The first things we need are minifier APIs available over http. This would make it easier to create minification scripts in any language, without requiring the developers to install any library on their computer/shared host/server. We already have google Closure Compiler, we still miss CSS and HTML minification. What about starting an appEngine for that?

unruthless commented 13 years ago

Another PHP solution: http://code.google.com/p/minify/. Haven't used it, but have heard good things. Seems to be well-documented, too, and the mailing list is pretty active.

louisremi commented 13 years ago

What about adding and ... conditional comments for jQuery and profilers? This way, build scripts could automatically replace local jQuery by one minified and hosted on a CDN. and get rid of development scripts as well.

Garowetz commented 13 years ago

How about we define what we want in a build script, since this is a project of getting everything in one package

Must Haves

Other Thoughts

louisremi commented 13 years ago

I've got a fork of HTML-minifier with an option to switch from development to production mode: http://github.com/lrbabe/html-minifier/commit/1987ecd1cb6abecbf8202c6acb13ee690c4e8cef#L1R612

roblarsen commented 13 years ago

It's not the server side solution people want, but today I quickly put together and Ant build script that does the following:

creates a cleaned up "publish" directory (useful for a build kit, no extraneous files) concats the project JS (my version moves jquery from the CDN to local source) minifies it revs the file name (min-001.js, etc.) minifies the CSS revs the file-name (min-001.css, etc.) adjusts the source of index.html to point to the new files.

with Ant installed and on your path "ant build" in the project directory will make the above magic happen. I'm going to look into the image tasks next.

http://github.com/roblarsen/html5-boilerplate

paulirish commented 13 years ago

@Rob, Totally kickass. This is really nice.

Ant looks like a good place to start this... it is kind of built for this exact thing and many folks already using Java for the yui compressor anyhow.

I think we could use more human-readable flags instead of @@SCRIPTS@@ and @@ENDSCRIPTS@@ but that can be adjusted.

Also obviously this will soon need to handle more than just index.html, we can iterate the html changes over a few files?

On image optimization, the Yahoo Perf link from above has the general techniques that Stoyan used on smush.it.. and then stoyans list of tools and google's page speed image optim deets

@Garowetz, definitely want these things done staticly at build time, instead of it happening on request. However I don't like static files that are pre-gzipped. Freaks me out.

@lrbabe, I'm all about html minification, though I only know of an ASP.NET html minifier and than kangax's.. Using a hosted minifier through an API is fine to me.. Can that work with Ant? I like your prod/dev flag changes. I assume they're done in a way that means the production is commented out in dev and dev variant is in play?

(Somewhat tangential: found this ant script for identifying files based on filesize -- may be good for disabling gzip on files <1k in the future.

@All - thanks for your interest here. I think getting something here that could work for 80% of users will be absolutely HUGE. it's my favorite aspect of boilerplate.

roblarsen commented 13 years ago

@paul, yeah, the flags are definitely for the work-in-progress version. I'll swap for something more sensible now that that bit is working.

We definitely can do multiple HTML files as the replaceregexp task accepts a FileSet argument.

As for images, there's already an ant task for optipng http://github.com/znerd/optipng-ant-task, which gives me a decent place to start. I've looked through Stoyan's notes already, I'll take a look at Google's tonight. Hopefully it won't be too bad putting together the full sweep. Maybe I'll post to exceptional-performance to see if anyone there has done it.

paulirish commented 13 years ago

the task accepts a FileSet argument.

Hot.

Maybe I'll post to exceptional-performance to see if anyone there has done it.

Good call.

Garowetz commented 13 years ago

I like it ... I think I'll be learning ANT now!

roblarsen commented 13 years ago

Hey,

Update. I made the small changes to the index file to make the concat section a bit more friendly.

Also, I've been looking at the image optimization solution... it would be possible to hack something together using existing tools, but it wouldn't be an easy 80% solution. It'd be fine for project work, because you can just say "install X and make sure it's on your path." For this? Not a real solution. There's no equivalent to the YU Compressor jar that I can easily leverage for the task. Instead there are at least 3 or 4 tools, in different languages, that aren't going to be as portable in the way needed by a project like this.

I got crickets from exceptional-performance, as well.

I'll ask around to see if there's some way to smooth out the rough edges that lie outside my limited Ant experience. Maybe a web service? I know it's possible to post from Ant. I've just never done it.

What I'd really love to see is one, java-based tool we could compile and ship with the project.

I can also look at putting in some HTML minimization in. I might have more luck there.

Garowetz commented 13 years ago

Is there anyway would could implement the suggestion that Irbabe had above about production and not production. Might be useful for people moving from local to server.

Also do we think that a better more concise tag is better for identifying the script section. I like what was there before because it was smaller. You could still leave in the descriptive text letting users know what the section is for, but it would allow it to be smaller like we're trying to accomplish in the stripped version.

On the Image optimizing...

Could we include the binary's of some of the tools with the project? Would there be some way of distinguishing the operating system that the build script is on?

roblarsen commented 13 years ago

I can implement the production/non-production thing, at least as I understand the intent.

If you look at the output, the descriptive comments are actually completely stripped out by the build script, even without HTML minification. That said, I can make it read whatever folks want it to read. It's very flexible, as long as the first few characters are suitably unique. I'm happy to defer to Paul on the wording as the comments are in his editorial voice.

I can include some binaries and there might be some way to differentiate what OS is involved, but there are still way too many variable for me to feel like that's a solution that will scale to a bunch of different machines. I'll hit up some Ant geniuses to see if there's some technique I'm just ignorant to that'll solve this.

Garowetz commented 13 years ago

Yep I missed that those were stripped out!

paulirish commented 13 years ago

Thanks for the update Rob.

Is there anyway would could implement the suggestion that Irbabe had above about production and not production.

I really want to avoid all these flags over the markup that people need to understand in order to get their javascript served correctly. I think we can do this with the build process being a touch smarter.

Could we include the binary's of some of the tools with the project?

Yes i have no problems with that. I think a /build/ folder is in order and would do all this. I expect the build script to only be run on the webhost, so we're talking linux only. Running the build process locally is just ... bound for problems and so supporting win/mac for the build process just doesn't seem worthwhile to me. Any opposing views there?

These Ant intros are mostly for me, but for anyone else ... : http://www.delicious.com/popular/ant http://www.exubero.com/ant/antintro-s5.html http://www.developer.com/java/j2me/article.php/989631/Building-with-Ant-Introduction.htm

roblarsen commented 13 years ago

if it's solely for linux, then I can more easily put something together.

Personally, I'd like to support win/mac local builds, (the above build should just work on any machine with Ant installed) but without some magic, I'm not sure that's going to be possible with the image optimization tasks included (at least in the short term.)

I'll take a look at a linux solution for the full build and then work in scaled down local targets (a "text" target or something that would just do the text optimizations.)

paulirish commented 13 years ago

I'd like to support win/mac local builds

That's probably the best move, yeah. .. What i'm trying to avoid is people who think all this is done locally somehow. I keep seeing stuff like "minimizing your javascript is good, but make sure to keep backups of your code." ugh. And thus, I really think the build process should happen on a completely different machine so people aren't confusing the build process with what is (or, rather, should be) in source control.

I spotted http://trimage.org/ today which apparently has a commandline API and handles png/jpg. Might be a good project to leverage.

roblarsen commented 13 years ago

I was looking at it and, moving forward, there is actually a hook in ant to fork tasks based on the OS, so we can create a hodgepodge of binaries to take care of this task on multiple systems.

Looking at trimage now. Hopefully it'll shake out as it would be great to have one tool to do the compression taks on both pngs and jpgs.

roblarsen commented 13 years ago

trimage needs a gui, apparently.

Skipping that.

Checked in a version that runs optipng on /images/* It's SLOW. Definitely going to create a couple of target options- one for just text (which is fast and would likely need to happen more often just to test minification vs. breaking scripts) and then one for everything.

Gonna look at some jpeg stuff now. Once I get jpegs crunched, what's left for a serviceable version of this?

roblarsen commented 13 years ago

got distracted and enabled win (and mac? can't test- it should just work with optipng on the path) support for the optipng step. It's distributable, so I included the binary in the project. I'm going to find some time tonight/tomorrow to do the jpeg thing.

roblarsen commented 13 years ago

As long as libjpg (and jpegtran) are installed on your unix/linux/mac system, the PNG and JPG tasks are now working.

http://github.com/roblarsen/html5-boilerplate

I've got two basic targets "build" which does EVERYTHING (including images) and "text" which just works on the text components.

Take a look and let me know what you think. In addition to any feedback, I'll add some documentation to the build file so that people can tweak as needed.

paulirish commented 13 years ago

Oh this is good fun! I've integrated your work into a branch (build) on my repo.

I moved build.xml and all of tools into a build/ folder, too. In addition to that I made some more tweaks: http://github.com/paulirish/html5-boilerplate/compare/b7315160...c43f559

But in general I really like where this is going.

roblarsen commented 13 years ago

Stripping comments is a good start. HTML minification scares me a little bit, but dropping comments doesn't seem as much like a grenade with the pin pulled.

Let me know what else you might want to see it do.

Garowetz commented 13 years ago

This does look great ... few thoughts

(yes I know, I make suggestions when I could just commit it myself. I'm just so pressed for time right now, but I will get into it when I can)

roblarsen commented 13 years ago

The segmentation exists already, although it could be improved. For example, right now, you can run "ant scripts" and it will copy the files over, concatenate them, rename them, minify them and change the source reference without running any of the other targets. I really just need to look at the dependencies to make sure each valuable target works as expected by itself.

A readme would be great and some more documentation in the build file would definitely be good.

paulirish commented 13 years ago

I added some crude ability to switch to the minified jquery, if you're currently using the unminified.. http://github.com/paulirish/html5-boilerplate/commits/build And also a feature in the minification where <!--! comments --> are retained, but the rest are dumped.

We can also look at http://code.google.com/p/htmlcompressor/ which kangax calls the best option at the moment. I'm just not sure how we can use his js-based one feasibly. Sad.

Rob, you can pull in from my build branch if you wanna tweak any more or whatever.

I guess the thing that is most pressing is the need to apply this to a set of .html files. I kinda did that down in the htmlmin section but that direction doesnt look very DRY. I also think about when people add a few jQuery plugins (or whatever 3rd party scripts) that require extra CSS.. How we fold in the new js and css files into the concat/min process here. Ideas?

Garowetz commented 13 years ago

What do you mean by "js-based"?

This htmlcompressor looks like it'll do everything we want. And even suggests ANT as an option. It includes YUI too! Still doesn't do the image optimizations but we've got that. I like it.

I take it we're not going to be able to dictate how users should include their js and css files are we.

roblarsen commented 13 years ago

Anywhere we need to do multiple file replace, the pattern looks like this:

<replace dir="${src}" value="wombat">
<include name="*/.html"/>
<replacetoken><![CDATA[multi line
token]]></replacetoken>
</replace>

exclude works there as well, so it's pretty straightforward.

@paul If I have time before the weekend (I'm out of town Friday/Saturday) I'll look at your branch and try to add some documentation, check out htmlcompressor, etc. If not, I can tackle whatever's left when I get back.

paulirish commented 13 years ago

What do you mean by "js-based"?

I meant kangax's html minifier. http://kangax.github.com/html-minifier/

Also yeah it looks like the htmlcompressor can run YUI on inline script/style which may be all we want it for.. I'm still a little wary about full whitespace removal. :)

I take it we're not going to be able to dictate how users should include their js and css files are we.

Not exactly, but if they would like to have 8 js files included and have the build process smush em all together, they have to follow a few guidelines.

paulirish commented 13 years ago

Just spotted this.. a rake port of the build script: http://github.com/avdgaag/html5-boilerplate/commits/rake

Very nice. Will be great to have this as an option for all the rubyists.

roblarsen commented 13 years ago

Very cool.

mikkotikkanen commented 13 years ago

Coolness.

What I would like to see is configurable ant script, in order to have it compatible with different types of file/folder structures and/or names. Which is bound to happen. Luckily this is easily done by using hefty set of properties.

The reason for this is that people will have, and will want to have, different configurations with their systems because of different requirements. For example, having single or even two script files, or even in one single folder, is just not feasible for larger projects, thus one should be able to easy to configure.

For example, I am using automated js-test-driver unittesting and scripts folder having java-like folder (/packet) structure, in order to handle all the files for our framework. This framework is then concatenated and minified upon publish.

Also, the order the files are concatenated together should be taken into account. Many JS frameworks offer possibilities to build your own framework libraries (for smaller filesizes), thus library files should be before the user written files in the final concatenated file.

Personally, I do this in the way that you define the library fileset (usually just scripts/libs/_/_) separately from scripts fileset (usually scripts/libs//**, excluding scripts/libs). These two are first concatenated together as two files, which are then concatenated together, with libs-concat being first.

When these would be defined in the "head" of the build script, the configuration and customization for each need would be a breeze.

mikkotikkanen commented 13 years ago

Oh, would like to see the compressor to also be able to handle php files.

paulirish commented 13 years ago

mikko. thanks a bunch for your interest. would you be interested in forking and adding in support for this sort of configuration?

The most up to date build stuff is at http://github.com/paulirish/html5-boilerplate/commits/build

I agree with you on the flexibility, for sure.

mikkotikkanen commented 13 years ago

I will check it out if I could incorporate parts from our build script.

Oh, and also, it needs to ignore .svn folders.

paulirish commented 13 years ago

mikko, this commit from garowetz probably makes sense to integrate while you're pumping up the build.properties changes: http://github.com/Garowetz/html5-boilerplate/commit/ff3d10eaff956

paulirish commented 13 years ago

I've integrated garowetz's best practices commit (a bit sloppily, sorry!) and added a few build options based on your intended output..

Also updated htmlcompressor which has a bugfix coming directly from a boilerplate user! http://code.google.com/p/htmlcompressor/issues/detail?id=16

newest is on build branch http://github.com/paulirish/html5-boilerplate/commits/build

mikko, can you build out your configuration work from here?

paulirish commented 13 years ago

I've also added a rough wiki page for documentation that we can build out: http://github.com/paulirish/html5-boilerplate/wiki/Build-script

mikkotikkanen commented 13 years ago

Paul, I'll check it out when I get a bit time for it. :)

Quite swamped at the moment (it's 23:50 in here and I'm still working, started 9:00 at the morning) with some architecture changes to our framework and development processes. Need to get this stuff done before it is needed in development.

mikkotikkanen commented 13 years ago

Can we reserve the publish/build etc. directories to ourselves or should we think of different folder structure? People might want to have build/publish etc. directories on their site- Either group all boilerplate folders under one folder (/boilerplate) or have the actual site living in "application/" folder, this way we could have "build/", "publish/" and even "demo/" sitting right next to it, without breaking anything.

roblarsen commented 13 years ago

I think there's a danger of overthinking the way this might be used.

I mean, I've already used boilerplate on a project and the first thing I did when I exported was change the folder structure for static assets to match the way I prefer it (js, images and css as a subfolder of a new folder called _assets.) I don't need Paul's master to change to suit the way I'm going to use it.

Especially since I'm using it to just create a transitory set of templates that will be passed off to some back-end guys to get wrapped into an application. All of the directories outside those referenced in the HTML could have gibberish names in my case since, at some point soon, my boiler plate directory will be left behind.

I think we should just provide a smart, common sense default and then document the hell out of the ways to customize it (a task I'm hoping to work on this weekend.)

mikkotikkanen commented 13 years ago

Well, the thing is, now that it has the js, css, html and image optimizations, it probably will be used as a base for web-pages: Person X would take the boilerplate and build their site off from that, so that the whole site can utilize the optimizations scripts.