Create build script to reduce browser footprint

njoyard commented 8 years ago

So as discussed before, the point is to reduce browser footprint by assembling stylesheets together and javascript sources together.

I have a working solution using a Makefile (see below). It was hacked in a few minutes, basically it generates a Depends.mk file to track dependencies (from <script> and <link> tags in index.html) and then:

a built/main.css by concatenating all stylesheets referenced by a <link> tag in index.html
a built/main.js by concatenating all javascript sources referenced by a <script> tag in index.html
an index.prod.html produced by removing all <link> and <script> tags and inserting new ones that link to the 2 files above (as pointed before, we can keep those and just comment them).

(Note: the Makefile actually only works with <script src> and <link rel="stylesheet"> tags, ignoring the others, as one would expect.)

The only problem is, given the routing in the project (based on symlinks to index.html) you have to actually swap index.prod.html and index.html to use it. That's good for development, but it may be confusing (especially because the Makefile actually uses index.html to generate things, so if you call it again after having swapped the files, it will most likely produce garbage).

An other solution would be to make it generate things in some other directory (but then we'd have to copy other dependencies such as scripts loaded from templates, too).

What do you think @fmassot @RouxRC ?

Makefile below for reference (it was actually built before the latest file tree rework, but it's easy to adapt).

DOCROOT := public
BUILTDIR := built

INDEX := $(DOCROOT)/index.html
ALLCSS := $(wildcard $(DOCROOT)/css/*.css) $(wildcard $(DOCROOT)/fonts/*.css)
ALLJS := $(wildcard $(DOCROOT)/js/*.js) $(wildcard $(DOCROOT)/js/*/*.js) $(wildcard $(DOCROOT)/js/*/*/*.js) $(wildcard $(DOCROOT)/js/*/*/*/*.js)

PRODINDEX := $(DOCROOT)/index.prod.html
PRODJS := $(DOCROOT)/$(BUILTDIR)/main.js
PRODCSS := $(DOCROOT)/$(BUILTDIR)/main.css

all: $(PRODINDEX) $(PRODJS) $(PRODCSS)

include Depends.mk

Depends.mk: $(INDEX) $(ALLCSS) $(ALLJS)
    grep '<script type="text/javascript" src' $(INDEX) | sed -r 's;.*src="([^"]+)".*;$(PRODINDEX): $(DOCROOT)/\1;' > Depends.mk
    grep '<script type="text/javascript" src' $(INDEX) | sed -r 's;.*src="([^"]+)".*;$(PRODJS): $(DOCROOT)/\1;' >> Depends.mk
    grep '<link rel="stylesheet" href' $(INDEX) | sed -r 's;.*href="([^"]+)".*;$(PRODINDEX): $(DOCROOT)/\1;' >> Depends.mk
    grep '<link rel="stylesheet" href' $(INDEX) | sed -r 's;.*href="([^"]+)".*;$(PRODCSS): $(DOCROOT)/\1;' >> Depends.mk

$(PRODINDEX): $(INDEX)
    cat $< \
    | sed -r '0,/<script type="text\/javascript" src[^>]+><\/script>/s//<script src="$(BUILTDIR)\/main.js"><\/script>/' \
    | sed -r '0,/<link rel="stylesheet" href[^>]+>/s//<link href="$(BUILTDIR)\/main.css" rel="stylesheet">/' \
    | sed -r 's/<script type="text\/javascript" src[^>]+><\/script>//' \
    | sed -r 's/<link rel="stylesheet" href[^>]+>//' \
    > $@

$(PRODJS):
    mkdir -p $$(dirname $@)
    cat $^ > $@

$(PRODCSS):
    mkdir -p $$(dirname $@)
    cat $^ > $@

@PHONY: clean
clean:
    rm -f Depends.mk $(PRODINDEX) $(PRODJS) $(PRODCSS)
    [ -d $(DOCROOT)/$(BUILTDIR) ] && rmdir $(DOCROOT)/$(BUILTDIR)

fmassot commented 8 years ago

Seems perfect to me.

I found the line ALLJS ugly but I'm not used to code makefile :)

One last thing, I noticed that not appending the google-diff lib introduced a js error. I did not investigate but you may have this issue.

njoyard commented 8 years ago

There's no recursive wildcard in make afaik, unfortunately. Maybe $(shell find ...) would work better. I'll have a look at that google diff issue, though.

njoyard commented 8 years ago

Implemented. To make dev/prod switching easier, I moved index.html to index.dev.html, and made index.html a link to it.

"make" will create the following files:

public/prod/main.js - concatenated scripts
public/prod/main.css - concatenated stylesheets
public/index.prod.html - index that uses the above instead of linking to multiple files

It will also make index.prod.html the "active" index by linking index.html to it. Also note that if no public/config.js file exists when calling "make", it will be copied from public/config.js.example.

"make clean" will remove what "make" created and make public/index.dev.html active again.

(this info is in the Makefile for future reference)

boogheta commented 8 years ago

This should all be detailed in the README (and mention that one in prod HAS to re-make every time the config.js is changed) The method adopted seems the best to me, this way we can always dev without needing to "make" anything, just a pull in prod requires to build consequently. The only thing I remain hesitant about is the fact that when we build the prod, we then have a repository in a situation of having a modifed file whereas this is the normal situation, so I'm wondering whether it would make sense for this rare specific case to use git update-index --assume-unchanged index.html

boogheta commented 8 years ago

Also considering the size of the resulting prod/main.js we might want to consider adding to the prod directory an htaccess forcing apache to serve gzipped data as we do here : https://github.com/regardscitoyens/the-law-factory-parser/blob/master/data/.htaccess

njoyard commented 8 years ago

I will update the README, indeed.

As for the git situation, it bothers me as well but it seems like usual practice (that's why you have a make distclean target in many source trees, to put the tree back in repo-pushable state - and that's why I made a clean target). Using --assume-unchanged is only temporary as it may be undone as soon as you pull again (not 100% sure about that, though). ~~Maybe an other solution would be to add and "DirectoryIndex index.prod.html index.html" directive to apache config and get rid of that symlink mess?~~ (scratch that, I forgot other html links)

boogheta commented 8 years ago

I don't really know --assume-unchanged well either, not sure you can "push" it indeed, that's the only track I found while browsing for some kind of solution

Otherwise I wonder whether it wouldn't be a better solution to remove index.html from the repository, and add a make dev that would just do the ln -s index.dev.html index.html. The only issue with this is that the code won't work as such when being pulled the first time but I don't feel like this is a huge requirement as long as the README says so.

@fmassot may have another idea?

PS: Another thing is I feel like having bash commands in the makefile makes it unusable under windows? Don't really care personally and I don't believe any dev on this project is under windows, but I just felt like it should be said before being forgotten ;)

njoyard commented 8 years ago

I know this is probably a small issue, but anyway, there's really no way of making those operations portable without depending on 3rd party tools (eg. grunt => node, etc). At least here we're only depending on standard *nix tools. You may be able to run that on windows with cygwin or some other gnu coreutils distro, but there's nothing standard in windows that allows scripting text replaces AFAIK (we don't even have telnet by default since w7, for ....'s sake). At least we support OSX here (and probably Android w/ busybox... yay).

boogheta commented 8 years ago

no I agree, this really only was to have said it then forget about it :)

njoyard commented 8 years ago

Re. the index/dev/prod thingy maybe we could also build every prod file in a separate directory and have the webserver serve that directory instead. This way the dev directory would be left untouched and we could just put the build dir in .gitignore?

boogheta commented 8 years ago

That's the last option indeed, I was considering it, then thought it would make the build part a bit longer having so many files to copy, but it has the advantage of maintaining the prod really static until any rebuild. I'm rather agnostic here, we need a third mind @fmassot ! ;)

(note that in this case it would be best that the build happens in a temporary directory that would only be renamed as build at the end so that a rebuild won't temporaily break the prod)

njoyard commented 8 years ago

Updated README in https://github.com/regardscitoyens/the-law-factory/commit/cd092d5b852067667acf9105f8efc6b9fc8408a8.

fmassot commented 8 years ago

I also prefer to have a separate build directory to leave dev directory untouched. And building it first in a tmp dir then mov in build sounds good too.

boogheta commented 8 years ago

It doesn't look like we ended this discussion but it seems that everybody agrees on building in built and at the end replace build with the content of built, I guess that's only a slight add to the Makefile to make and we can close this one?

njoyard commented 8 years ago

agreed.

njoyard commented 8 years ago

Done in 223b8a7. Added a separate make install to kind of confirm that "yes, I want to blow up my prod right now".

regardscitoyens / the-law-factory

Create build script to reduce browser footprint #148