Add option to Base#precompile to only build dependency graph

[vjpr]

When using the Mincer server, I like to have an HTML tag generated for each dependency. The problem is that templating engines are synchronous and asset.compile is asynchronous. This means that on first request, the templating engine is unable to render the template with each dependency as a separate tag because Asset#toArray() returns an error.

Workarounds

• Precompile on app load.
  • Problem: slows down server start.
• Serve up a single html tag when !asset.isCompiled, and then on next refresh use expanded tags.
  • Problem: Messy, and it breaks breakpoints.

Solutions

1. Provide an option in the Base#precompile method to only build dependency graphs for assets without actually compiling files.
   • Allow Asset#toArray() to return array of dependencies if this precompile options has been passed in.
2. Provide a synchronous method on Asset to resolve dependencies.

This would allow server to start quickly.

---

Seems the only way to do it would be to add an option to Asset#compile to prevent compilation which then flows through when traversing dep graph in processed.js#resolve_dependencies.

[ixti]

In fact it's not as easy as it sounds. Directives processing is an ordinary preprocessor, that is involved in compilation process. The possible solution is to extract directive processing to build dependencies graph.

But then again, let's say you get dependencies graph at first, then server sent bunch of script tags, then these files needs to be compiled upon request, and you'll get the same delay, but distributed over two stages (before sending html, after browser will start fetching scripts).

[puzrin]

IMHO, the most realistic solution is to speedup full recompile on the start. That needs better dependency graphs from CSS processors (LESS/Stylus). When unchanged assets are properly cached, startup time become acceptable.

[vjpr]

But then again, let's say you get dependencies graph at first, then server sent bunch of script tags, then these files needs to be compiled upon request, and you'll get the same delay, but distributed over two stages (before sending html, after browser will start fetching scripts).

At present, a minimum of two page refreshes are needed. One to kick of the compile process and another one when they are ready.

My development workflow currently involves nodemon which restarts my server on every server-side code change, and livereload in my web browser which refreshes the browser on a backend/frontend file change.

Yes, the delay will be the same, but it will work seamlessly with my development workflow. At the moment I manually refresh the page 2+ times until it loads.

[vjpr]

IMHO, the most realistic solution is to speedup full recompile on the start. That needs better dependency graphs from CSS processors (LESS/Stylus). When unchanged assets are properly cached, startup time become acceptable.

I would always love faster compiles.

However, at the moment my app takes 4s to compile on new 2.7ghz MBP Retina. About 100 files. This means every server-side change means I have to wait 4s before browser refresh.

I would expect pre-building the dependency graph would be extremely fast meaning I could refresh my browser immediately after restarting the server.

Sprockets didn't have to address this problem because everything is synchronous.

[ixti]

At present, a minimum of two page refreshes are needed. One to kick of the compile process and another one when they are ready.

If you are using Base#precompile if front of your requests, you should not meet such problem. Or I'm missing something.

Sprockets didn't have to address this problem because everything is synchronous.

That's true. Unfortunately due to asynchronous nature of Node.JS renderers (like Stylus, LESS) I couldn't have same approach. That's was why I introduced Base#precompile in Mincer. So for example you can use it as middleware server in order to be sure your assets are compiled before got accessed.

[vjpr]

If you are using Base#precompile if front of your requests, you should not meet such problem. Or I'm missing something.

The problem is I want to be able to refresh my browser instantly upon server restart using a tool like LiveReload.

With this change, the server gets the asset requests, and responds when it finishes compiling the assets.

Without this change, I can only refresh my browser 6 seconds after the server restarts or else the server can't render the page because it can't insert the JS and CSS tags for each asset dependency. I find myself continually pressing the refresh button until the assets are compiled which is annoying.

[vjpr]

Just thinking out aloud here on a possible async template workaround. It's a bit hacky.

When rendering dynamic templates on the server we compile our templates twice.

The first time with a mocked version of template helpers - using an approach like in Sinon.JS stubs.

We could then evaluate all the method locals which are used in the template engine (since we know what args they are invoked with by examining our stubs), and wait until they complete using async.series before we call the synchronous template compiler.

And now async helpers are supported.

Performance won't be too bad either because the template can be precompiled into javascript code once using the hamlc.template() method or passing { client: true } to jade.compile().

This requires no changes to Mincer. I'll try it out.

[ixti]

I guess I still don't really understand exact problem. So I'll try livereload on this weekend to better understand the problem. Thanks for your help and interest in Mincer :)) Together we'll make Miner much better :))

[vjpr]

Mincer is awesome - can't thank you guys enough! Our product wouldn't be possible without it.

[vjpr]

So I've managed to implement my idea and it works well. I can now use asynchronous helpers in any template.

This means you can call Asset#compile from a helper and the server will not respond until the compilation is completed.

Now my app compiles itself on first http request, instead of server start.

---

Instead of writing the following in Express:

res.render 'index.haml'

I now use this:

    console.time "Rendering index.html"

    # Pre-render template.
    _ = require 'underscore'
    locals = _.clone res.locals
    stub = sinon.stub locals

    filename = process.cwd() + '/views/index.haml'

    # Compile template method.
    tmpl = hamlc.compile fs.readFileSync filename, 'utf8'

    # 1st run to spy on which methods from locals are required for rendering.
    tmpl locals
    evaluatedLocals = {} # [method name][ordering of call]
    tasks = [] # Async tasks to be run.

    for name, spy of locals
      if spy.called
        # Evaluate local for each time it was called.
        for i in [0..spy.callCount - 1]

          do (name, spy, i) ->

            tasks.push
              name: "#{name}##{i}"
              run: (taskFinished) ->

                done = (err, val) ->
                  return taskFinished err if err
                  evaluatedLocals[name] = {} unless evaluatedLocals[name]?
                  # Store the evaluated method from each call.
                  evaluatedLocals[name][i] = do (val) -> val
                  taskFinished()

                logger.trace "Evaluating #{name}() with args:", spy.args[i]
                # We need a way to check if a method is sync or async.
                # No callback will be called if its sync.
                # We could require all helpers to be async, however the majority of
                # helpers are synchronous, so it should be opt-in from the template.
                val = undefined
                if typeof _.last(spy.args[i]) is 'function'
                  # Template helper is asynchronous.
                  val = res.locals[name] _.initial(spy.args[i])..., done
                else
                  val = res.locals[name] spy.args[i]...
                  done null, val

    # Wait until all locals have been evaluated.
    async.forEach tasks, (task, done) ->

      logger.trace 'Running task', task.name
      task.run done

    , (err) ->

      if err
        logger.error err
        return res.send 500, err

      # Create a new stub that responds to our calls with the evaluated local
      # for the n-th call.
      stubLocals = {}
      for name in _.keys evaluatedLocals
        # Create a closure to share `n` amongst all invocations of each
        # method: `name`. This allows us to pass different args to the same
        # method call.
        do (name) ->
          n = 0
          stubLocals[name] = ->
            logger.trace "Rendered #{name}##{n}", evaluatedLocals[name][n]
            str = evaluatedLocals[name][n]
            ++n
            return str

      html = tmpl stubLocals

      # 2nd run to get html.
      res.send html

      console.timeEnd "Rendering index.html"

Asynchronous helpers must call an empty function as the last argument.

haml-coffee example:

!= @js('application.js', ->)

---

Another simpler solution would be to run Environment#precompile on first http request. Either manually specifying which assets to precompile or using some of the code above to find which assets need precompiling.

---

Shortcomings

• Cannot use an asynchronous helper for a conditional block which contains other asynchronous helpers. Control paths must be known in advance.

Example:

- if @extensionAsync(->)
  != @js('extension.js')

This is not possible without multiple passes.

---

Another alternative is to use a regex to scan source files for required assets in helper tags. We then precompile these assets before rendering the template.

Shortcomings

• All helpers are compiled even if there is a conditional statement that would prevent it from being compiled.
• If asset helpers are dynamically generated (very rare), they will not be precompiled.
• The regex search string must be modified for each type of template syntax. Plain syntaxes like jade might make this problematic.
• Much slower than checking Asset#isCompiled and then Asset#compile on each asset. This could potentially be a bug with precompile. Precompile should check whether isCompiled before compiling.

[vjpr]

Just realised another place I use Asset#compile is in an assetPath helper in Stylus templates. This is necessary for getting the correct digest paths for image urls.

So for async templates to work correctly the Mincer #evaluate methods will need to be changed, or the prototypes monkey-patched. Same with EJS and the others.

[puzrin]

Closed, because precompile no longer needed.