Expose browser usage to node.js

hhsnopek commented 10 years ago

Currently I cannot access any of the CLI modules due to the use of commander.js; Could you expose these options for node.js?

All these need to return as functions, not a file

Multiple files into one js function
multiple files with multiple/corrosponding functions
include/exclude headers (in which case user adds toffee.js to their scripts)

samccone commented 10 years ago

+1

malgorithms commented 10 years ago

hey guys, can you be more specific about how you'd use this? Like show me some sample code of an ideal call you'd make to toffee, and then how you'd use the result of that call?

hhsnopek commented 10 years ago

The developer/user that uses accord writes all the content to a file, we don't want to touch file writing at all in accord. Note: the examples assume the user is already writing the file to their choice, according to their file structure.

file structure (for all examples):

.
├── app.coffee
├── assets
│   ├── css
│   ├── favicon.ico
│   ├── img
│   ├── js
│   │   ├── main.coffee
│   │   ├── templates
│   │   │   ├── picture.toffee
│   │   │   └── post.toffee
├── package.json
├── readme.md
└── views

Client compile a string

example usage with accord:

fs = require 'fs'
accord = require 'accord'
toffee = accord.load('toffee)'

toffee.clientCompile(```
{#
  for supply in supplies {:<li>#{supply}</li>:}
#}
```, options)
  .catch(console.error.bind(console))
  .done (res) -> console.log.bind(console)

Client compile a file

example usage with accord:

fs = require 'fs'
accord = require 'accord'
toffee = accord.load('toffee)'

toffee.clientFileCompile('assets/templates/picture.toffee', options)
  .catch(console.error.bind(console))
  .done( (res) -> fs.writeFile('./public/js/templates/picture.js', res)

Multiple files into one js functions

Compile all the toffee files within templates to a single templates.js (or another name) or prepended into main.js

example usage with accord:

fs = require 'fs'
accord = require 'accord'
toffee = accord.load('toffee)'

toffee.clientFileCompile('assets/templates', options)
  .catch(console.error.bind(console))
  .done( (res) -> fs.writeFile('./public/js/templates.js', res)

Multiple files into multiple js functions

compiler all the toffee files within templates into their corresponding js files Excluding this one because the user can just use the method clientFileCompile in a loop

Ability to include or exclude the toffee headers

Thus allowing the user to have toffee.js load separately from the templates themselves

malgorithms commented 10 years ago

@HHSnopek - in each of these cases, you're just console.logging "res", so I can't tell what res is supposed to be. (You mention a function, but I don't know what that function is expected to take or return for parameters.) For example:

multiple files into one JS function

toffee.clientFileCompile('assets/templates', options)
  .catch(console.error.bind(console))
  .done (res) -> console.log(res.toString())

Can you show me some example calls to this res, so I can see how it's used?

Also, is this a correct summary of what you're trying to do: you want accord.load('toffee') to return some object which has clientCompile, clientFileCompile, etc., defined. So what you need to do is write each of these functions using toffee's standard exports?

hhsnopek commented 10 years ago

the res is the function that is returned form the toffee engine itself;

toffee.clientFileCompile('assets/templates', options)
  .catch(console.error.bind(console)
  .done( (res) -> fs.writeFile('./public/js/templates', res)

This would produce:

.
public
├── js 
│   └── templates.js

clientCompile, clientFileCompile are all pre-written (jade, ejs, handlebars, etc) in accord so that the dev/user can have universal access to each without having a separate way of calling each of these(according to their docs)

Meaning toffee's standard exports can have it's own convention on naming these functions, but for our purpose well change them to the listed prior. All I need toffee's engine to do is be itself, but expose it's cmdline options of -o ,-d, -n allowing node users to compile client-side templates :smile:

Note: I've edited the previous post to reflect writing to files by the user

malgorithms commented 10 years ago

the res is the function that is returned form the toffee engine itself;

but in your example (where, actually, you're using it as a string):

toffee.clientFileCompile('assets/templates', options)
  .catch(console.error.bind(console))
  .done( (res) -> fs.writeFile('./public/js/templates.js', res)

some questions:

is res a string or a function? If it's a function, what does using it look like when it's the compilation of an entire directory? can you show me a couple calls of res() in that case?
is it only something that's supposed to be used in the browser?

hhsnopek commented 10 years ago

right now I can call

toffee.compile(```
{#
  for supply in supplies {:<li>#{supply}</li>:}
#}
```, options
).catch(console.error.bind(console))
.done( (res) -> fs.writeFile('./public/js/templates.js', res)

templates.js will look like this:

function (x) {
      return v.run(x);
    }

What I'd like it to do when you call clientCompile that it will look similar to this:

toffee.compileClient(```
{#
  for supply in supplies {:<li>#{supply}</li>:}
#}
```, options
).catch(console.error.bind(console))
.done( (res) -> fs.writeFile('./public/js/templates.js', res)

template.js looks like:

var toffee;("undefined"==typeof toffee||null===toffee)&&(toffee={}),toffee.templates||(toffee.templates={}),toffee.states={TOFFEE:1,COFFEE:2},toffee.__json=function(e,t){return null==t?"null":""+JSON.stringify(t).replace(/</g,"\\u003C").replace(/>/g,"\\u003E").replace(/&/g,"\\u0026")},toffee.__raw=function(e,t){return t},toffee.__html=function(e,t){return(""+t).replace(/&/g,"&amp;").replace(/</g,"&lt;").replace(/>/g,"&gt;").replace(/"/g,"&quot;")},toffee.__escape=function(e,t){var f;return f=null!=e.__toffee.autoEscape?e.__toffee.autoEscape:!0,f?void 0===t?"":null!=t&&"object"==typeof t?e.json(t):e.html(t):t},toffee.__augmentLocals=function(e,t){var f,n;return f=e,n=f.__toffee={out:[]},null==f.print&&(f.print=function(e){return toffee.__print(f,e)}),null==f.json&&(f.json=function(e){return toffee.__json(f,e)}),null==f.raw&&(f.raw=function(e){return toffee.__raw(f,e)}),null==f.html&&(f.html=function(e){return toffee.__html(f,e)}),null==f.escape&&(f.escape=function(e){return toffee.__escape(f,e)}),null==f.partial&&(f.partial=function(e,n){return toffee.__partial(toffee.templates[""+t],f,e,n)}),null==f.snippet&&(f.snippet=function(e,n){return toffee.__snippet(toffee.templates[""+t],f,e,n)}),null==f.load&&(f.load=function(e,n){return toffee.__load(toffee.templates[""+t],f,e,n)}),n.print=f.print,n.json=f.json,n.raw=f.raw,n.html=f.html,n.escape=f.escape,n.partial=f.partial,n.snippet=f.snippet,n.load=f.load},toffee.__print=function(e,t){return e.__toffee.state===toffee.states.COFFEE?(e.__toffee.out.push(t),""):""+t},toffee.__normalize=function(e){var t,f,n,o,l;if(null==e||"/"===e)return e;for(n=e.split("/"),t=[],n[0]&&t.push(""),o=0,l=n.length;l>o;o++)f=n[o],".."===f?t.length>1?t.pop():t.push(f):"."!==f&&t.push(f);return e=t.join("/"),e||(e="/"),e},toffee.__partial=function(e,t,f,n){return f=toffee.__normalize(e.bundlePath+"/../"+f),toffee.__inlineInclude(f,n,t)},toffee.__snippet=function(e,t,f,n){return f=toffee.__normalize(e.bundlePath+"/../"+f),n=null!=n?n:{},n.__toffee=n.__toffee||{},n.__toffee.noInheritance=!0,toffee.__inlineInclude(f,n,t)},toffee.__load=function(e,t,f,n){return f=toffee.__normalize(e.bundlePath+"/../"+f),n=null!=n?n:{},n.__toffee=n.__toffee||{},n.__toffee.repress=!0,toffee.__inlineInclude(f,n,t)},toffee.__inlineInclude=function(e,t,f){var n,o,l,r,u,_,a,i,p;for(o=t||{},o.passback={},o.__toffee=o.__toffee||{},r={},i=["passback","load","print","partial","snippet","layout","__toffee","postProcess"],_=0,a=i.length;a>_;_++)n=i[_],r[n]=!0;if(!o.__toffee.noInheritance)for(n in f)u=f[n],null==(null!=t?t[n]:void 0)&&null==r[n]&&(o[n]=u);if(toffee.templates[e]){l=toffee.templates[e].pub(o),p=o.passback;for(n in p)u=p[n],f[n]=u;return l}return"Inline toffee include: Could not find "+e};
;

;
(function() {
  var tmpl;

  tmpl = toffee.templates["/basic.toffee"] = {
    bundlePath: "/basic.toffee"
  };

  tmpl.render = tmpl.pub = function(__locals) {
    var supply, __repress, _i, _len, _ln, _ref, _to, _ts;
    __locals = __locals || {};
    __repress = (_ref = __locals.__toffee) != null ? _ref.repress : void 0;
    _to = function(x) {
      return __locals.__toffee.out.push(x);
    };
    _ln = function(x) {
      return __locals.__toffee.lineno = x;
    };
    _ts = function(x) {
      return __locals.__toffee.state = x;
    };
    toffee.__augmentLocals(__locals, "/basic.toffee");
    with (__locals) {;
    __toffee.out = [];
    _ts(1);
    _ts(2);
    for (_i = 0, _len = supplies.length; _i < _len; _i++) {
      supply = supplies[_i];
      _ts(1);
      _ts(1);
      _ln(2);
      _to("<li>");
      _to("" + (supply != null ? escape(supply) : ''));
      _to("</li>");
      _ts(2);
    }
    __toffee.res = __toffee.out.join("");
    if (typeof postProcess !== "undefined" && postProcess !== null) {
      __toffee.res = postProcess(__toffee.res);
    }
    if (!__repress) {
      return __toffee.res;
    } else {
      return "";
    }
  };

  true; } /* closing JS 'with' */ ;

  if (typeof __toffee_run_input !== "undefined" && __toffee_run_input !== null) {
    return tmpl.pub(__toffee_run_input);
  }

}).call(this);

It would be nice to have options.header: boolean to include or exclude the toffee headers

malgorithms commented 10 years ago

ok, so what you're asking for is a clientCompile which returns a string? That string is a big piece of JavaScript, which, when evaluated, defines a variable toffee, which is a dictionary of templates?

So in theory, you could do this in Node, and it would work?

clientFileCompile('assets/templates', options)
  .catch(console.error.bind(console))
  .done( (res) -> 
     # hypoethically speaking, since res is a string....
     eval res # this defines toffee
     some_html = toffee.templates["/bleah/foo.toffee"].render {name:'chris'}

malgorithms commented 10 years ago

to be clear, is the only use case of clientFileCompile that it's a string which is going to be served up in the browser, in a <script> tag, or is "res" something which could ever be used itself in a node program? you mentioned it's a function earlier, I think.

hhsnopek commented 10 years ago

Sorry, we took out the possibility of compiling a folder of toffee files into their corresponding js files, more simplistic to just have the dev/user to loop through all the files in the folder themselves

accord = require 'accord'
toffee = accord.load('toffee') # this line loads the toffee methods (render, renderFile, compile, compileFile, clientCompile, clientFileCompile)

# Compile all the templates into a single js file!
toffee.clientFileCompile('assets/templates', options)
  .catch(console.error.bind(console))
  .done( (res) -> # write file here with res.toString() )

Overall I need one method exposed (called whatever you'd like) that will compile a string into a client-side string that includes(or excludes if option.header is false) the toffee header

Thus allowing dev/user to serve templates.js within a script tag, then able to render or do that want with the template on the client-side.

TLDR; All accord is doing is giving them the template back as a string, with or without the headers

I need: one new method that will compile a client-side template, and a new option that will allow the dev/us to include or exclude the toffee header

Example of template.js with toffee header:

var toffee;("undefined"==typeof toffee||null===toffee)&&(toffee={}),toffee.templates||(toffee.templates={}),toffee.states={TOFFEE:1,COFFEE:2},toffee.__json=function(e,t){return null==t?"null":""+JSON.stringify(t).replace(/</g,"\\u003C").replace(/>/g,"\\u003E").replace(/&/g,"\\u0026")},toffee.__raw=function(e,t){return t},toffee.__html=function(e,t){return(""+t).replace(/&/g,"&amp;").replace(/</g,"&lt;").replace(/>/g,"&gt;").replace(/"/g,"&quot;")},toffee.__escape=function(e,t){var f;return f=null!=e.__toffee.autoEscape?e.__toffee.autoEscape:!0,f?void 0===t?"":null!=t&&"object"==typeof t?e.json(t):e.html(t):t},toffee.__augmentLocals=function(e,t){var f,n;return f=e,n=f.__toffee={out:[]},null==f.print&&(f.print=function(e){return toffee.__print(f,e)}),null==f.json&&(f.json=function(e){return toffee.__json(f,e)}),null==f.raw&&(f.raw=function(e){return toffee.__raw(f,e)}),null==f.html&&(f.html=function(e){return toffee.__html(f,e)}),null==f.escape&&(f.escape=function(e){return toffee.__escape(f,e)}),null==f.partial&&(f.partial=function(e,n){return toffee.__partial(toffee.templates[""+t],f,e,n)}),null==f.snippet&&(f.snippet=function(e,n){return toffee.__snippet(toffee.templates[""+t],f,e,n)}),null==f.load&&(f.load=function(e,n){return toffee.__load(toffee.templates[""+t],f,e,n)}),n.print=f.print,n.json=f.json,n.raw=f.raw,n.html=f.html,n.escape=f.escape,n.partial=f.partial,n.snippet=f.snippet,n.load=f.load},toffee.__print=function(e,t){return e.__toffee.state===toffee.states.COFFEE?(e.__toffee.out.push(t),""):""+t},toffee.__normalize=function(e){var t,f,n,o,l;if(null==e||"/"===e)return e;for(n=e.split("/"),t=[],n[0]&&t.push(""),o=0,l=n.length;l>o;o++)f=n[o],".."===f?t.length>1?t.pop():t.push(f):"."!==f&&t.push(f);return e=t.join("/"),e||(e="/"),e},toffee.__partial=function(e,t,f,n){return f=toffee.__normalize(e.bundlePath+"/../"+f),toffee.__inlineInclude(f,n,t)},toffee.__snippet=function(e,t,f,n){return f=toffee.__normalize(e.bundlePath+"/../"+f),n=null!=n?n:{},n.__toffee=n.__toffee||{},n.__toffee.noInheritance=!0,toffee.__inlineInclude(f,n,t)},toffee.__load=function(e,t,f,n){return f=toffee.__normalize(e.bundlePath+"/../"+f),n=null!=n?n:{},n.__toffee=n.__toffee||{},n.__toffee.repress=!0,toffee.__inlineInclude(f,n,t)},toffee.__inlineInclude=function(e,t,f){var n,o,l,r,u,_,a,i,p;for(o=t||{},o.passback={},o.__toffee=o.__toffee||{},r={},i=["passback","load","print","partial","snippet","layout","__toffee","postProcess"],_=0,a=i.length;a>_;_++)n=i[_],r[n]=!0;if(!o.__toffee.noInheritance)for(n in f)u=f[n],null==(null!=t?t[n]:void 0)&&null==r[n]&&(o[n]=u);if(toffee.templates[e]){l=toffee.templates[e].pub(o),p=o.passback;for(n in p)u=p[n],f[n]=u;return l}return"Inline toffee include: Could not find "+e};
;

;
(function() {
  var tmpl;

  tmpl = toffee.templates["/basic.toffee"] = {
    bundlePath: "/basic.toffee"
  };

  tmpl.render = tmpl.pub = function(__locals) {
    var supply, __repress, _i, _len, _ln, _ref, _to, _ts;
    __locals = __locals || {};
    __repress = (_ref = __locals.__toffee) != null ? _ref.repress : void 0;
    _to = function(x) {
      return __locals.__toffee.out.push(x);
    };
    _ln = function(x) {
      return __locals.__toffee.lineno = x;
    };
    _ts = function(x) {
      return __locals.__toffee.state = x;
    };
    toffee.__augmentLocals(__locals, "/basic.toffee");
    with (__locals) {;
    __toffee.out = [];
    _ts(1);
    _ts(2);
    for (_i = 0, _len = supplies.length; _i < _len; _i++) {
      supply = supplies[_i];
      _ts(1);
      _ts(1);
      _ln(2);
      _to("<li>");
      _to("" + (supply != null ? escape(supply) : ''));
      _to("</li>");
      _ts(2);
    }
    __toffee.res = __toffee.out.join("");
    if (typeof postProcess !== "undefined" && postProcess !== null) {
      __toffee.res = postProcess(__toffee.res);
    }
    if (!__repress) {
      return __toffee.res;
    } else {
      return "";
    }
  };

  true; } /* closing JS 'with' */ ;

  if (typeof __toffee_run_input !== "undefined" && __toffee_run_input !== null) {
    return tmpl.pub(__toffee_run_input);
  }

}).call(this);

Example of templates.js without the toffee header

(function() {
  var tmpl;

  tmpl = toffee.templates["/basic.toffee"] = {
    bundlePath: "/basic.toffee"
  };

  tmpl.render = tmpl.pub = function(__locals) {
    var supply, __repress, _i, _len, _ln, _ref, _to, _ts;
    __locals = __locals || {};
    __repress = (_ref = __locals.__toffee) != null ? _ref.repress : void 0;
    _to = function(x) {
      return __locals.__toffee.out.push(x);
    };
    _ln = function(x) {
      return __locals.__toffee.lineno = x;
    };
    _ts = function(x) {
      return __locals.__toffee.state = x;
    };
    toffee.__augmentLocals(__locals, "/basic.toffee");
    with (__locals) {;
    __toffee.out = [];
    _ts(1);
    _ts(2);
    for (_i = 0, _len = supplies.length; _i < _len; _i++) {
      supply = supplies[_i];
      _ts(1);
      _ts(1);
      _ln(2);
      _to("<li>");
      _to("" + (supply != null ? escape(supply) : ''));
      _to("</li>");
      _ts(2);
    }
    __toffee.res = __toffee.out.join("");
    if (typeof postProcess !== "undefined" && postProcess !== null) {
      __toffee.res = postProcess(__toffee.res);
    }
    if (!__repress) {
      return __toffee.res;
    } else {
      return "";
    }
  };

  true; } /* closing JS 'with' */ ;

  if (typeof __toffee_run_input !== "undefined" && __toffee_run_input !== null) {
    return tmpl.pub(__toffee_run_input);
  }

}).call(this);

hhsnopek commented 10 years ago

@malgorithms Any thought on this?

malgorithms commented 10 years ago

hi Henry, I added a new configurable_compile export function that should get you what you want, especially if you're traversing the files yourself. Here are some example usages:

toffee = require 'toffee'

str1 = t.configurable_compile ' This is a template;  #{partial "test_2.toffee"}', {
  headers:true,
  filename:'/foo/test_1.toffee'
}

str2 = t.configurable_compile ' This is another template', {
  headers:false,
  filename:'/foo/test_2.toffee'
}

str1 and str2 are both javascript; you could paste them into your browser. note the first one includes the headers, and the second one doesn't.

filename is important as it's how the templates are accessed inside the toffee file and crucial for partials to find each other. If you pasted those 2 strings into your console you could do this:

toffee.templates['/foo/test_1.toffee'].render({})

Even though test_1 refers to test_2 relatively, it would still find it. (The goal here is to act like the node version, which would use fs to find it.

There are some other options, too:

minimize : minimizes the javascript to limit file size. This is pretty slow, as uglifyJS is pretty slow. It has to do a lot of JS parsing
to_coffee: if you want to return coffeescript instead of javascript (not sure why you'd want to)

finally, if you just want to get the toffee headers by themselves, so you can prepend them in front of a bunch of files compiled without a template:

toffee.getCommonHeadersJs(true,true,true)

the 3 true values aren't really worth explaining, but you can look at the code if you care.

this function is pretty simple and defined in index.coffee, so I encourage you to take a look if you want to make changes or write something else for yourself. everything it uses is exported already, so really this is just a wrapper function to save you work.

hhsnopek commented 10 years ago

@malgorithms Beautiful! :grinning: I'll look at this and let you know if I have any questions

hhsnopek commented 10 years ago

@malgorithms This works perfectly! Thank you so much

One finally question that needs to be resolved; can toffee throw an exception when there is an error?Currently I'm just getting a response from toffee that says:

Expecting &#039;EOF&#039;, &#039;START_TOFFEE_COMMENT&#039;, &#039;START_COFFEE&#039;, &#039;END_TOFFEE&#039;, &#039;CODE&#039;, got &#039;END_COFFEE&#039;</span></pre>

If toffee would throw an exception that would be awesome

hhsnopek commented 10 years ago

~~Looking at the view src(https://github.com/malgorithms/toffee/blob/master/src%2Fview.coffee#L246), this should throw an exception and output with console.error rather than console.log~~ Is there a reason you don't let the error propagate and output it through the console?

malgorithms commented 10 years ago

there are 2 kinds of errors:

compilation errors (a toffee language violation, or a coffeescript one)
a runtime error (you accessed a variable that was undefined or whatever)

This would cause a compile error: #{foo#{bar}}

As would this, using a coffee keyword: #{var foo}

But this would cause a runtime error: #{foo.aint.defined}

the cases are handled somewhat differently and happen in completely different places. But the big problem when I started making toffee was that if I just let the error happen, and it was caught by a try/catch in Express/whatever-in-node, it would output a very useless stack, especially since toffee code is converted to coffee is converted to javascript. So instead, the error is caught internally, and the default (overridable) is that the template outputs nice HTML and pretends there is no error, highlighting possible source lines from the original toffee. Like this:

This isn't really possible on the browserified version, since the toffee doesn't exist after conversion. So it can't exactly return pretty toffee, all highlighted. So you have 2 options for each of 2 questions:

on runtime errors, should I catch them and return them? Or do you want to catch them? And should it log anything automatically?

on compile errors, should I catch them and return them? Or do you want to catch them? And should it log anything automatically?

hhsnopek commented 10 years ago

Runtime & compile errors: don't return, catch, or log them - It should just throw the error and let it bubble up

Also, you don't need to use try/catch unless you want re-throw a more specific error

malgorithms commented 10 years ago

hi Henry - I hoped to work on this last week, but now I'm going to be on vacation for the next 10 days or so...so it'll have to wait a bit unless you want to poke around in there.

hhsnopek commented 10 years ago

Hey Chris - I was wondering what the status is for this, thank you again for everything

hhsnopek commented 10 years ago

Accord now supports Toffee completely (almost), once errors are properly propagated we can officially add Toffee to accord! :grinning:

hhsnopek commented 10 years ago

@malgorithms hey, any updates on this? If you need some help, I can definitely jump in and help as much as I can

hhsnopek commented 9 years ago

@malgorithms if you're unable to switch toffee to propagate errors, do you have a solution for detecting errors?

malgorithms commented 9 years ago

hey Henry -

The simple answer is still that you can have it call back with errors instead of masking them with pretty printed results. This is an engine setting, so just make your own engine.

toffee = require 'toffee'
engine = new toffee.engine({ prettyPrintErrors: false, prettyLogErrors: false});

engine.run './foo.toffee', {some_var: "bar"}, (err, output) ->
   console.log err
   console.log output

As you can see, this will call back with a standard JavaScript error object instead of returning a fake good result. It fits the async cb model which is getting popular and makes a lot of sense.

Hope this helps. If I understand your request, you might prefer a "sync" instead of "async" version of this, where it returns a result and instead actually causes an uncaught error (which you would catch) when there's an error. This would be a pretty deep change, so I'm unlikely to get to that now.

If you want to avoid an engine and all its benefits (file monitoring, layout forming, etc.) and are creating a view on the fly, you can also pass these same "pretty" params to the view constructor.

malgorithms commented 9 years ago

Oh, also, engine.render might be preferable to engine.run since they're aliases and render matches the toffee.render name.

run is a vestige, and I'll deprecate eventually.

malgorithms / toffee