Literate CoffeeScript support

[billymoon]

It seems that having any file.coffee.md named files in the source folder breaks the toaster, with this error...

bash-3.2$ toaster -c
path.existsSync is now called `fs.existsSync`.

/usr/local/lib/node_modules/coffee-toaster/node_modules/coffee-script/lib/coffee-script/coffee-script.js:51
      throw err;
            ^
Error: Parse error on line 22: Unexpected '='
    at Object.parseError (/usr/local/lib/node_modules/coffee-toaster/node_modules/coffee-script/lib/coffee-script/parser.js:477:11)
    at Object.parse (/usr/local/lib/node_modules/coffee-toaster/node_modules/coffee-script/lib/coffee-script/parser.js:554:22)
    at Object.exports.compile.compile (/usr/local/lib/node_modules/coffee-toaster/node_modules/coffee-script/lib/coffee-script/coffee-script.js:43:20)
    at l.__t.Builder.l.compile (/usr/local/lib/node_modules/coffee-toaster/lib/toaster.js:1:16001)
    at l.__t.Builder.l.build (/usr/local/lib/node_modules/coffee-toaster/lib/toaster.js:1:13322)
    at l.__bind [as build] (/usr/local/lib/node_modules/coffee-toaster/lib/toaster.js:1:570)
    at f.exports.Toaster.f.build (/usr/local/lib/node_modules/coffee-toaster/lib/toaster.js:1:24845)
    at new f (/usr/local/lib/node_modules/coffee-toaster/lib/toaster.js:1:24530)
    at Object.exports.run (/usr/local/lib/node_modules/coffee-toaster/lib/toaster.js:1:23865)
    at Object.<anonymous> (/usr/local/lib/node_modules/coffee-toaster/bin/toaster:7:27)

[arboleya]

This is very weird, because any file that doesn't end with .coffee is left out in the file's search.

Check this line: https://github.com/serpentem/coffee-toaster/blob/master/src/toaster/core/builder.coffee#L44

You'll see this:

result = fsu.find folder.path, /.coffee$/m

I've added a file app.coffee.md inside the examples/single_folder/src/app to test it and the build/compile occurred fine without any erros.

Can you provide more info so I can investigate it better?

[billymoon]

Here is the blog post from coffee-script author about it: http://ashkenas.com/literate-coffeescript/

I tried to replicate your issue, but it is working as expected. When I have app.coffee it compiles everything fine. When I rename it app.coffee.md it compiles everything except that file ok (so no error on command line) as it is simply ignoring that file.

The REGEX

I think the regex to match coffeescript files, including literate coffeescript should be...

/\.(litcoffee|coffee(\.md)?)$/

Note: the dots are escaped, otherwise they are matching any character, not a dot

This will match all files ending in .litcoffee, .coffee.md, and .coffee.

I found several parts of your code that rever to /.coffee/, which I assume should probably also be changed to the same as above. Also, replacement of .coffee for .js should probably also use above regex.

Literate CoffeeScript

Literate CoffeeScript concept is extremely simple. It is just Markdown files, named .litcoffee or .coffee.md that are compiled by coffee-script, where only code blocks (indented by four spaces) are compiled, and all other content is ignored.

Firstly, you need a more up-to-date version of coffee-script for it to work at all. I would switch to the latest, 1.6.1. I was getting an old version in your package, and therefore coffee-script was throwing errors itself.

I have only used literate coffee-script from the command line. It automatically uses literate mode when the filename is one of the new extensions. I expect your script calls coffee-script from inside, rather than running a command line on the target files.

I will look into the changes made in coffee-script to enable literate mode, to try to figure out how the literate version is called, and get back to you. I can not find any documentation to explain how to call from within a nodejs app yet.

[arboleya]

Oh well, thanks for clarifying. I'm not a literate-programmer myself so I guess I completely misunderstood you first comment. Lets make a fresh start.

About the regex, you're right about the dots. Also, the version in master (0.6.12 now) is pretty old since I'm working on a complete refactoring of the library (that changes everything), so my focus is in this refactoring now instead of the current version.

That said, lets get back to your problem.

Yes, Toaster use CS in introspective mode, as a library instead of a CLI. If I'm guessing it right, I'll need to set some option when compiling the file in order to inform the respective .litcoffee / .md file for each .coffee file. I'll take a look into this usage to see how it works.

[arboleya]

Ok, there's no 'respective' file at all. Both source and literate annotations resides in the same file. The differing of plain .coffee files and .litcoffee or .coffee.md appears to be based on the file extension.

I suppose you're using docco to generate the docs?

[billymoon]

Any markdown renderer can generate the docs, because the files are perfectly ordinary markdown files... or you can just read them in a text editor (as that is how markdown is designed). I think this is a (very smart) diversion from docco, to use a more standard, very expressive documenting syntax.

In fact, there are probably accidental literate coffee-script files, that pre-date the compiler being able to handle them - simply when people are blogging/documenting their coffee-script using markdown.

[arboleya]

There's an undocumented option literate which you can pass when compiling scripts.

cs = require 'coffee-script'
fs = require 'fs'

filename = 'app.coffee.md'
file = fs.readFileSync filename, 'utf-8'
literate = /\.(litcoffee|md)$/m.test filename
compiled = cs.compile file, {literate}

console.log compiled

With this literate option set to true, .litcoffee and .coffee.md files will compile properly.

As you said, there's some places with mention to .coffee extension only. This cases must to be adapted, and the compile cases must to inject the literate option.

I'll see if I can find the time to apply this on top of maste. If not I'll add this in the modular-refactoring I'm working on.

Pull requests are very welcome if you (or from anyone following this topic) have urgency on having this feature implemented on top of the master/dev branch and got the time to implement it right away.

Note: Thank you very much for the Stylus package for Sublime! :)

Just now I realised who you are. :congratulations:

[billymoon]

I had a go at adding it to master, and for a simple case it was pretty easy, but after testing with more files, I was coming up against a problem. As far as I can tell, your script concatenates the source files together, and then compiles right?

To me, it would be more intuitive, to compile each file separately, and then concatenate them. This would allow different source file types (which literate coffee-script is, and could include others in the future), and also, would allow the source files to retain their header comments in the final output file...

/**
 * My library one header...
 */
(function(){ ... library one minified code ... })()

/**
 * My library two header...
 */
(function(){ ... library two minified code ... })()

I am going to leave this alone for now, as I don't want to waste efforts on something that will become obsolete soon (master branch). Let me know when you finish re-factoring.

[arboleya]

In the debug mode files are compiled individually and saved individually as well.

Compiling files individually before the concatenation would include CS helpers (extends, etc) in each file, which is very bad. As CS doesn't provide a way for excluding helpers for certain cases, it'd require some extra mapping and cleaning regexes to handle the extra code.

I believe the header's comments is actually an aesthetic thing since it wouldn't affect functionalities, and you can split libraries using more than on toast call in your config file in a way the output of one library acts as a vendor for another. Anyway I'm pretty sure it can be done in others ways too, point is that I'm trying to focus on functionalities right now. Thanks for the insights.

As for the docs you are trying to generate, if it's an immediate option take a look at codo. It generates API docs (okay, not literate) properly for toaster projects when packaging options is set to true (default).

[billymoon]

Re: debug mode… I can't get it to work. I am doing toaster -d -c . and it is failing with an error where it is trying to compile a string, which is mixed coffee-script and literate-coffee-script. I don't know exactly where the string is built from - but the issue remains. The string is concatenated from several files before being compiled.

Re: Headers I actually find it quite useful. I was working on a project recently, and was trying out many various encryption libraries. Each one was minified onto one line, and had a header, so I was simply copy/pasting them (some from the web) into a single file to include. After I had done all that copy/pasting, and chosen the best ones, I don't see the need to split them all up again into files. I think it makes quite a lot of sense to allow a module's components to be itemised this way.

The bigger benefit from compiling each file individually, and only concatenating afterwards, is that you can compile many languages, not just the coffee-script. What about if I want my stylus files compiled too for example? Or some Dart files? If you are refactoring now, it is a good time to consider how to make your software more versatile, even if you don't add the functionality now - at least you could make the space for it to be added in the future.

Re: codoc Thanks for the link. It looks nicer than the other ones like it. However, I think it is much more useful to generate docs after javascript is compiled, as my projects usually have some javascript, and some coffee script source files. Also, although I have been using JSDoc, I find that at least 50% of the generated docs is cruft, and it is hard to get an overall understanding of the code. I am quite excited about this literal approach to documenting, as it solves that issue for me - as it is comments written by humans, for humans, and can be complimented by an auto-generated API from parameter tags.

The thing that is really great about literal coffee-script, is that the markdown files are editable/renderable anywhere that can handle markdown. There is no other special requirement - and because the documentation is right there with the code, it makes it extremely easy to keep it up-to-date. And because it is markdown, you can be very expressive, rather than just writing a list of parameters.

On Mar 27, 2013, at 8:40 PM, Anderson Arboleya notifications@github.com wrote:

In the debug mode files are compiled individually and saved individually as well.

Compiling files individually before the concatenation would include CS helpers (extends, etc) in each file, which is very bad. As CS doesn't provide a way for excluding helpers for certain cases, it'd require some extra mapping and cleaning regexes to handle the extra code.

I believe the header's comments is actually an aesthetic thing since it wouldn't affect functionalities, and you can split libraries using more than on toast call in your config file in a way the output of one library acts as a vendor for another. Anyway I'm pretty sure it can be done in others ways too, point is that I'm trying to focus on functionalities right now. Thanks for the insights.

As for the docs you are trying to generate, if it's an immediate option take a look at codo. It generates API docs (okay, not literate) properly for toaster projects when packaging options is set to true (default).

— Reply to this email directly or view it on GitHub.

[arboleya]

Debug (-wd)

Literate coffee-script wont work on master in any way by now.

Headers

If you're doing it manually, surely. But if you're using dependencies in a more fashion way, using NPM for example, there's no copying paste, just the concatenation of many files. You'll probably never pay attention on headers. But it surely has its utilities.

Docs [...] and can be complimented by an auto-generated API from parameter tags

What do you mean by an 'auto-generated API from param tags? Are referring to some specific tool?

I agree with you, maybe I just need to dive into this literate thing a little more, to explore its beauties. I don't know if it replaces API docs completely, maybe a merge of both can be even more interesting?

[billymoon]

Re: Debug

I had modified the master a little, but was not successful in getting it to work reliably, because at some point, the coffee-script compiler is called, with a string of coffee and literate coffee concatenated together. This was happening only in some circumstances, but often enough to be a showstopper for me. The solution in my mind, to allow for addition of other languages also, is to re-factor so that all files are always compiled independently, then concatenated, rather than concatenated and compiled.

Re: Headers

Yes, this is not an important point, but something I would find useful never the less, for example, when authoring a micro framework, where you could easily compile everything so that each function is minified onto a line, and it's usage info retained as a comment above. I would use your script more if it was more versatile like that.

Re: Docs

Yes, an augmentation between the formulaic API approach, and the expressive markdown approach is excellent. I see no conflict, you can just add the JavaDoc style tags into the markdown. The only question is how will it be used. Maybe JavaDoc style will be used for documents expected to be read by module consumers, and literate comments by module developers. Personally, I have started using literate style for the root document, where an overall architecture needs to be explained, and regular coffee-script for the other documents.

On Mar 27, 2013, at 11:55 PM, Anderson Arboleya notifications@github.com wrote:

Debug (-wd)

Literate coffee-script wont work on master in any way by now.

Headers

If you're doing it manually, surely. But if you're using dependencies in a more fashion way, using NPM for example, there's no copying paste, just the concatenation of many files. You'll probably never pay attention on headers. But it surely has its utilities.

Docs [...] and can be complimented by an auto-generated API from parameter tags

What do you mean by an 'auto-generated API from param tags? Are referring to some specific tool?

I agree with you, maybe I just need to dive into this literate thing a little more, to explore its beauties. I don't know if it replaces API docs completely, maybe a merge of both can be even more interesting?

— Reply to this email directly or view it on GitHub.

[arboleya]

I got it, in master the release/output file is always generated whether you set de -d option or not, that's why you may be getting an error.

I thought a lot about adding more languages to coffee-toaster, because it just makes sense. This will require lots of extra work and a different architecture based on plugins, to be able to grow as needed. I started the idea it but it was leading me to an ever endless re-re-re-factoring, so I decided to delay it a bit for now in favor of a little solidness and sobriety.

Regarding this multi-language approach, I recommend you to check brunch.io. It didn't meet my specific needs, but definitely worts a try. It's very versatile, check it out.

[arboleya]

@billymoon Hi, I just want to tell you that I'm discontinuing Toaster in favor of it's new fork named Polvo.

More info here:

https://github.com/serpentem/coffee-toaster https://github.com/serpentem/coffee-toaster/wiki

NOTE: Literal CoffeeScript is supported in Polvo.

[billymoon]

Looks great - thanks for the pointer.

On Apr 12, 2013, at 7:04 PM, Anderson Arboleya notifications@github.com wrote:

@billymoon Hi, I just want to tell you that I'm discontinuing Toaster in favor of it's new fork named Polvo.

More info here:

https://github.com/serpentem/coffee-toaster https://github.com/serpentem/coffee-toaster/wiki

NOTE: Literal CoffeeScript is supported in Polvo.

— Reply to this email directly or view it on GitHub.