Allow extensions/plugins

[puzrin]

Could you explain details, what's planned to include in this issue?

It's now e bit difficult to extend parser. Main processing in lexer is hardcoded monolite loop https://github.com/jonschlinkert/remarked/blob/master/lib/lexer-block.js#L99-L374. Additional markup extentions can be added only via pre/post-processing, and that's very inconvenient:

• simple method, like replace to html with regexp is not safe and can have collision with followed remarked call.
• right processing (via nested trees) has almost the same complexity as writing remarked from scratch.

It would be nice to split lexer loops to sequence of named functions. That will allow to inject extentions into right place and reuse the rest of remarked code. Side effect is decreasing parser speed, but i don't think it will be critical.

Let me know what do you think. I could try to help, if will be sure, that result will be accepted to mainstream.

[jonschlinkert]

It would be nice to split lexer loops to sequence of named functions

this is exactly what I've been thinking. we're on the same page completely.

we're using this with assemble, verb and other projects, so I definitely plan to keep maintaining it. I'd love to have some help with refactoring

[puzrin]

Glad to know, that our opinions about lexer improvement are the same. I see 2 possible ways to create injectable respoisibility chain:

1. keep functions in hash, with priorities. like { 10: bold, 20: li, ... } - a bit hardcoded, but should not be a problem for extention.
2. keep named functions in array - then user need to check function name in order to find place to inject new handler. Something like lexer.before("li").add(myHandler)

I think (2) with before/after/replace/remove helpers should be ok. A bit more code for configuration, but should simplify/speedup runtime on parse.

• is it ok?
• are there any other places that need such care to make remarked pluggable?

[jonschlinkert]

I had considered/planned on using a pattern similar to jade. I really like how the parser, lexer and compiler are organized. You have good ideas though, I'm open to anything, especially if you're willing to do some of the heavy lifting.

are there any other places that need such care to make remarked pluggable?

I'll have to think about that. I'm not sure about other areas that should be pluggable. I know that I'd rather have a maintainable code base than the fastest markdown processor around. and I think the parsing and lexing should be the focus initially - rendering can be done in so many different ways - I think if we do this right, users/implementors should be able to render whatever/however they want. e.g. from markdown to markdown, or HTML, or create renderers to generate slide decks, etc. (I'm doing this now with remarked, and it's a pain).

[jonschlinkert]

btw, you might be interested in this: https://github.com/vfmd/vfmd-spec.

[puzrin]

Thanks for link to vfmd.

It seems Parser.prototype.toc could be replaced with something more extendable, but that's not as critical as in lexer. For parser one can just wrap default function to process additional tokens.

The more important point is, that all classes must be immutable, to exclude possible interference between different components, which use remarked and need different modifications. Now it's expected, that changes will be done via global singletons like require('parser'), require('lexer-inline') and so on. It would be more safe to describe remarked in top level file as class, where parser/lexer are instance properties. Then anyone will be safe to modify local instance, instead of hacking global objects:

var md = new Remarked(options);

md.parser = my_customized_parser;
md.lexer_inline = my_customized_lexer_inline;
...

PS. Yes, i confirm, that wish to spend some resources for remarked improvements. That will cost me less, than solving current problems with kludges or creating "one more markdown" fork.

[jonschlinkert]

awesome! looking forward to it

[puzrin]

Ping! See pr. Futher improvements require serious api change and source rewrite. I will do those anyway, but interested in giving result back to remarked. Need to coordinate efforts.

[jonschlinkert]

great! I just got online, I'll pull it down and take a look!