Consideration of a "Macro" standard for Markdown.

[howardroark]

This is a clone of a ticket @ marked.js. Looking at all your great work with assemble I think we are similar wave lengths. Maybe you have thought about this problem as well.

I work in the advertising industry and can see how Markdown will start to make headway there. The usage concept is simple enough that new writers and some old will have an easy time getting their mind around it. This was always the downfall of HTML. I can foresee a future where developers are delivered plain-text/syntax instead of MS Word documents (or even better never involved in that process again). The industry has a very strong desire to maintain brand identity at all times and Markdown is such a sane way to separate content and style. This is especially true for slide based presentations, which is how agencies acquire all new work. The de facto today is Keynote. It lacks the ability to separate content and style and also has no remote execution ability. Two things that the modern browsers can do with great ease.

That said, there are certain economic trends that need to be accounted for. Third party things like youtube, vimeo, twitter... All of these things have libraries/APIs and industries are aware that they should be able to easily find their way into web based content. It is a rational expectation. The issue is that you end up falling back to HTML again.

I came across this example of Macros and thought it was a very sane approach. The trick would be the overall strategy to get custom Javascript in play without having to add it to marked.js itself.

Designing a common syntax for handling Macros that other projects could be helpful to the future of Markdown as well. Something like this would be so handy...

![:macro params](value)
![:vimeo 600x400](13697303)

Though, this may be a bit more sane...

![:macro](Params, Values, Jison... pass to parser as a string)
![:vimeo](13697303 600x400)

I can also see a lot of potential for this idea in terms of scientific writing. A Macro that could parse natural language with something like Jison into a diagram could be very useful. Though when something like Jison is involved a multiple line syntax would be helpful.

![:venn](
Soccer = alex, casey, drew, hunter
Tennis = casey, drew, jade
Soccer ∪ Tennis
)

Maybe better?...

!!:venn
Soccer = alex, casey, drew, hunter
Tennis = casey, drew, jade
Soccer ∪ Tennis
!!

Spit out some Canvas or SVG...

I understand that this may be out of scope for this project, though I feel this is something that should be accounted for at the parser level. Having a common syntax for dealing with these situation and an expectation to pass natural language as an input would really aid the process of fluent writing and idea generation. I could be as simple as having a venn.js or vimeo.js in a folder called macros which is relative to the path of marked.js.

[doowb]

@howardroark have you seen markdown-symbols?

@jonschlinkert from the description, using resolve-dep and markdown-symbols, you could probably implement this macro/plugin features pretty quickly. I haven't dug into the parser as much as you yet so I'm not sure where it would go. This would be nice for things like I was doing with the UML diagram helper too.

[howardroark]

Hey! markdown-symbols is certainly a quality solution for that goal. Though, it does go down a slippery slope in confusing web UI with a true document. It reminds me of the problem that a CMS causes in a client-developer relationship. The client starts asking that the CMS can do anything and then the developer ends up being the only one who can use it because of the amount of things to remember.

The idea here is to start to harden a methodology for how custom UI "Macros" would work in Markdown. With hopes that other projects could follow suit down the road (maruku, red carpet...). This way the user can distinguish true Markdown from a custom Macro. This way you avoid having a Markdown syntax for every single eventuality. The end user would eventually become aware that the individual Macros are not a universal thing. It is more of a greater adoption thought process for me.

The biggest first step is to design a universally agreeable syntax and methodology for the process. If it is thought through well enough before execution it could stand a chance for greater adoption. I personally like the idea of the syntax leveraging the image tag because it will just show up as a broken image when parsed elsewhere. Like via GitHub.

The real trick is updating the parser to have a sane, simple and universal method for dropping a Macro in.

Funny you should mention UML diagrams! I made a little service that leverages this project to allow you to pass the lines of Jison as part of a JSON embedded get parameter. You are totally on the same wave length as me there! It just runs the code through phantomJS. It is buggy and slow, but a fun proof of concept. GitHub does a great job of caching the images though.

![sequence](http://get-diagrams.com/sequence?json=[
    "Andrew-->China: Says Hello",
    "Note right of China: China thinks about it",
    "China-->Andrew: How are you?",
    "Andrew-->>China: I am good thanks!"
    ])

![sequence](http://get-diagrams.com/sequence?json=[ "Andrew-->China: Says Hello", "Note right of China: China thinks about it", "China-->Andrew: How are you?", "Andrew-->>China: I am good thanks!" ])

[doowb]

Nice UML project! I might have to update my handlebars helper to use that. I think we're thinking of the same thing here. Using marked-symbols to create the macro tag that will be used for all macros (which is an implementation issue and can be done differently in different markdown processors). The specific macro will be registered with remarked before running like helpers are registered with Handlebars. I think the work is coming up with the syntax to use and where to add them to the parser.

[jonschlinkert]

whoa! that UML project is awesome. nice work! The fact that those can be embedded in comments is really cool.

I think the work is coming up with the syntax to use and where to add them to the parser.

yeah I think that's right, the hard part is deciding on a syntax that is unlikely to conflict with other syntaxes. Adding them in the parser or creating the patterns to parse them is the easy part (IMHO). marked-symbols is just a collection of Strings parsers (and actually I need to update marked-symbols b/c the Strings API has changed).

I like the concept, happy to help. I created marked-symbols for a similar purpose. On that note, I'm considering introducing first-class plugin support to remarked, so that plugins like this would be automatically loaded and used. But that has it's own challenges. I think remarked needs to be refactored a bit before that will work the way I'd like it to.

(btw, here is the uml helper @doowb mentioned.)

[howardroark]

I can't take any credit for the UML project, I just stuck it on a web server :P

I am huge proponent of describing what you are about to build and the rational for it's existence before you begin working. I feel like both of you are of much the same point of view.

I just discovered your assemble project and already feel like it has a place in the strategies I am working on. One thing the project's About section could use is a personal story about its role in this world. It did take me a while to digest the nature of the project, but I get it now! I also agree that Jekyll and GitHub pages have some awkward limitations. Though they have uncovered an amazing economic opportunity. The Pages platform is genius and will likely begin to chip away at the inclination to default to Wordpress in the world of independent web contractors. I have been introducing non-savvy people to the concept of Markdown and they tend to love it after a little time with it. They also really get the concept of not needing to rely on a LAMP stack to deliver their content. With the growing array of start-up providing RESTful 3rd party tools it only makes sense to go this way. Why not a platform that runs assemble on each commit! Hook it up to GitLab! Shit have it all wrapped up in an VM image and deliver it to the cloud on per organization basis. My brain is just a goin' now!

I am currently working to describe and design some workflows that leverage Vagrant and Salt to build out Ubuntu based development environments on the fly. I find the largest barrier for new developers is getting everything running and setup. Automation is key! This way you can gracefully introduce yourself to concepts like Grunt (and assemble). Though if you are just getting paid to write some CSS you can be up and running just like the rest of the team. There is such a massive market of intelligent UI developers who fear Linux and the command line. Yet they can spin up some crazy shit in the web browser, they just rely on FTP to get it on the web. Ex-Flash developers really.

Wow, I got side-tracked. The other side of this story is that Markdown is the perfect solution to give content creators the perfect amount of control and access to the appropriate logical layer. It is the only solution that really separates presentation and content. Just like we are all agreeing... It would be so great to hear a non-savvy Markdown writer ask "Hey, is there a Macro for this?".

I'd love to take part in any discussions you may be having related to your refactoring or about the assemble project. Would you have any interest in scheduling a Google Hangout?

Long term I am trying to bill my self as an "Open-source Strategist". The idea being that a company hires me to help them figure out how to better align themselves with these economic trends. I have had a lot of experience in the industry of selling "technology" to the corporate world so I know how to navigate it pretty well. Projects like assemble are exactly what I am looking for. These days consultants tend to just confuse business leadership into spending dollars rather than giving them the proper understanding and a confidence that things will improve over time. It is also rational to argue the costs will go down as well.

[howardroark]

@jonschlinkert @doowb What if Marcos were a strategy that relied on bower and require.js?

Considering that Macros will require Javascript and CSS and most often involve the DOM it would not be realistic to render much more than a HTML placeholder on the sever-side. Though you would also need to gather dependancies for execution on the browser side. What if Macros themselves were GitHub repos that were registered with bower? The tweak to the Markdown processor would be understanding that should it encounter a Macro it would then need to see if it has already gathered it's dependencies. It would then need to assemble a common.js file for require.js to act on. The would mean that all of the logic required could be nested in the bower enabled Git repo of the macro itself.

I think the way to look at this effort is by designing a common Markdown syntax that gets rendered into a common HTML placeholder. Each macro would understand that there will always be a common format and make due from there.

There is a great little node tool here that could be part of the way there... https://github.com/yeoman/bower-requirejs

The Markdown:

![:bower-name](
   String of options to act on.
)

![:js-sequence-diagrams-macro](
    Andrew-->China: Says Hello
    Note right of China: China thinks about it
    China-->Andrew: How are you?
    Andrew-->>China: I am good thanks!
)

The HTML:

<div data-macro="js-sequence-diagrams-macro" data-options="Andrew-->China: Says Hello
    Note right of China: China thinks about it
    China-->Andrew: How are you?
    Andrew-->>China: I am good thanks!" />

It's a bit wild, but I can't think of a strategy that better leverages what is out there.

[jonschlinkert]

well, the plugin part is pretty simple. Internally we would just use resolve-dep, and do something like (pseudo-codish):

var local = resolve.local(foo); // user-defined
var plugins = resolve.npm(['remarked-*']).concat(local);

var stack = [];
plugins.forEach(function (plugin) {
  stack.push(require(plugin));
});

It's really just a matter of deciding how heavy we want to make remarked. I'd like to keep it light and fast, but the main goal is to make it more useful and usable. I'll need to keep thinking on this one

[howardroark]

Totally agree, light and fast are key. Especially when you are making last minute iterative updates. I'll think on this as well. Apart from updating the HTML with proper placeholders the plugin would not do much more than cache static files from bower in the assets folder, and manage a common require.js file. This way the build process would just chuck everything into the _gh_pages folder each time. Though it would necessitate assemble templates that make proper use of AMD. CSS would also be an issue, but there are plugins to let you handle CSS in an AMD style.

The main thought process is... How do you separate Macros from the build process as much as possible and ensure that they can be rich and dynamic. The use of bower would allow macros to more easily align with a build process and have their own home for collaborative development.

Thanks for giving this some thought either way! I know I can be a little intense, haha :P

P.S. Do you have any articles on how you deal with the gh-pages branch as part of your build process? I'm thinking about always having two clones of a single repo, one checked out to master and the other gh-pages. Then symlink the gh-pages version to the _gh_pages folder and have a script that does a forced push.

[puzrin]

A there any final conventions about macros format?

For blocks i like

!!name param1 param2 ...
...
!!

or

!!:name param1 param2 ...
...
!!

For inline - don't know.

[jonschlinkert]

A there any final conventions about macros format?

what are your thoughts?

[puzrin]

I have no experience/stats of intensive markdown use in different situation. For my nodeca project i need only blocks, and this one looks attractive (minimalistic, readable, non conflicting):

!!name param1 param2 ...
...
!!

That's not principal question, because lexers are now pluggable.

[howardroark]

I have been thinking about this problem from the POV of a non-developer for quite a while. I have been showing young writers Markdown and they really do like how fluid and simple it makes writing. They also get why having a subtle amount of syntax makes the document more universal. Not only that, we were using Jekyll and updating right through GitHub... They loved the nature of how Git based version control let's them evolve ideas quickly (so long as they kept their lines at 80 characters!).

I am planning to build a service that allows writers of any sort to leverage Markdown and static sites to get their ideas on the web while leveraging Git. The service will be designed to leverage GitHub's API, OAuth, and it's free static site hosting. Off the bat I was going to use a post processor client side to turn the Macros into html and then let the unique Javascript for each Macro tackle the DOM. Though long term I would like to add the ability to process the Markdown and Macros server side with tools like assemble :)

I really think there is a solid case to have a Macro syntax that is ubiquitous in Markdown. All the processor would really need to do is produce a small amount of HTML and offer the ability to hook in a post process callback. It may be violating the Unix philosophy a touch to try and make remarked do too much, but I think nudging Markdown itself to evolve a little is worth while.

I would argue that the most suitable syntax is as follows...

block style

{{macroname
CONTENT
WITH
   FORMATTING

       PRESERVED
}}

inline style

{{macroname CONTENT            WITH FORMATTING}}

Both of these would ultimately output the same markup...

<span data-macroname="macroname">
CONTENT
WITH
   FORMATTING

       PRESERVED
<span>

At this point the individual Macro's own Javascript would do it's magic and turn out some useful content for the user. This means that each Macro would be responsible for converting natural language or some language based expression into real DOM content. This helps stay true to the original philosophy of Markdown and ensures that Macros will always work for the purpose writing. As an example you could make it simple for scientists to convert formulas into graphs and charts, or maybe just let a blogger add in a youtube video {{youtube YOUTUBE_ID}}. Macros would be tailored to the intended audience. The point is that the basic pattern is there and can be learned in mass.

I think that exclamation points are not quite elegant enough, the shape of the curly brace really encapsulates the content nicely. Plus this curly braces feel natural inline with text and as a block on their own. This format also ensures that a minimal amount of foreign syntax is required.

BTW, I am drawing a lot of my inspiration from prose.io

[puzrin]

@howardroark , i don't like {{ + }} syntax for 2 reasons:

1. It looks broken for blocks (open but not closed scopes)
2. Using the same markup for blocks + inlines will add ass pain to parcer. You will be unable to cheat with pairs search for nested blocks. It's very convenient, if you know that block closing can be only at the start of following strings.
3. Your example does not show how to pass params if required. That's a useful feature.

[howardroark]

I must admit that I am more of an idea guy than a programmer. I really see the economic potential of Markdown + Macros.

Your logic totally makes sense when considering both points together. I could see the idea of inline and block macro syntax registering with a writer. Mustache for inline and exclamation for block sounds like a healthy standard :) Though I think params part may be too programmer focused. It also assume that you will be asking remarked to do custom work each time. On Aug 29, 2014 12:37 PM, "Vitaly Puzrin" notifications@github.com wrote:

@howardroark https://github.com/howardroark , i don't like {{ + }} syntax for 2 reasons:

1. It looks broken for blocks (open but not closed scopes)
2. Using the same markup for blocks + inlines will add ass pain to parcer. You will be unable to cheat with pairs search for nested blocks. It's very convenient, if you know that block closing can be only at the start of following strings.

— Reply to this email directly or view it on GitHub https://github.com/jonschlinkert/remarked/issues/9#issuecomment-53899978 .

[puzrin]

@howardroark take a look at wiki templates syntax and octopress blocks syntax. That's not my personal invention, just summary from real life. If you don't need params - nobody make you use those. But if you suggest good/universal format, it should be flexible.

[howardroark]

I will take a look! My main goal is to keep things ultra simple for writers, but I totally agree with the need to be flexible. Would you see macros as being something that would be registered before remarked runs?

function myMacro(param1, param2) {
  var math = param1 + param2;
  var content = %s;
  return "<span>"+ content + math + "<span>";
}

remarked.registerMacro(myMacro);

A fella named @golyshevd made a tool called macroed that does something like this. Though, the syntax is just a bit too complicated for my liking. It is very extendable though! Would be great for developers, you may find inspiration there.

Food for thought!

[puzrin]

I think, that implementation details are not significant. Formatting is the key question. It must be:

1. visually attractive (easy to read, easy to write)
2. flexible (allow params if needed)
3. be ok for blocks and inlines (personally, i think that different syntax will be better both visually and for programming)

macroed concept (offset based) is really interesting. So, it's my second favorite:

!!name param1 param2
  content
  ...

params can be like this: center=left big red or center=left, big, red and so on - not principle question

One more idea - if block content starts with offset, then no needs to close it. Ib content is without offset, then closing tag is mandatory. Advantage is, that offset is not convenient for big blocks. Disadvantage - you can not know, how to select better format when generate markdown.

PS. Inline marcos are still uncovered (those can be self-closed tags and open + close pairs).

[jonschlinkert]

yeah, it's kind of similar to https://github.com/jonschlinkert/markdown-symbols. although I didn't really have a need for this, so I just put something together quickly at the time

[jonschlinkert]

@puzrin I think it will be easier for everyone to conceptualize the pros and cons of some of these syntaxes once unit tests are implemented, or functional examples etc. sometimes it's faster to write the code than to talk about it lol. either way, I'll probably yield to your judgement, given your experience with js-yaml and other projects

[jonschlinkert]

wow, funny how I can look at code I did just a couple of months ago and cringe...

[puzrin]

@puzrin I think it will be easier for everyone to conceptualize the pros and cons of some of these syntaxes once unit tests are implemented, or functional examples etc. sometimes it's faster to write the code than to talk about it lol.

Let's slpit problem to 2 parts - user requirements (formatting/api) and implementation. You are right, that i have no problems with implementation. But i have problem with knowledge of markdown use cases. Collecting user opinions is completely different area, that can't be replaced with my fantasy and my programming skills. For example, when i readed tickets here and in marked, i've found many cases, that i could not expect.

It's very useful to have someone separate, who can collect ideas and feedback from big projects, set priorities and filter incoming info. That will help to reduce rewtites count. I still need to try alternate ways to take over the world - nobody yet did fast & high quality js image resizer, and i wish to do it right after markdown :)

[jonschlinkert]

lol, awesome. we'll leverage each other's strengths then! I'm good at user requirements and I've written thousands of pages of markdown documentation (500+ repos, some with hundreds of pages of docs). This will make my life easier for sure!

[golyshevd]

Guys, found recently https://www.npmjs.org/package/deep-marked, did you seen?

[puzrin]

@golyshevd the same klugdes collection, but with another color. Nothing interesting.