Features wishlist for new architecture

[puzrin]

moved to new repo https://github.com/jonschlinkert/remarkable/issues/2

I'm experimenting with "next generation" markdown parser, that sould solve huge number of problems, caused by using regexps to parse tree-like content (in fact, nested targs is not flat list, that's a tree). This ticket is attempt to validate, that new architecture will be able to solve all existing issues.

• [x] Footnotes - doable.
• [x] Macros.
  • blocks - doable, nestable.
  • inline - doable, but no convention about format.
• [x] HTML - doable. Inline only - tags will be autoclosed at the end of block, if no closing pair found.

If you know feature, that you wish to have directly or via plugin, place a comment here with link to details. For begining, it would be nice to review huge number of issues in marked repo, and collect useful ones.

If you need to discuss implementation details about specific feature - create a separate issue. This one is only to validate, that no more rewrties needed.

Known limitations:

• html tags will not be able to wrap multiple blocks
  • IMHO, that's not needed in real life :) . Everything can be done via block macro extention in more convenient way
  • anyway, almost none of markdown implementations can do it, and nobody died yet :)

[jonschlinkert]

this is great!

If you know feature, that you wish to have directly or via plugin

Ironically, I was just saying to @doowb a day or two before your first comment. I would love to have dynamic features similar to jade but geared towards markdown - and obviously not whitespace sensitive. I think if we were to blend in some ideas from literate coffee we could have something really amazing. If we did a full re-write, honestly I think the jade code base would be a better starting point than marked. of course, I'm just thinking out loud, these are just ideas. Just improving the current code base is awesome as well.

Some other thoughts...

macros

FWIW, I really like the macros concept, but I don't think adding them directly to the lib is the best route. it would be better to allow users to register macros, similar to handlebars helpers.

templates

I've also played around with the idea of using templates to render the HTML. (I started adding a couple of examples and haven't a chance to come back to it since we're trying to release v0.6.0 of assemble). IMO, although not as fast as native, templates are more familiar and approachable to most developers than creating custom renderers.

With templates users could, for example, render HTML with heading ids and anchors the way they want them to be, without having to write custom renderers to do it. they could just pass them to the options or we could load them based on some convention.

From markdown, to markdown

Given the importance of documentation in github projects, I think first class support for rendering to markdown would be amazing. A good use case is converting code comments to markdown. This could be done with templates of course, but when it would be nice to support this natively.

HTML to markdown

Again, this is not a request, but IMO it would be awesome to do in the same library. Why not?

[jonschlinkert]

hmm...

This one is only to validate, that no more rewrites needed.

To that point, if we continue to go down this road, do you think we should make this a non-fork of marked.js? I'm fine either way, but if we do we should do it early. With these ideas it's worth considering a full re-write. Of course we can leverage code from marked.js and other libs and give attribution to those libs, but it would no longer really be a fork. again, just thinking out loud.

If you do want to go that route, we could use https://www.npmjs.org/package/remarkable. if not, no worries, I can just drop the name so someone else can use it.

[puzrin]

1. Block macros will be option. But i must be sure, that effective implementation is possible with new parser logic.
2. templates - i've seen this code, but didn't understood benefits, and left intact. If renderer allow quick overrite for each tag dumper, then you can just replace function in renderer instance. Allowing templates looks like overkill. See lexer plugins logic - it's very easy to modify any part, and you don't even need to access rules/blocks to write your code.
3. toMarkdown - IMHO that's completely separate task, that has no common methods with parser. At least, i don't see benefits from keeping this 2 different things in one package. But my knowledge is limited :)
4. Non-Fork. YES. I just don't like to make buzz before can show rezult. That's more management question, related to promotion and copyright/control. I've sent you details to email from your github profile. Take a look. Name change is not a problem at all. Currently i do 2 things in parallel - cleanup existing branch & dig into new architecture.

[jonschlinkert]

templates - i've seen this code, but didn't understood benefits.... Allowing templates looks like overkill

fair enough, these are good points. I haven't considered this well enough in light of the new changes. It probably is overkill, I just personally find that templates are super-easy to use and drop into a project, but the changes you've made will allow me or anyone to choose whatever we want to do.

and left intact

Feel free to remove or change any code here. It's nice to have a peer review. This is how we learn :-)

toMarkdown - IMHO that's completely separate task, that has no common methods with parser.

this is true. I'm also a fan of keeping a separation of concerns and modularizing code... my motivation is that there are no "dominant" HTML to markdown libs written in javascript, and bundling this makes the project more interesting. but I think your instincts are correct. maybe we can revisit later with a different project if you're at all interested

Non-Fork.

:+1: great. I'll take a look at the email.

[puzrin]

my motivation is that there are no "dominant" HTML to markdown libs written in javascript, and bundling this makes the project more interesting.

I know more simple way to make project attractive. It should do the same as others, but multiple times better and 10x times faster :) .

On practice it will be very useful, if someone scan marked issues tracker, and create a list of top irritating things (not yet fixed in remarked too).

[jonschlinkert]

ha! indeed.

it will be very useful, if someone scan marked issues tracker,

I'd be happy to do that. My daughter's one-year b-day is tomorrow and we're having a party for it :-) so probably Sunday/Monday. have a good weekend!

[puzrin]

That will be really helpful. I used the same concept, when developped pako - analyzed disadvantages of all existing modules & cared about speed. After release the major projects like js-zip & browserify switched to pako in 1-2 months. Don't know if parser speed will be significantly faster, but existing regex-based archtectures are source of infinite ass pain for developpers, who need syntax enhance.

https://gist.github.com/puzrin/257feacd84c7b2911376 - here is microbrainstorm about new project name.

[jonschlinkert]

Do you want me to link the issues here or just summarize them? I'm doing this now.

[jonschlinkert]

I'd prefer to link them fwiw

[puzrin]

If links contain all details, that's enougth.

[jonschlinkert]

• [ ] typographical features: https://github.com/chjj/marked/issues/27. Macros should be able to handle these
• [ ] to-do lists: https://github.com/chjj/marked/issues/107
• [ ] whitelist certain tags when sanitize is used: https://github.com/chjj/marked/issues/133

[jonschlinkert]

regarding the name, I think we should change it now to remarkable. otherwise every issue we create or work we put into this existing project will create more work to migrate. especially as I link to issues, those will need to be updated. thoughts?

I can add more todos after we decide

[howardroark]

Before you get rolling I would just like to cheer you on!

I see a great potential for Markdown to start being adopted outside of the development community. I have worked with many people who have struggled endlessly with wysiwyg and html when trying to customize content for the web. Philosophically I think a Markdown document should be completely readable before it is parsed without knowledge of the conventions in place.

I am looking forward to see what you come up with in terms of Macros, and would be more than willing to participate in any conversations :) I would love to get this tool in place for a browser based split pane editor.

[puzrin]

@jonschlinkert I'm ok with remarkable. Please, add me there to pushers. Or, if possible, to owners until first release - it will be more simple for me to give access to additional core programmers.

A couple of comments on features:

• Typography. That's the same class of tasks, as links autoreplace and emails autoreplace. It's reasonable to to such post-processing in renderer, when it writes text node. Not in lexer-parser. Don't see significant problems.
• ToDo lists. Ok. Don't see any conflicts
• HTML whitelist - yes, already designed. Should not be a problem.

[jonschlinkert]

:+1:

[jonschlinkert]

Regarding the list of features, fwiw I'm not casting judgement on the features/wish list yet, I'll just continue to add items to the list.

Regarding remarkable, since (as far as I know) there is no way to un-fork a repo I created a new repo: https://github.com/jonschlinkert/remarkable. I'll add everyone there, as well as on npm.

Does anyone know of a tool to migrate issues? there are only a handful, so it will probably be easier to move them manually, but I like automation tools lol

[jonschlinkert]

@puzrin please list the people you need added to the repo/npm. thx

[puzrin]

@jonschlinkert add https://github.com/rlidwka & https://github.com/Kirill89 please. Only to repo. And me to npm, for ~1 month.

[jonschlinkert]

:+1:

[jonschlinkert]

@puzrin what's your npm name? it's giving me an error

[puzrin]

https://www.npmjs.org/~vitaly :)

[jonschlinkert]

ah, yes. thanks!

[jonschlinkert]

all done.

[puzrin]

Closed - moved to new repo.