Incremental parse

[puzrin]

We should be able to parse 1 root-level block, starting from XXX line.

Simple partial update looks like this:

1. Locate root-level container for changed content
2. Parse it, and get the end position.
3. Check that taill from block end to document end is the same length as before.
4. If not - parse one more block and repeat (3)

Question:

• Is it ok or we should have more presize parse at inline level?

[geyang]

Personally I think incremental parse is not a requirement. for two reasons:

1. for non-react/angular2 systems, the re-paint during DOM update is always going to be more significant, therefore parsing time is not the bottle neck.

   Also, the user can opt to use array diff patch algo to diff two AST returned by markdown parser. It appears to me that parsing is almost always going to be global and done via a full pass, in a state-less manner. So I don't see diff-parse a hard requirement.

2. I think this kind of partial/incremental parse is easy to do at root 1 level, but if you do it at root-1 level, you are going to quickly going to run into roo-2 level and so on, and in the end just parsing the whole thing.

   You can limit the depth of the parse. But you will still have to go through the whole document.

[puzrin]

Term "incremental parse" needs clarification, because some things are technically impossible. For example, if you add "```" (fenced block start) that will change all document till the end.

Tehnically, if you change line:

• you can detect affected root-level block (even now)
• you can start parse from that block start until block end and get last line number
• then you can find next unaffected block, and apply diff to AST.

So, if someone need "parse from line XXX till block end" - that's doable. More granular parse is more difficult and will not give enougth value IMHO.

Uncovered: need to check what happens with references & footnotes.

[geyang]

So, if someone need "parse from line XXX till block end" - that's doable. More granular parse is more difficult and will not give enougth value IMHO.

I agree with you, just want to point out that a depth-limiter is a good design from the API design perspective for the parser. It is a simple API that allows custom granularity and really empowers the user of this parser.

[puzrin]

Existing depth limits in markdown-it are related to implementation. It's possible to make speed linear, not depending on nesting. Reference implementation did it. But we prefered to spend time for pluggable architecture and other more important things.

[aluminum1]

I would like to query something.

for non-react/angular2 systems, the re-paint during DOM update is always going to be more significant, therefore parsing time is not the bottle neck.

It seems to me that this is false, at least for big files. Consider the incremental DOM demo. If you copy and paste the codeblock on the left about fifty times, to make it into a big file that markdown-it has to parse, we find that there is significant lag each time you type a character. This lag is basically the same if we use incremental DOM repainting or inner HTML repainting (i.e. repainting the entire DOM). Therefore, the lag can't be due to the painting - it must be due to the parsing time. Markdown-it is having to reparse the entire document each time someone pushes a character, and this causes it to be very slow. Or am I mistaken?