Parser

[tangjeff0]

This feature is for parsing block-level syntax – e.g. • here is [[a link]] – and rendering it to the correct HTML via Hiccup.

We use the instaparse library.

This file may be useful for nested links: https://github.com/athensresearch/athens/blob/master/data/nested-link.hiccup

5

25

44

[roryokane]

My current knowledge of what needs to be done:

Exploratory tasks

• [X] Interpret unknown syntax as regular text, not a parse error (#44)
• [X] Basic parsing of links
• [X] Figure out how to parse backslash-escaped characters
  • The url-link rule shows how to do this.
• [X] Figure out how to parse nested formatting
  • The bold rule shows how to do this.
• [ ] Figure out whether a way to convert parsed ASTs back to markup is necessary, and how to design it if so (related to #44)
  • As discussed on Discord, updating page links ([[foo]]) after a page rename currently uses patterns.cljc. I see that the regexes in patterns.cljc will fail in some cases, so ideally the parser could be used to find and replace page link targets. Note that this use-case wouldn’t necessarily require being able to serialize all AST nodes, only AST nodes that could be within page links.
  • One of my two ideas for parsing nested links (#5) would require this feature. If the other idea is used, it wouldn’t be required.
  • This could cause problems in one of these two ways, depending on the implementation:
    • The parse tree might have to remember which syntax was used to construct a given meaning, e.g. #foo vs. #[[foo]] and \(a\) vs. (a) in a URL. This would complicate the parser.
    • The serialization might just arbitrarily choose one of the possible syntaxes, causing the user’s markup to be reformatted. The markup would render the same but have different raw text from what the user inputted. This has potential to go wrong if the user types a syntax error that would auto-format away the wrong parts of their syntax.
• [ ] Figure out whether we need to pass the :start-index and :end-index of each parse rule all the way through each transformation to the final AST, and how to design it if so.
  • As discussed on Discord, this might be necessary to place the text cursor at the point of the clicked text when converting a rendered block into a textarea after a click.
• [x] Parse nested links (#5)
  • Example Hiccup output: data/nested-link.hiccup
  • I see that Roam is missing a feature regarding this: it doesn’t highlight which link you will click when you hover. Can that be implemented with the Hiccup output above, or will we need a different structure?
  • Is there any way to trick the browser into using a non-nested a element to do the link-hover style change and link-click page change, so we don’t have to reimplement that in JavaScript?
• [ ] Compile a list of all markup syntax Roam supports
  • Roam has the Help popup in the bottom right and also the Help database. But neither of them mentions all syntax: in a Roam user’s YouTube video, as well as in the Athens example database, I saw undocumented syntax foo:: that turns into “foo:”. Research: is there other undocumented syntax like this?

Straightforward tasks

Implement known syntaxes

I think these tasks should not be attempted until most of the exploratory tasks are done, because some of those exploratory tasks would require writing new code for every rule, and those tasks would be more tedious if more rules had to be accounted for at that time.

For each of these syntaxes:

1. Research Roam’s handling of edge cases by writing sample text in Roam
2. Decide on Athen’s desired handling, possibly taking cues from other markup syntaxes like CommonMark
3. Implement the parsing and initial transformation in parser.cljc. Remember to update any other parser rules that allow the new syntax to be nested inside it.
4. Write tests for that rule in parser_test.cljc.
5. Implement rendering of that markup as Hiccup in parse_renderer.cljs.

• 1. [ ] parsing only
  2. [ ] rendering the embedded block
• [x] ![image](http://example.com/foo.jpg) syntax (#25)
• [X] **bold** syntax
• [ ] __italics__ syntax
  • Will be implemented similarly to bold.
  • Something I was thinking about and didn’t bring up for discussion yet: would we rather eventually implement Markdown-compatible syntax, so italics would be _this_ and *this*? If so, should we implement Roam-compatible syntax now and write a converter later, or are there few enough Roam users that we can start with Markdown-compatible syntax?
• [ ] ~~strikethrough~~ syntax
  • Will be implemented similarly to bold.
• [ ] ^^highlight^^ syntax
  • Will be implemented similarly to bold.
• [x] `code` syntax
  • Won’t allow nested syntax. But figure out how ` escaping works, if it’s supported.
• [ ] ```code block``` syntax
  • In Markdown, the text on the same line as the first ``` is a language identifier, but in Roam, it’s the first line of the code. Implement Roam syntax to start with since it’s slightly easier, but what’s our long-term plan?
• [ ] $$LaTeX$$ syntax
  1. [ ] Parsing text between the $$ delimiters
  2. [ ] Integrating a LaTeX library to format the contents
• [ ] header:: syntax
  • This syntax is undocumented in Roam’s help.

[tangjeff0]

Thanks for collecting all these thoughts together @roryokane.

Before I respond to your comments, my first question is: what are your thoughts on putting tests cases or examples in Devcards?

We need not actually do unit tests, and of course, the parser should still be cljc, but it would be easier for me to reason about example cases if I could visually identify things like markup and links in addition to the string inputs and hiccup outputs.

In general, I'm having some difficulty making mental models for these parsing problems, especially for the two ideas regarding nested links: remembering vs reconstructing source text. I believe your explanations are good, but parsing is just kind of an abstract subject to me. End-user output would be more user-friendly, and I think it would encourage others to start thinking about these tricky parsing problems!

Lastly, I think ultimately the designers will need to design things like nested and bolded links, bolded and highlighted text, and tranclusions of nested, bolded, highlighted, italicized text! It might be useful to use Devcards in the development of the parser, not just at the end 😄

---

I see that Roam is missing a feature regarding this: it doesn’t highlight which link you will click when you hover. Can that be implemented with the Hiccup output above, or will we need a different structure?

In Roam's case, the inner link is contained within a link class. Inner links can thus be distinguished by custom CSS. So if we have a similar DOM and CSS class setup, our designers can handle this UX.

Can't distinguish with default CSS

But with custom CSS you can easily detect the inner link

---

should we implement Roam-compatible syntax now and write a converter later, or are there few enough Roam users that we can start with Markdown-compatible syntax?

We can always worry about converting and accreting later. Basically using the same logic you pose here:

I think these tasks should not be attempted until most of the exploratory tasks are done, because some of those exploratory tasks would require writing new code for every rule, and those tasks would be more tedious if more rules had to be accounted for at that time.

---

In Markdown, the text on the same line as the first ``` is a language identifier, but in Roam, it’s the first line of the code. Implement Roam syntax to start with since it’s slightly easier, but what’s our long-term plan?

Short-term, maybe all we do is syntax highlighting Roam or Markdown style.

Long-term, I know I would like a REPL experience, where Athens replaces Emacs/VS Code and we can run arbitrary code in code blocks. We already have sci and datascript boxes :). But this is kind of v2 stuff.

---

header:: syntax

This is easy to implement on the parser side, but not on the Datascript, so maybe not a huge priority.

[tangjeff0]

Added basic devcards in #151, viewable at https://athensresearch.github.io/athens/cards.html#!/athens.devcards.parser

[roryokane]

I found this page that attempts to document Roam’s syntax: roam-tricks – Roam Shortcodes. Some of the syntaxes described there I hadn’t been aware of:

• [alias]([[link]]) – a combination of the syntaxes that Athens currently calls url-link and page-link
• {{foo}} codes
  • Each individual template, such as {{e}}, {{iframe: URL}}, or {{[[DONE]]}}, should have its own parsing logic so that it can capture the arguments it needs.
• : codes
  • :diagram
  • :hiccup (e.g. :hiccup[:code 'code'])

I also learned of RoamTutorial – Secret & Advanced Features, which doesn’t reveal any syntaxes the other page doesn’t also include, but does go into more detail on how they are supposed to be rendered.

[tangjeff0]

Alias is pretty cool. {{[[TODO]]}} and {{[[DONE]]}} are pretty important. :hiccup we can probably support quite easily with https://athensresearch.github.io/athens/cards.html#!/athens.devcards.sci_boxes by @tomisme !

[tangjeff0]

Writing this so I don't forget: recursive block refs lead to stack overflow and app freezes.

[tangjeff0]

We also want to auto-create pages with # and #[[]] @HaojiXu

[thesophiaxu]

Related: https://github.com/athensresearch/athens/issues/44#issuecomment-636139109

[tangjeff0]

Also are pages created are inner links auto-created? i.e. [[nested [[links]]]] should created [[nested [[links]]]] as well as [[links]].

[thesophiaxu]

Also are pages created are inner links auto-created? i.e. [[nested [[links]]]] should created [[nested [[links]]]] as well as [[links]].

@tangjeff0 Yes, they are auto-created in earlier commit.