Gracefully hide annotation blocks in comments

[DivineDominion]

Hey,

not sure if you want to discuss this, but since it's on GitHub and Issues are allowed, here you go :)

The block layout of your example

---
Annotations: 0,95 SHA-256 1132bf5e376a605f5beed4b204456114  
@Human: 0,20 33,4 45,6 62,4  
&AI: 20,13 37,8 51,11 66,29  
...

reminded me of local variable settings in editors like Emacs.

There, the local settings or metadata blocks compose more gracefully with the content (historically: source code) by using comments.

I think using HTML/Markdown comments for these annotation purposes would compose better without breaking text rendition of all other Markdown tools that don't use the Markdown-Annotations "extension" to the syntax.

1. No breaking changes.
   • Every tool that supports comments in Markdown can deal with the annotation blocks.
   • Strategically, that could help with adoption.
     • In theory, at least, because in practice, this annotation feature soft-locks users in: Tools that don't support this won't update the character ranges, so editing in a potato text editor will render the perfectly fine annotations from before useless.
     • But if users remain in iA Writer 7 for writing, their exporters will benefit from not having to understand a new syntax. Users can still use tools like Marked or Pandoc to export to HTML, PDF, ... without having to remove the block manually, first.
2. Convention over re-invention.
   • Using existing syntax plus a convention for special features instead of inventing new syntax elements altogether makes adoption easier for app developers.
   • It's difficult work to change an existing Markdown parser syntax: no matter if it's tree-sitter based, or a CommonMark spec compatible one, a parser generator like Bison/Yacc/..., or some ancient home-cooked C parser, introducing changes to the weird grammar of one's Markdown dialect of choice opens a large can of worms and potential bugs.
     • People working on parsers don't mind as much. But the casual iOS app maker relying on open source libs does.
   • It's quite easy to keep the parser unchanged and add a post-processing step to check all comment blocks in the document for annotations. This code can live inside the app space, outside the parser.

Here's a re-interpretation of your syntax proposal that doesn't require a Markdown syntax/parser extension at all:

• replace the --- and ... block markers (only one of which denotes a Markdown element, a horizontal rule) with <!-- and -->, removing the part from the Markdown text in all tools that respect HTML comments
• Keep the key-value pair style of the contents
• start the comment block's opening line with a marker for parsers, e.g. Document Settings:, so that parsers know where to look (and which comment blocks to ignore)

Example:

<!-- Document Settings:
Annotations: 0,95 SHA-256 1132bf5e376a605f5beed4b204456114  
@Human: 0,20 33,4 45,6 62,4  
&AI: 20,13 37,8 51,11 66,29  
-->

• It's a regular, valid HTML multi-line comment block.
• Apps can iterate over all detected HTML comments and filter for those starting with literally "<!-- Document Settings:\n", ignoring the rest, to interpret as annotations.

Alternatives considered

YAML frontmatter

Obvious contender for metadata in general.

• Adding 'backmatter' makes range computations simpler than any frontmatter (independent of YAML):
  • If you just type a single letter at character position X, you can't start counting at the beginning of the document. You need to start after the frontmatter. Otherwise, madness ensues:
    • Typing at position X = 0, the frontmatter will need to denote a range of 0,1 as human input.
    • This range itself is 3 characters wide, plus whitespace etc., shifting the position that the user typed at in the document.
    • Potentially you need to add all the other lines to the frontmatter as well for the first edit. So the value of the position X is shifted to the right by the whole delta of the frontmatter.
    • So if X was 0, and the frontmatter change to denote the range 0,1 but also the checksum etc. is 100 characters total, the actual position of X is now100`.
    • This in turn affects the frontmatter having to change to 100,1, which shifts X two more characters to the right
  • Ignoring the frontmatter in character counting is the only sane way :)
• Using YAML has better platform effects/benefits
  • if tools already support YAML frontmatter, it's easier to interpret 3 new keys (Annotations, @Human, &AI) over adding support for a backmatter block;
  • if tools neither support YAML frontmatter nor annotations in the backmatter, adopting YAML would yield higher returns because they unlock compatibility with many other tools, whereas spending engineering time to adopt the backmatter syntax produces compatibility with iA Writer 7's current annotation feature only
• The iA announcement mentioned that the backmatter approach gets "out of the way". I guess that's why it's put at the end.
  • But tools that do understand the annotation can opt to hide in in the UI.
  • Tools that neither understand YAML frontmatter nor your annotations will break the annotation ranges, so the only benefit is that the user doesn't have to scroll past it as they break it.

[dkgrieshammer]

The Idea itself is very good (as always). I wonder if not a .git that tracks authorship by nature would be an option; this is usable in quite a lot of editors already and allows tracking of various sources;

[deletosh]

THIS.

The plain-text writing word are already converging on standards and extending off it: HTML and YAML for metadata.

I suspect the focus of devs should be on implement/adopting the annotation core innovation here not re-inventing wheels.

If this follows this core proposal, I already see how it will gain instant adoption in tools like Obsidian (that already has this standard, not it's implementation) and several others thought capture apps.