Extract out prosemirror remark package

[marekdedic]

Hi, I'd like to use ProseMirror for my project, however, I'd prefer remark over markdown-it. After some googling I found this project and I se you have already done exactly that, if I understand it correctly.

Would it be possible to extract out a separate package for prosemirror markdown editing based on remark. Basically, an alternative to https://github.com/prosemirror/prosemirror-markdown, just with remark instead of markdown-it?

I'd be willing to help, but honestly, I don't even know where to start at this point.... :(

[benrbray]

Hello! Thanks for your interest! I can probably give you some pointers. Here is a high-level summary of how Noteworthy's markdown parsing&serialization are handled (actually, this will be a nice refresher for me, since I haven't touched the code in about a year :) ).

Before starting, you should be familiar with:

• the general structure of ProseMirror schema
• the unist / remark syntax tree format

Parsing & Editing

At a high level, here is how Noteworthy goes from from a (remark)-Markdown string to a ProseMirror instance that supports all the same syntaxes:

1. MarkdownAst.parseAST takes a markdown string and returns a MarkdownAst, representing an entire document.
   • see also src/common/markdown/markdown-ast.ts which defines the structure of a Markdown document
2. The parser is created from an EditorConfig instance.
   • EditorConfig.parseAST converts a string to an Md.Node abstract syntax tree representation
   • EditorConfig.parse converts a string to a ProseMirror Node instance, which can be used inside a ProseMirror editor (with the appropriate schema).
3. The parser used by EditorConfig is created by the makeParser helper function in src/common/markdown/mdast2prose.ts. This is where all the magic happens.
   • The md2ast parameter is just a regular remark parser, built here. For now, Noteworthy always uses the same pre-defined set of remark extensions to parse the markdown string into an AST.
   • After calling the remark parser to convert the string into an AST, the AST needs to be converted into a ProseMirror Node. The trouble is that there is not a simple one-to-one mapping between ProseMirror nodes and the Unist AST format. As a simple example, remark/unist represent bold/italic text as nodes in the AST, but ProseMirror represents them as Marks, which are like extra annotations on a node.
   • To deal with this complexity, each EditorConfig takes in a list of "extensions" which define a mapping between ProseMirror Nodes and REmark ASTs.
4. At the moment, there are only two kinds of extensions. Both kinds must declare a ProseMirror schema snippet, and can define new keyboard shortcuts and input rules.
   • Node Extensions: (defined in src/common/extensions/node-extensions.ts) Contribute a new kind of ProseMirror Node by implementing an instance of the NodeSyntaxExtension interface.
   • Mark Extensions: (defined in src/common/extensions/mark-extensions.tx) Contribute a new kind of ProseMirror Mark by implementing an instance of the MarkSyntaxExtension interface.

I think the best way to get an understanding for how it all works is to look at some of the existing node/mark extensions. Here are some things to pay attention to:

• The createMdastMap, prose2mdast, etc functions can all return some constant enum values to indicate certain default behaviors. For example, if there is a 1-1 mapping between a ProseMirror node and an Mdast node, the extension can use MdastNodeMapType.NODE_DEFAULT and Prose2Mdast_NodeMap_Presets.NODE_DEFAULT to supply the default behavior. The ParagraphExtension is a good example of this.
• An example of a more complex extension is the CitationExtension, which defines the mapping explicitly. The underlying remark extension comes from my remark-cite package.
• There is also the matter of YAML metadata. In a remark AST, the YAML metadata is part of the tree itself. In ProseMirror, the YAML metadata is represented as a document attribute. I'm not really happy with how Noteworthy currently handles YAML (and I hope to replace YAML with something more well-behaved like JSON) so I won't go into the details now.

(side note: As you can see, the division between node and mark extensions is very ProseMirror-oriented. An extension can contribute either a node or a mark, but not both. I think I chose this more out of convenience, especially since I was trying to use ProseMirror's schema type parameter to allow for richly-typed composable document formats with good type hints. But in light of #31, all of that will break anyway in new version of ProseMirror. So perhaps it is better to relax the distinction between node/mark extensions in the future.)

Serialization

At a high level, here is how Noteworthy serializes the contents of a ProseMirror editor into a Markdown string:

• Similar to parsing, each extension defines a prose2mdast function which converts the ProseMirror document tree into a Markdown AST.
• Afterwards, we use the same remark processor instance to serialize the Markdown AST into a string.
• See makeSerializer in src/common/markdown/prose2mdast.ts.

Comments

At the moment, I think the implementation is pretty ad-hoc and tangled up with some Noteworthy-specific stuff, but in principle it should be possible to factor all the remark-prosemirror interop into a separate package. I'm not sure it's something I have time for at the moment, but I would certainly welcome some feedback / PRs to help move it in that direction.

Hopefully this is enough to get you started. Please let me know if you have any questions and I'll do my best to reply!

[marekdedic]

Wow, thanks for the extremely thorough and informative response!

I hope to get started on trying this soon-ish, I'll post here if I get to it and you'll see if it makes sense :) I actually think this would make things like #31 easier for you as well...

[marekdedic]

Hi, I have started on this in https://github.com/marekdedic/prosemirror-remark, heavily drawing from your code, but at the same time re-architecting things to make more sense as a standalone package (and also I am using the updated ProseMirror types, so I am much more lax about types, at least for the moment...)

It is nowhere near done, but I have the necessary infrastructure in place to support both ProseMirror nodes as well as marks for both parsing and stringification. I would love to hear your feedback if you get any chance to look at it.