Support Frontmatter in MD files

[paulhibbitts]

Is your feature request related to a problem? Please describe. I'd love to use the Collaborative editor to share Page files for the Grav CMS (a flat-file CMS). Right now any Frontmatter (https://jekyllrb.com/docs/front-matter/) originally in the file seems to be re-saved in a different format. Frontmatter is used quite widely in both static site generators and flat-file CMSs.

Here is a simple example of Frontmatter for testing purposes:

---
title: 'Classic Modern Architecture'
date: '17:34 06/27/2017'
---

Here is the main content area of the MD file.

When saved, this is the result:

---

title: 'Classic Modern Architecture' date: '17:34 06/27/2017'

Here is the main content area of the MD file.

Describe the solution you'd like Any frontmatter contained in a MD should be visually distinct from the main content of the MD file, and also be editable (like a Code Block perhaps?)

Describe alternatives you've considered I've not been able to find any so far.

Additional context Add any other context or screenshots about the feature request here.

Thanks very much! The new collaborative editor is awesome to see in Nextcloud!

[juliushaertl]

Thanks for your suggestion. While front matter is not really specified in commonmark, I still agree that we should make sure that it is not changed. So we should make sure to strip it from the markdown parsing and add a simple code block for editing at the beginning of the document.

[paulhibbitts]

That sounds great @juliushaertl , thank you for considering this suggestion.

[brainchild0]

You could also consider rendering the displayable content, such as title, in appropriate styles. But somehow the user would need to able to add or to edit the other fields.

Also, not sure, but if the block is appearing disassembled after changes are saved, then maybe there is a separate issue with the way documents are processed. A MarkDown editor ideally should, I think, preserve as much literal content as possible from the original document, not attempt to produce an entirely new document from the logical features of the AST.

[hugoroy]

Hi, I think this feature is definitely important :-)

But I don't think it should be rendered as text like the rest. These blocks are supposed to be metadata (see, e.g. https://pandoc.org/MANUAL.html#extension-yaml_metadata_block which specifies it cleary, or https://jekyllrb.com/docs/front-matter/ which specifies it implicitly with the special rules IMO). The point is: you can have a lot more metadata in these blocks, not just "title" and "author" (I use them extensively to process with Pandoc). Trying to use them to render anything will probably lead to a lot of chaos…

So it probably requires a lot of work to get right from the UI/UX perspectives (the simple solution could be to treat it like a codeblock - but instead of inserting the codeblocks delimiter, use "---" and "---"; but this does not feel very satisfactory…)

[brainchild0]

But I don't think it should be rendered as text like the rest. These blocks are supposed to be metadata.

The point is: you can have a lot more metadata in these blocks, not just "title" and "author

The suggestion is not to render all metadata as stylized text, only the elements that appear visually in the document, such as title and author.

Naturally, other data must be shown in another manner. A collapsible code block would allow easy editing, while still offering a distraction-free view of the document that resembles final output with respect to display of the title and similar fields.

[dyamon]

I saw the issue and since this is something I'm quite interested in, I tried to sketch something. The first attempt is to use a foldable table layout

Folded:

Unfolded:

And following @brainchild0 suggestion, a version with some of the metadata rendered differently

Folded:

Unfolded:

I don't feel like this solution is intrusive w.r.t. to the goal of keeping Text distraction free. Let me know what you think!

Still, I have no experience with the code base and I have close to 0 knowledge of JS, so I don't know if this is even feasible.

That said I think simply fixing the formatting bug that breaks the yaml frontmatter should be enough for now, and having the yaml block printed verbatim at the beginning of the file is a "good enough" solution, that allows the use of Text with CMSs like PicoCMS.

[juliushaertl]

@dyamon Thanks a lot for the mockups, this looks great imo but let's bring @nextcloud/designers in here for some additional feedback.

Regarding the implementation, we probably need to implement our own frontmatter plugin for markdown it that then returns a node that can be parsed by tiptap. It could be based on https://www.npmjs.com/package/markdown-it-front-matter

[skjnldsv]

Looks cool to me too :)

[paulhibbitts]

Thanks very much for putting that together @dyamon , I guess the question is how frequently would users want to edit Frontmatter? I like the idea of making it less visible for users wanting to focus on actual Markdown content etc. Might "Frontmatter" be a more expected term in this context (vs. Metadata)?

[dyamon]

I guess the question is how frequently would users want to edit Frontmatter?

You are right, in my head the foldable section would appear only if the user inserts an opening and closing --- (or a closing ...) at the beginning of the file.

To put it even more "out of the way" for the user, the menu bar might be a better choice. Something like this maybe?

Might "Frontmatter" be a more expected term in this context (vs. Metadata)?

Yes definitely, I originally put "Metadata" as a (possibly more user-friendly?) placeholder. I guess the discussion is the same in the menu bar example regarding which icon to use to indicate "Frontmatter"

[brainchild0]

I like visual representations submitted by @dyamon.

I make a few comments:

1. I most like the final image, which invokes my earlier suggestion of a formatted display of relevant fields.
2. If the metadata are shown in a table format, I would suggest distinctive formatting, such as shading behind the text or along the margin.
3. The representations provided are not distracting, but I would personally favor a version of the collapsed representation that does not add vertical spacing. Possibilities include a simple, small arrow, without the word metadata, just above the body text, or the collapsed representation being located in the margin, outside the region used by the body text.
4. Many metadata blocks will be simple, flat tables, but MarkDown processing applications support metadata with complex formatting that includes nested tables and lists. Further, YAML is a detailed standard, and many features rarely used by end users must still be handled by conformant processors according to specification. These considerations point to the need, at least in some cases, for editing of the YAML data as plain text.

---

Might "Frontmatter" be a more expected term in this context (vs. Metadata)?

Yes definitely, I originally put "Metadata" as a (possibly more user-friendly?) placeholder. I guess the discussion is the same in the menu bar example regarding which icon to use to indicate "Frontmatter"

I would understand frontmatter to be an informal synonym, in this context, by analogy to book publishing, to the more precise term metadata. If not, the distinction needs to be clarified.

[nathanlesage]

Thank you for considering this! In fact, I had one file completely ripped apart due to this (P.S.: I think, https://github.com/nextcloud/text/issues/593 is related). While I certainly like the looks of the new editor, this is definitely a deal breaker, as I would like to emphasise that indeed frontmatters are quite common among Markdown editors. In fact I don't know any (Desktop) app that does not support frontmatters.

Furthermore, I think a simple non-parsing codeblock at the top would be quite sufficient, especially if it takes more resources to render the frontmatter nicely (which would be a cool addendum). Because the fact that it should not break it is definitely much more important.

[paulhibbitts]

Any updates on this possible enhancement @juliushaertl ? This would really widen the uses of the Collaborative editor ,and I am really jazzed about using it along with Grav CMS!

It also looks like others are trying to edit Markdown files with Frontmatter with the new editor: https://twitter.com/zettlr/status/1225449750411010048

[ghost]

I can confirm that @paulhibbitts idea is great! Page-sync with external storage works seamlessly with grav cms so having the frontmatter issue solved would be awesome, because right now pages look like this if you do it :). So having the frontmatter being interpreted correctly would be a really awesome solution.

[jancborchardt]

Agreed, @dyamon’s mockups are very nice! :)

Since we always do it step-by-step, some considerations:

1. We should first start as @juliushaertl mentioned and simply exclude the Frontmatter part to not mess that up. That is, still displaying it to be edited.
2. When that works reliably, we can look into collapsing it. In-place seems best, as the tag/metadata icon in the heading might be missed. And especially when you do use these tags, they should be visible.
3. Then we can check to work on a table-based representation of the content. :)

[mcnesium]

Just pointing out that there are similar issues elsewhere with suggestions to integrate existing third-party modules, which might be worth to take a look at…

[brainchild0]

I think that overall the stepwise approach proposed by @jancborchardt is very sound.

I make a few slight comments:

• Perhaps it is best to stay open to the possibility of one or two additional further steps, which might include formatting the title information such as it might appear in a published representation.
• The metadata block would not in all cases consist of a table with only primitive values, but rather might include values of arbitrarily nested structures such as lists and further tables. It is important to have a sense of how data from the general YAML data model might be displayed and might be edited in each stage such as to preserve optimal usability and to minimize destructive effects for editing documents from other sources.
• Perhaps approaches other than inline folding would be more aligned to the WYSYG spirit. As the word metadata appearing inline is not included in what you get, perhaps it equally ought not to be included in what you see. A button with a popup window might be a more suitable metaphor, and I personally have yet to be persuaded by any objections offered against this approach.

[ghost]

This is a very cool and needed addition to the text editor tool!

Are any of you familiar with Netlify CMS? It's a lightweight GUI that works on top of multiple static-site platforms by simply specifying config.yaml and then it renders various form fields like:

In addition to the nice collapsible rendering by @dyamon perhaps the text editor could also borrow inspiration here and offer form fields to edit the values:

title --> text input 
date --> date picker
lang --> dropdown
ingredients --> tokenized elements in text input

If anyone wants to work on this, has some available time,and needs a small budget to work with- I am part of a collective that allocates funding to open source projects that we use. We would be happy to explore funding some dev help on this issue, read more here.

[Spartachetto]

I'd like to remember that YAML headers are used also in other file formats.

I tried to advocate for support of Rmarkdown, a markdown file used by R language. That file format is markdown + YAML header + code snippets.

If you support YAML headers, it would be enough, IMHO, to show the code snippets as code (without any kind of syntax highlighting...!) and we'd have a gorgeous support for Rmarkdown files outside a classic IDE.

For reference https://github.com/nextcloud/viewer/issues/783#issuecomment-785957462

[claell]

I agree to @nathanlesage that at a short look this seems at least to be related to #593. Maybe even a duplicate, can somebody check (@paulhibbitts for example)

If this is a duplicate, I suggest continuing in #593. Although this issue is older, the description of #593 is broader.

[dyamon]

@claell I would say the two issues are addressing different problems.

The one you mention is talking about the fact that the editor is not able to keep the structure of a md document (as plain text), and coincidentally this seems to mess up YAML frontmatters as well.

This issue is asking for a way to have metadata (included in the frontmatter) to be somehow integrated in (or at least understood by) the editor.

Edit: clearly solving #593 would be a good start for this issue but that's not what we are asking for. I would also say that the issue you mentioned has higher priority. I don't want to sound harsh, but I feel like a markdown editor that is not able to keep the plain text formatting is close to useless.

[claell]

@dyamon I believe that the original comment of this issue was mainly (but maybe not only) about having the plain text changed when saving the file. This issue then drove a bit towards also rendering frontmatter correctly.

Now we can either repurpose this issue to only be about rendering and treat everything related to changes on saving in #593 or we mark this one as a duplicate of #593 and create a new issue about correctly rendering frontmatter.

[dyamon]

I believe that the original comment of this issue was mainly (but maybe not only) about having the plain text changed when saving the file. This issue then drove a bit towards also rendering frontmatter correctly.

Actually, you are right. I based my comment on the title of the issue and how I was interpreting it.

Now we can either repurpose this issue to only be about rendering and treat everything related to changes on saving in #593 or we mark this one as a duplicate of #593 and create a new issue about correctly rendering frontmatter.

I most likely won't be able to help with the issue, so I'll spare you my 2 cents on this :smile: I guess it's a matter of convenience for the people you are going to tackle this.

[brainchild0]

A solution strictly for #593 would seem unlikely to add any support for front matter, understood as a YAML block which interrupts Markdown processing. The issues overlap, but are separate, to my mind. This issue is valuable for the original purpose. Rendering is a further topic that may be addressed in a separate issue topic if necessary, but I find nothing confusing or limiting about the discussion so far, mostly an open exploration of enhancements related to the original concept. I suggest keeping all the issues open as they are, with the benefit of the earlier note that this issue is indeed related to the other.

[paulhibbitts]

If #593 also addresses the issue of saving YAML as expected (i.e. correctly) then that would address my initial reason for this issue. Related to that, as has been discussed on this thread, is then a more effective presentation of that YAML for editing.

I would tend to keep this issue open after #593 has been addressed, with the next step being better presentation of YAML for editing - thoughts?

[brainchild0]

If #593 also addresses the issue of saving YAML as expected then that would address my initial reason for this issue

I am not sure that #593 does address saving YAML properly, as the implementation seemingly would depend, as distinct from text in Markdown syntax, on special processing of a YAML block, which is not discussed in #593.

One difference to consider is that while a characterization of saving a document properly may be obvious to a human such as one of us, entering the question with an understanding of the appearance and relevance of a YAML block, the application would not necessarily perform as we desire without an implementation of logic particular to this case, that is, the ability to separate the YAML from the Markdown. Thus, closing this issue in favor of #593 (which I realize you specifically are not suggesting) would seem to me to leave no open request for this new logic, which is needed to achieve the effect you have requested.

[paulhibbitts]

Thanks for your additional comments @brainchild0 , I would agree that this issue highlights the specific use case of YAML support vs. other expected formatting upon Save etc.

[claell]

I am rather confident that at least the issue with inserted or removed line breaks when saving is covered by #593.

But I am not certain about that and there might be additional issues involved when it comes to saving YAML. Additionally, this issue also tracks rendering and editing of YAML.

So I guess this just stays open as is.

[brainchild0]

I am rather confident that at least the issue with inserted or removed line breaks when saving is covered by #593.

Why? YAML has completely different syntax from Markdown. If a document is processed by an application unaware of the former, then how could we make any reliable prediction to such an effect as yours?

[claell]

Because there already is an issue with the application removing (or messing with) line breaks in "Markdown syntax" sections. Basically it should not mess with existing plain text at all and this is currently not the case. For that the underlying syntax of a section does not matter at all.

[brainchild0]

Basically it should not mess with existing plain text at all and this is currently not the case.

Does the application currently identify the region of the YAML block as one to rewrite verbatim as appearing in the source?

[claell]

Apparently we are talking about different things here. I was talking about not altering existing YAML code, while you seem to be talking about editing that (or creating it).

For simply leaving YAML code in Markdown files intact (i.e. untouched when the document is saved), I guess #593 can help (although its solution might not fix this). But I agree that it gets out of scope for that issue when it comes to editing or adding that.

[brainchild0]

Apparently we are talking about different things here. I was talking about not altering existing YAML code, while you seem to be talking about editing that (or creating it).

No, I am also referring, in my recent comments, to the idea of not altering the YAML block (hence "rewrite verbatim"). My earlier question points to the possibility that an application might leave segments of Markdown unaltered, but not similarly protect a YAML block, because of confusion over an assumption that the the YAML is rather some kind of Markdown, which it, under certain conditions, may attempt to alter.

Thus, one question is whether users currently experience problems of the kind originally reported. Yet, the broader issue is that a robust solution depends on recognizing the YAML block as a region of the document not subject to Markdown processing, and that realizing this solution depends on specific logic for detecting the YAML block.

This issue topic is the only currently open that directly indicates the need to include logic that guards the YAML block from any accidental effect, which might occur, without such logic, through some later change to the application, even if such an effect is not occurring currently.

[claell]

Alright, thanks for clarifying that.

In that case, I think #593 is at least partly suitable, else we probably want to create a separate, more broad issue for this, as this is not only affecting YAML blocks, but all kinds of other segments that might reside in a Markdown file like LaTeX for example.

At least I hope there can be a solution that does not aim to first detect different content types in order to protect them afterwards, but that plain text is not altered in general at all if sections are not touched in the editor.

I can see however, that in the long run detection of YAML blocks is a good thing (to render them for example), but I'd assume that would be later down the road.

[brainchild0]

At least I hope there can be a solution that does not aim to first detect different content types in order to protect them afterwards, but that plain text is not altered in general at all if sections are not touched in the editor.

This distinction may be illusory. What you are calling "plain text" is simply document sections that the user wishes not to be processed as Markdown. The application is a WYSIWYG editor for Markdown, which means that it must follow one of two designs. First, it may treat the entire document as Markdown. Alternatively, second, it may invoke a method for detecting certain sections as not Markdown. I believe the current design corresponds to the first option, which, as argued, is not a solution for the desired behavior.

I suggest there is no possible solution that treats Markdown as Markdown, and YAML as not Markdown, but cannot detect the difference.

[susnux]

It should be mentioned that this would or might break commonmark support, as without a newline between text and the second --- it would be interpreted as a setext heading.

With a empty line instead it would be interpreted as thematic break.

[brainchild0]

@susnux: Yes, it is true, but the pattern is common, and normally handled by a special case. Many processors handle the exception seamlessly.

[xplosionmind]

I am sorry: the issue is closed, but in my Nextcloud instance front matter is messed up anyways… might there be something I am doing wrong?

[tcitworld]

The issue is closed because it's fixed in the next major version of Nextcloud.

[hippi-viking]

It does not seem fixed in Nextcloud 25. :(

[susnux]

It does not seem fixed in Nextcloud 25. :(

@hippi-viking sorry to hear, would you mind to provide your test file and the file after saved by the text app?

[hippi-viking]

@susnux In the meantime i figured it is still a known and open issue that Text strips the YAML header from Markdown files - even if the file is only opened for reading. Problem is that this breaks PicoCMS. Historically Simple text editor was recommended to be used, but that is also broken under NC25, so at the moment there is no working text editor in NC for editing PicoCMS Markdown files. For a test file if you type in the following 3 lines and later try open it it will be stripped: --- Header: test ---

[susnux]

I could reproduce this: On the first time opening the file it works (I opened a file containing already a frontmatter). After editing and downloading the changes are correctly saved, but when opening the file again the frontmatter is stripped. I have an idea whats going on there, I will look into this but probably not before (mid of) next week.

[xplosionmind]

I could reproduce this

Could the issue be reopened?

[hippi-viking]

@susnux Thank you for looking into this!

@xplosionmind I think it is not necessary, it is a known issue as per https://github.com/nextcloud/cms_pico/issues/116 .