Templater should parse, merge and write out frontmatter gracefully

[McGlear]

Is your feature request related to a problem? Many of the recent bug reports and feature requests concern managing and updating frontmatter/properties. app.fileManager.processFrontmatter() has been proposed as a workaround for some of those problems, but has

• brought with it its own issues (especially regarding race conditions)
• failed to provide solutions for some of them (retrieving, parsing and applying frontmatter from the yaml block of a simple template file, for example)
• left most of the work to the end user

Describe the solution you'd like I propose that Templater implements parsing and handling of frontmatter properties, whether they are initially defined as objects (compare console.log(tp.frontmatter))

{
    "modified": "2024-05-29T16:36:31+02:00",
    "tags": ["source"]
}

or as YAML strings

`---
modified: "2024-05-29T16:36:31+02:00",
tags: ["source"]
---`

Usage Example: Modularization of Template files. For example, I might want to add certain properties to all notes on "Beliefs" to define how strongly I believe in something and why, and also add a little section to add further comments. Let's call that an Epistemology Module.

My note on a belief: Cows are mammals.md

---
created: 2020-12-12
tags: [#belief]
---
# Cows are mammals

I believe that cows are mammals. Because.

I only learned about epistemology statements in 2024, but have started adding them to all my beliefs:

epistemology-module.md

---
tags: [#belief, #epistemology]
credence: 60
effort: 30
---

## Epistemology statement
How strongly do I believe this? Hint: When creating a file, the default credence is <% tp.frontmatter["credence"] %>
How much effort did I put into this?
When did I last update this statement? <% tp.date.now() %>

I now want to tp.file.include("[[epistemology-module.md]]") in my belief-template.md template for new beliefs and also be able to Insert template into existing ones.

When I create a new file using epistemology-module.md, it creates a new file with the correct YAML block just fine, but still fails on inserting the tp.frontmatter["credence"] (because the frontmatter of the new file is not defined yet, even though it is written out in the template). When I add that epistemology-module.md to the existing belief, it inserts the YAML-block at cursor position, not processing it at all, and obviously still fails inserting the credence.

To fix this, all frontmatter that is handled during execution of a script should:

• Be parsed from YAML to an object (possibly replacing <% tp.stuff %> where appropriate).
  • This is important during tp.file.include(tFile) or when Inserting a template to an existing file when the template in question contains its own metadata block and ...
  • ... useful for filling tp.frontmatter during creation of a new file
• Be merged gracefully when pulling frontmatter from more than one source (due to a non-empty file, several tp.file.include, etc.):
  • new keys should be added
  • duplicate keys should be overwritten with the new value if that value is a primitive...
  • ... or merged if the value itself is an object/array
• Be written out as a unified YAML-block at the top

Upon inserting the module into the existing file, we should thus get:

Cows are mammals.md

---
created: 2020-12-12
tags: [#belief, #epistemology]
credence: 60
effort: 30
---
# Cows are mammals

I believe that cows are mammals. Because.
## Epistemology statement
How strongly do I believe this? Hint: When creating a file, the default credence is 60
How much effort did I put into this?
When did I last update this statement? 2024-05-29

Describe alternatives you've considered Currently, I use a haphazardly hacked-together mergeFrontmatter() function, write weird JS-Object definitions for my templates and modules (instead of simple .md templates) and then create a template literal containing the template to use for tp.file.create_new(). My module-definitions have a method() for updating an existing file with that module (e.g.: an excalidraw-module would be adding the excalidraw-plugin: parsed property using processFrontmatter() and the Excalidraw body %%\n# Excalidraw Data\n## Text Elements [...] by inserting that literal at the cursor position). I have not found a way to parse the template literals through Templater in the same step, yet (because only tp.file.create_new() accepts a string for a template and Templater does not currently expose its parse_template() function). Therefore, if my template or module definition includes eta-tags, I hit my "Replace templates in the active file" hotkey afterwards.

Additional context Both the built-in Templates-Plugin as well as the Excalidraw-Plugin parse templates of the format

---
excalidraw-plugin: parsed
---

# Excalidraw body ....

without any problems and can create a new file from such a template, or add that template to an existing file (merging the new key into an existing file's frontmatter).

[Zachatoo]

Accessing frontmatter on file creation (eg <% tp.frontmatter["credence"] %>) would require significant changes to how Templater works. Right now the template is executed as a whole using the rusty_engine parser, which is then returned to the Templater plugin to insert into a newly created file. That parser is written in Rust, and I do not know enough Rust to make any meaningful contributions there to support determining the existence of frontmatter before the whole template has been processed. To be frank, I doubt it will ever happen.

Merging of frontmatter is something more easily accessible, and currently being tracked here #1218. It's near the top of my list because as you've mentioned, without it, there's a lot of time spent on workarounds on the user's side.

[McGlear]

I am neither comfortable with rust nor with js (nor with any programming that goes beyond my very limited self-taught amateur level), so feel free to ignore anything beyond this sentence 😅: When using tp.file.include, templater appears to operate with different scopes/namespaces for the two files. Passing variables from one file to another requires to attach variables to the tp object (as in tp.uservariablename). I therefore assumed that parts of templates are sometimes isolated from other parts anyway (and found said solution to the problem of variable scope arising from that issue). That makes me wonder what would happen if, after merging frontmatter from the various files, one would split frontmatter (anything between the first two appearances of triple dashes surrounded by nothing but "\n") and content (everything else), then first parse frontmatter through rusty, in a second step parse the returned string via a yaml parser into a js object (and giving access to that object to the rest of the template through tp.frontmatter), then parse the content through rusty-engine, then concatenate a frontmatter string (using the yaml parser in the opposite direction) and content string.

[Zachatoo]

I can't merge the frontmatter up front, I have to do it as I process each file, since I have to process the whole file in order to know if there are any tp.file.includes(...) to process to go process other files.

---
key: value
---
<%*
const file = await tp.system.prompt("Template file to include");
// This could be any file, I can't know up front that I need to merge the frontmatter from this file with another file until the content is processed
tR += await tp.file.include(`[[${file}]]`);
-%>

There's additional complexity with trying to process the frontmatter before the content, or even determining what the frontmatter is without processing the whole or partial file. In the below example, <% "---" %> isn't the traditional way to denote frontmatter, but it's a relatively common practice I've seen among power users to keep the frontmatter in source mode for templates while staying in live preview mode. It'd also need to support reusing a value across the frontmatter and content by being able to define variables before or within the frontmatter and reference it in the content.

Template:

<%*
const value = await tp.system.prompt("Value");
-%>
<% "---" %>
key: <% value %>
<% "---" %>
# <% value %>

Result:

---
key: INPUT_VALUE
---
# INPUT_VALUE

I'm not trying to be discouraging or anything, I'm just trying to showcase that it is a difficult problem to solve, and there's not a lot of time that can be put into solving it, since we're all volunteers here 🙂 I appreciate your thoughts on this topic, it's made me consider some of these ideas that I hadn't before.

[McGlear]

I can't merge the frontmatter up front, I have to do it as I process each file, since I have to process the whole file in order to know if there are any tp.file.includes(...) to process to go process other files.

---
key: value
---
<%*
const file = await tp.system.prompt("Template file to include");
// This could be any file, I can't know up front that I need to merge the frontmatter from this file with another file until the content is processed
tR += await tp.file.include(`[[${file}]]`);
-%>

Ah, didn't think of that one, even though I use includes that depend on user input myself...

There's additional complexity with trying to process the frontmatter before the content, or even determining what the frontmatter is without processing the whole or partial file.

True, even though I'd say adding some constraints in the documentation and enforcing the use of either --- or <% "---" %> could still be acceptable - but people with other solutions might well disagree and of course trying to parse front matter differently could lead to issues with backwards compatibility, especially for more complex templates with heavy use of the scripting capabilities within templater.

It'd also need to support reusing a value across the frontmatter and content by being able to define variables before or within the frontmatter and reference it in the content.

Template:

<%*
const value = await tp.system.prompt("Value");
-%>
<% "---" %>
key: <% value %>
<% "---" %>
# <% value %>

Result:

---
key: INPUT_VALUE
---
# INPUT_VALUE

This one seems trivial (but I could be wrong): add the recommendation to use tp object key value pairs rather than more narrowly scoped variables, so the values remain accessible during the use of that tp instance. We have the same limitation with any includes: variables aren't accessible across template-files, even when declared as const. But they are when they are part of the tp object, and the tp object is writable.

Template:

<%*
tp.value = await tp.system.prompt("Value");
-%>
<% "---" %>
key: <% tp.value %>
<% "---" %>
# <% tp.value %>

Result:

---
key: INPUT_VALUE
---
# INPUT_VALUE

Still, you are right in that I thoroughly underestimated the effort required to manage all possible cases, especially regarding user input (and thus cases that can't be handled by a pre-processor of some sort).

[McGlear]

I'm thinking about closing my issue in favor of #1218, but currently consider the phrasing of that issue as too narrow: templater needs to not only merge current frontmatter of the note to be edited with that of a single template file, but frontmatter across all includes, too.

[Zachatoo]

Good idea. I'll update the other ticket to include some of the criteria you've mentioned here and then close this ticket when I'm done.