search

scripting / Scripting-News

I'm starting to use GitHub for work on my blog. Why not? It's got good communication and collaboration tools. Why not hook it up to a blog?
120 stars 10 forks source link

File-level metadata in Markdown? #95

Open scripting opened 6 years ago

scripting commented 6 years ago

I wonder if anyone's thought about a way to add file-level metadata to a Markdown document. In Frontier's website framework, we used a # to delimit a value, something like:

#title "My Test Page"

We have the same thing in HTML and OPML, in the <head> section:

<title>My Test Page</title>

The idea goes all the way back to C on Unix in the 70s:

#include "mymacros.h"

The directives are not part of the rendering. You don't see them when you read the document. But the values are available to software processing the document.

It seems since we're in The Age of JSON, something in JSON, delimited by a # might be appropriate?

metadata = {

title: "Hello World", tags: ["fun", "wisdom", "greetings"] }

This just came up in a project I'm working on, but it's not the first time I've encountered it.

tomraymo commented 6 years ago

Jekyll, for example, uses YAML for this purpose. Don't know if that works for your needs. https://jekyllrb.com/docs/front-matter/

scripting commented 6 years ago

@tomraymo -- very interesting. It's a different syntax from what we used in Frontier, they didn't have to appear at the beginning of the document (IIRC). Any line that began with a # was processed as a directive as the page is rendered.

I can already see this is going to be a productive thread. :-)

oevl commented 6 years ago

In a similar way to # directives in the Website Framework, the static site generator Hugo uses the concept of "Front Matter" and supports TOML, YAML, and JSON

https://gohugo.io/content-management/front-matter/

kantel commented 6 years ago

RubyFrontier does it exactly like the website framework. In fact it is a port of the Frontier's website framework to Ruby, written by Matt Neuburg. My blog entries look like this:

#title "Twine: Change Default Save Location – 20180908"
#maintitle "Twine: Change Default Save Location"

## Worknote: Twine -- Standard-Speicherort ändern

Die Desktop-Version von [Twine](cp^Twine 2) nutzt als (Zwischen-) Speicherplatz der einzelnen Stories
unter MacOS X das Verzeichnis `Dokumente/Twine` (`Documents/Twine`) im Home-Verzeichnis des
Nutzers. Das ist vor allem dann von Nachteil, wenn man zum Beispiel mit mehreren oder einfach auch
nur an unterschiedlichen Rechnern an einer Story arbeitet und diese zum Beispiel via [git](cp^git)
synchronisieren möchte. Unglücklicherweise ist dieses Verzeichnis in der Applikation fest verdrahtet
und eine Änderung auf Applikationsebene leider nicht möglich. …

Whereby #maintitle my own directive is (I use it in my template). To make an optical difference between directitves and headlines I decided to start only with second level headlines. But this is up to you.

RubyFrontier works with different flavours of Markdown. Out of the box it supports the original Markdown (written by John Gruber), Multimarkdown and kramdown. I use kramdown. But it is relative easy to install additional Markdown flavours.

bomberstudios commented 6 years ago

Another vote for Jekyll's Front Matter. It is well documented, and even if the YAML format is a bit alien, it's really easy to explain: "oh, it's just like Jekyll Front Matter, here are their docs : )"

karlcow commented 6 years ago

MultiMarkdown metadata? Or maybe I misunderstood the initial requirement.

scripting commented 6 years ago

I'm so glad I asked this question!

It's really gratifying to see that all CMSes evolved to solve this problem.

There's a natural flow to the design of these things.

It becomes even more interesting when the content lives inside an object database.

And here's the curious thing -- GitHub repos are basically object databases. ;-)

Very much like the structures we use in Frontier to form the storage system for the website framework we developed in Early Web Days.

emk commented 6 years ago

I've used Jekyll's YAML metadata before. It's pretty reasonable.

Pros:

  1. If a Markdown file contains Jekyll metadata, GitHub will render the metadata as a nice-looking table at the top of the file.
  2. GitHub Pages supports Jekyll, which makes it easy to publish a directory of static files as a website.
  3. Several other static site rendering tools support Jekyll metadata, so it's become a bit of a de facto standard.
  4. If you want to support lists in your metadata, you can write something like:
tags:
- programming
- thoughts

This will become ["programming", "thoughts"].

Cons:

  1. YAML looks easy to parse, but it's actually fairly complex. There are JavaScript libraries which will do all the work.
  2. Because YAML is relatively complex, YAML parsers have historically had a fair number of security bugs. This really only matters if you allow other people to upload YAML to server.
rahuldave commented 6 years ago

Pelican in Python also uses YAML frontmatter. Since jekyll supports it and provides a way for templates to get at the frontmatter, it can be used for many things. For example, I use it to support data downloads for by bayesian stats class. See http://am207.info/homeworks/AM207_HW1.html : the data download is supported in this fashion. Here is the markdown: https://github.com/AM207/2018fall/blob/master/homeworks/AM207_HW1.md

Here the markdown is produced by a custom template from a Jupyter notebook. When i need direct markdown support, I use a markdown editor such as Typora which has explicit YAML support...

kwebble commented 6 years ago

In a personal tool I've implemented this similar to HTTP headers, by separating metadata from content by 2 newlines:

key1: value1
another-key: the second value

The content of the page.

The tool knows about some keys, for example labels, and adds them in a consistent way when rendering the page.

scripting commented 6 years ago

Question about YAML and the format for "frontmatter" in Jekyll in GitHub.

The frontmatter is delimited by a pair of ---'s.

Can I assume the line delimiter is a \n?

In other words can I search for "---\n"?

Or do I have to allow for the possibility of a \r or a combination of \r\n?

I'm only concerned with how GitHub does this.

bomberstudios commented 6 years ago

Here's the REGEXP they use: https://github.com/jekyll/jekyll/blob/4271495fcaad4073b2375b76089041acd16ffb49/lib/jekyll/document.rb#L13

scripting commented 6 years ago

I've got YAML working with my GitHub as CMS experiment.

Here's an example of a post so you can see what it looks like.

This is exactly equivalent to using JSON, my server converts back and forth between YAML and JSON, so my app only ever sees the JSON.

BTW this post is a copy of a post I wrote in 1999 about Edit This Page functionality which was then working in Manila, our web CMS.

scripting commented 6 years ago

JavaScript code that converts between YAML and JSON and vice versa using js-yaml.