Write recipes in YAML for simpler verification and site building?

[AJ-Williams1]

You said in #444 that you were open to a different way of building the site. I know you were probably looking for a more drop-in replacement for blogit, but there are a lot of advantages to writing recipes in YAML, and I think that the slight added complexity up-front will be worth it for increased maintainability in the long run.

Why not just blogit?

While I respect the philosophy behind blogit, and its simplicity, the way that it does things is . . . messy. The code is difficult to read, and even more difficult to extend. As you noted, it is also quite slow. In addition, as has been mentioned in a few issues, because markdown is not machine-readable, it is difficult to programmatically verify whether a recipe follows the guidelines.

Why YAML?

Using YAML means we treat recipes like data rather than like formatted text, which makes sense since a recipe really is data (a list of ingredients, a list of steps, a number of minutes to cook, etc). YAML also strikes a nice balance between being human-friendly (easy on the eyes and easy to write by hand) and machine-friendly (can be parsed and generated easily from any language).

My suggestion

I would suggest we use YAML as the format for all future recipes, and convert all existing ones to YAML. Most of the conversion could be done automatically, as almost all existing recipes follow the guidelines. There are, last I checked, about 30 that have sections outside of the guidelines laid out in example.md, more than half of which could be fixed by adding a notes section to the standard. With all recipes in YAML, a simple home-grown static-site-generator would be needed, but creating one would be relatively easy with the recipes already being data.

I am willing to do all the conversion of existing recipes, and to write the preliminary static-site-generator. See later section for what I have done already.

Pros

• There are also mature libraries for reading and writing YAML for almost every programming language, so there would be no need to manually parse recipes with the likes of sed and awk.
• Because it is machine-readable, recipes can be easily checked for whether they follow the guidelines, which saves you huge amounts of time looking over every single one.
• Since the recipes are read into whatever language as data, it is easy to work with them, so simple mistakes (like capitalization or misspelling section titles) could be fixed automatically with no human intervention.
• Again because the recipes are data, exactly how the final HTML gets formatted can be easily adjusted, so if you ever want to change how the recipes are laid out, it is easy to do.
• Having a home-grown static-site-generator means it can be as minimal as you like, with only the features we need.
• Having a home-grown static-site-generator is also more extensible, as any desired features can be added directly into it rather than being patched on top.
• YAML gives people less freedom to format things the way they like, which makes recipes more standard.
• It is somewhat easier for people to follow the standard, e.g. they don't have to worry about ordered vs. unordered lists, that is handled when the YAML is converted to HTML.
• This system would be more maintainable as the site grows.
• Having recipes stored this way makes it easier to implement something like the search function mentioned in #153, and it would be trivial to format recipes for the terminal without having to run the html through lynx.
• If desired, full markdown syntax could be supported within the YAML sections, as each section could be converted to HTML using something like pandoc. Then the YAML sections would each be strings of HTML that could be trivially inserted into a template.

Cons

• Converting existing recipes takes time and effort
• Changing how recipes are written might be jarring for some people (though the website is new enough that it probably wouldn't be too bad)
• YAML gives people less freedom to format things (though, as noted above, this could also be seen as a pro)
• Writing a home-grown static-site-generator takes time and effort.

What I have done already

• I translated your example.md file into YAML, so you can see what it might look like: see it here
• I wrote a proof-of-concept python program to check the validity of recipe. Obviously it needs revision, but it at least demonstrates how easily you can verify things: see it here

[DGKSK8LIFE]

Converting existing recipes takes time and effort Changing how recipes are written might be jarring for some people (though the website is new enough that it probably wouldn't be too bad)

Disagree. YAML is quite high level; to change a recipe you would just go in and edit a key-value pair. I think YAML could be the ideal technology if we were to build a home-grown SSG; the main problem, as you mentioned, is formatting. At the same time, formatting for most, if not all the recipes is quite homogenous anyways (and desirable). What I would do is use a prexisting YAML parsing lilbrary, iterate over a YAML file, look for certain keys (for example Ingredients:, then convert those values into html elements. This would allow for greater recipe uniformity and make automatically auditing submissions a hell of a lot easier (For example, I could write an action that looks at PRs, ensures that all sections like Ingredients are present, then approves from there, etc).

[DGKSK8LIFE]

One thing to note is that Markdown can also be parsed and is translated into HTML when rendered on the web, so we could do something similar to what I described above but with MD instead of YAML, though with HTML and Markdown there are a lot more elements to sanitize and check than YAML.

[AJ-Williams1]

What I would do is use a prexisting YAML parsing lilbrary, iterate over a YAML file, look for certain keys (for example Ingredients:, then convert those values into html elements. This would allow for greater recipe uniformity and make automatically auditing submissions a hell of a lot easier (For example, I could write an action that looks at PRs, ensures that all sections like Ingredients are present, then approves from there, etc).

Yes, that was the idea, and is what my preliminary python program does. I didn't want to do this as a github action because there has been discussion of moving to a different platform, so I don't want to become dependent on github.

[AJ-Williams1]

Also, if this is the direction Luke wants to go, we should make the switch sooner rather than later, because the number of recipes to convert will only grow with time.

[DGKSK8LIFE]

Also, if this is the direction Luke wants to go, we should make the switch sooner rather than later, because the number of recipes to convert will only grow with time.

Agreed. I think we should automate the conversion system regardless of the preexisting recipe size atm.

[tempersama]

I tried converting the salsa-verde.md into yaml and ran it through the script linked below. And I was able to at least get the same HTML structure that is shown on the actual website right now. Its missing any form of styling right now but would be trivial to add + its only 1 python script for YAML to HTML compiling.

There's also some basic validation that the script enforces right now:

The following keys must be included:

• title - String
• description - String
• ingredients - List
• directions - List
• notes - List
• tags - List

Optional keys:

• images - List
• contribution - String or List

This should make it easier to get some kind consistent structure across all new articles since it won't compile if YAML doesn't meet the requirements. Also right now it just writes to standard-out.

https://gist.github.com/anon-temper-sama/8e412a7d4ac1df38a1db921184d6b9b1

[TTWNO]

Agreed. I think we should automate the conversion system regardless of the preexisting recipe size atm.

A few hundred isn't too bad. I will do 50 (just at me if something is agreed on). YAML would (likely) enable automatic conversion afterwards if we decide to go to another new format later, so the effort is easily worth it.

If we are going to use a new format, then we should also set the measurements as separate things IMO. Having them be computer-readable would help long term.

i.e.

- amount: '1 1/2'
  measurement: 'cups'
- amount: '2.5'
  measurement: 'tbps'

I'm sure there's already a library for cooking measurements to be automatically converted to some internal data type for export, not that it would be hard to implement manually.

I know this has been over-talked in some other threads, but worth mentioning.

[AJ-Williams1]

@anon-temper-sama

https://gist.github.com/anon-temper-sama/8e412a7d4ac1df38a1db921184d6b9b1

This is great, thanks for doing that! I think for a production script it would be better to plug the YAML data into HTML template files, so it's easier to change things (a little more like what we have now), but this is a good proof-of-concept. I was also considering writing a script to convert the inner text of the YAML from Markdown to HTML, so we could support full Markdown even within YAML blocks. I may write a script to to do this today if I have time. If we did that, then we would have blocks of HTML that could just be plugged into the templates, which I think is a suckless-ish way to support more styling.

Also wondering, for future versions of this, would it be better to have separate scripts for verification and conversion to HTML? It might make for cleaner code in the conversion script, since it could assume the YAML has already been validated by another script. I am envisioning perhaps a system like this for website building:

            +---------------------------+
curent      |semi-automated conversion  |
md      --> |(scripts with some human   +-->  yaml recipes
recipes     |intervention to fix things)|     (with markdown
            +---------------------------+     in each section)
                                                    |
                                                    |
                     +--------------------------+   |
                     | verification script      |   |
    confirmed,       | (checks for correct      |<--+
    corect YAML  <---+ sections and correct     |
    recipes          | types (list, string, etc)|
       |             |                          |
       |             +--------------------------+
       |
       |
       |   +-------------------------+
       |   | markdown-to-html        |     recipes with
       |   | conversion script       +-->  html inside
       +-->| (replaces markdown      |     yaml strings
           | within the yaml strings |           |
           | with html)              |           |
           +-------------------------+           |
                                                 |
                   +----------------------+      |
                   | publishing script    |      |
                   | (plugs bits of html  |      |
      WEBSITE  <---+ from yaml into html  |<-----+
                   | templates, generating|
                   | the actual site)     |
                   +----------------------+

Of course, the first row of that diagram is only there until we have transitioned over (assuming we do transition over), and the third row is optional, it just depends whether we want to support markdown for things like bold and italics.

I'd love to hear other thoughts on how we might do the conversion, and whether people think supporting Markdown is a good idea.

@TTWNO

If we are going to use a new format, then we should also set the measurements as separate things IMO. Having them be computer-readable would help long term.

I agree, but my concern is that this keeps people from doing things like "a pinch of salt," "a handful of beans," etc. We could just make it optional to put measurements like that, but then people might just never do it. It also seemed like Luke wasn't interested in the other threads, so we'll have to see what he says about both this and the YAML idea in general.

IMPORTANT to consider before doing anything else

Before we start converting recipes to YAML, we need to A) See if Luke is even willing to go forward with any of this YAML business, and get his input, and B) discuss and decide on the final details of what a proper YAML recipe should be. It will be a pain to change them afterwards, so we should do our best to think of everything and establish a standard before we start converting current recipes.

[unicows]

There is a recipe scheme already created. A python recipe scraper can pull recipes from websites that support the microdata. Nextcloud has a recipes plugin that does this as well and stores the recipes in a file folder as json recipe schema files. A standard exists that would be worth looking into. I was looking into writing a markdown -> recipe schema converter and adding the microdata to the HTML or generate json-ld files next to the html since I cannot import recipes on this site via HTML microdata.

[dther]

Forgive my bikeshedding, but why not TOML? TOML parsers are more lightweight and MinimalistTM, making them more suited for simple recipe->machine data->recipe transformation, and scaling better with large file sets.

[AJ-Williams1]

Forgive my bikeshedding, but why not TOML? TOML parsers are more lightweight and MinimalistTM, making them more suited for simple recipe->machine data->recipe transformation, and scaling better with large file sets.

TOML is nice, I personally think YAML is better for 2 reasons:

1. It just looks prettier (to me, I know this is completely subjective).

2. It supports lists in a cleaner way, which I think will make ingredients and directions much nicer. For example: In YAML:

   directions:
   - Do something
   - Do something else
   - >
   These can
   easily
   be multi-line with 
   newlines ignored
   - |
   Or multi-line with
   newlines preserved
   like so
   
   This is a second paragraph of the same direction.
   This allows people to use full markdown format within directions/ingredients, as I mentioned might be nice.

   Whereas in TOML we would have something uglier like this:

   [recipe]
   directions = [
   "first direction",
   "second direction",
   """
   multi-line
   direction
   """,
   ]

I just feel the YAML way is much cleaner and easier for humans to read/write in this application. That may just be me though.

[DGKSK8LIFE]

Agreed.

[dther]

I see your point, flexibility and human readability matters much more in this usecase. Best of luck!

[AJ-Williams1]

There is a recipe scheme already created.

This could be good, I only worry that Luke may think it over-complicated. We'll have to see what he says.

A python recipe scraper can pull recipes from websites that support the microdata.

I'm not sure whether Luke wants this to be purely community-contributed rather than pulling recipes from other websites, but even if we did want to pull recipes this way, we could still do it without using the schema ourselves if we didn't want to.

Nextcloud has a recipes plugin that does this as well and stores the recipes in a file folder as json recipe schema files. A standard exists that would be worth looking into.

I don't love JSON for this, even though it is easier to parse and easier to integrate with anything web-related, just because I think easy human-read/write-ability is more important, but I'm open to other opinions. And, as usual, we'll have to see what Luke says. I think that he wants things to be git-based (no pun intended) rather than using Nextcloud, but who knows.

I was looking into writing a markdown -> recipe schema converter and adding the microdata to the HTML or generate json-ld files next to the html since I cannot import recipes on this site via HTML microdata.

I have a feeling Luke may not like this for its complexity, since I think he is going for a simple, static, pure HTML recipe site with nothing fancy, but I do see the benefit of having this; it could act as a sort of API for anyone using the python scripts you mentioned. However, I think trying to parse markdown by hand and convert it to a schema will be very difficult. I chose YAML because it has the friendliness of markdown but is much easier to deal with and convert.

Again, we'll have to wait to get some input from @LukeSmithxyz on all of this in order to choose which direction to go, if he's interested at all.

[unicows]

There is a recipe scheme already created.

This could be good, I only worry that Luke may think it over-complicated. We'll have to see what he says.

The intention would be to add this on the back end during compile, leaving the markdown alone, but that would mean parsing markdown...

I think that he wants things to be git-based (no pun intended) rather than using Nextcloud, but who knows.

I mentioned Nextcloud as an application of reading another sites' microdata. One can pull a favorite recipe from based.cooking or any other compatible website into their personal Nextcloud instance so they do not have to go searching through bookmarks.

I have a feeling Luke may not like this for its complexity, since I think he is going for a simple, static, pure HTML recipe site with nothing fancy, but I do see the benefit of having this; it could act as a sort of API for anyone using the python scripts you mentioned.

I agree. This machine compatibility may be better suited as a separate project that parses the predictable markdown.

It seems #170 and #158 already cover the concept.

[dm17]

Nice! I think UML or flowchats are also be a great way to display cooking data (especially to prevent getting lost). I see someone is working on a paid version of this: http://samimatilainen.com/flowchartrecipes/ https://www.facebook.com/flowchartrecipes

[simonhughxyz]

If you are going for a data first YAML approach, then it might also be a good idea to also separate the YAML recipes into its own repo. That way people can write their own applications that take advantage of the recipe repo.

[DGKSK8LIFE]

If you are going for a data first YAML approach, then it might also be a good idea to also separate the YAML recipes into its own repo. That way people can write their own applications that take advantage of the recipe repo.

Ideally, you'd write a script to convert al existing markdown recipes into YAML.