Markdown node type

[scripting]

Some inquisitive Drummerites found that I was playing around with a new nodetype called markdown.

The idea is that if a node has that type, when rendering it in Old School, we convert the text from Markdown to HTML.

If you want an example of a markdown node, use the Open URL command in the File menu to open this outline.

http://scripting.com/publicfolder/misc/markdownExample.opml

This is how it renders through Old School.

It works okay -- but I wasn't ready to document it.

What I need most is an intelligent user of Markdown or two, who also uses Drummer, to have a discussion with about the tradeoffs.

You know I think that's a pretty good offer, btw. People could really pitch in and help on these projects. Just sayin.. I think it's a lot more interesting to be part of the design process than reporting bugs after things are locked down.

PS: This is the relevant code in Old School. As you can see it doesn't do very much. It doesn't generate any indentation according to the outline structure, and puts two newline characters at the end of every node.

[scotthansonde]

I like the idea of being able to use Markdown in an outline. Using Drummer to use (and see) basic formatting and links, and Markdown for anything more complicated, that would be a nicer experience than for example on GitHub issues where you have to keep switching between "Write" and "Preview" to see even the most basic formatting.

I also realize some writers would rather not see Markdown codes in their words as they work. So it wouldn't be for everyone, but it's a nice option to have!

[am1t]

I see how this can be useful, especially the blockquote, inline code and lists. However, I haven't yet warmed up to the idea of a markdown type node in outlines. Even in WordPress block-based editor for that matter, marking a block as Markdown first before typing Markdown kills the flow for me.

I do wonder though, given that we can enter HTML in outlines, why not always parse for Markdown and convert to HTML? It would be one less level of friction.

Apologies if I am missing something obvious or what I speak is not inline with what you have in mind.

[scripting]

@am1t -- thanks for the comment, and glad to see you back in the loop. ;-)

First, I just posted a comment about one of your posts here.

You don't have to decide in advance to use Markdown. Just start using it, and then when you're ready go back and set the type. It doesn't have to be for the whole post it could just be a single node.

The reason it doesn't and can't work like HTML is that the processing of HTML happens at the end, in the browser. All we have to do is let it pass through unmodified.

But Markdown isn't interpreted by the browser. So we have to do it. How do I know that the # you entered into a post is meant to be interpreted by markdown or let pass through? I need to be told explicitly that's what you want otherwise the users will be very unhappy. ;-)

Please keep pushing like this. I think I'm unusual as a developer, I need user minds to engage with on this stuff. Even if I can't do exactly what you want, I will learn what you want and might be able to provide it another way.

[kenbooth]

Could an entire outline be labeled as Markdown by editing the OPML Header? The thought of changing each node seems a little fiddly to me. I would personally be either all in with Markdown or all in with HTML.

[scripting]

@kenbooth -- it's a little early for that kind of thing. i might still be the only person who's used the feature. let's see how it goes.

[am1t]

and glad to see you back in the loop

@scripting It's good to be back. It's a festive month here in India, so have been travelling places :)

Now that I read your thoughts, I see how marking a node as Markdown is a good option to start with. I had something similar to what @kenbooth suggested in mind -- all Markdown or none. But that can come later.

Supporting Markdown selectively on nodes, which is user defined, looks like the simplest option. I am just worried that the outlines will become messier than what they are now (text-only).

[vincode-io]

I'm using Markdown embedded in OPML for Zavala. I use it very sparingly at this point. I only support a couple codes: bold, italic, link, and image. I find it much more readable than embedding HTML in the OPML file.

I also support exporting outlines as Markdown documents.

This:

<?xml version="1.0" encoding="UTF-8"?>
<!-- OPML generated by Zavala -->
<opml version="2.0">
<head>
  <title>Zavala OPML+Markdown Example</title>
  <dateCreated>Thu, 18 Nov 2021 16:17:56 GMT</dateCreated>
  <dateModified>Thu, 18 Nov 2021 16:47:47 GMT</dateModified>
  <ownerName>Maurice C. Parker</ownerName>
  <ownerEmail>mo@vincode.io</ownerEmail>
  <ownerID>http://vincode.io</ownerID>
  <expansionState>0,2</expansionState>
</head>
<body>
  <outline text="This would be considered a **Level 2** Heading" _note="This is text associated with the Level 2 Heading">
    <outline text="This would be considered a _Level 3_ Heading" _note="This is text associated with the Level 3 Heading"/>
  </outline>
  <outline text="This is an unordered list with a [link](http://scripting.com)">
    <outline text="This is item 1"/>
    <outline text="This is item 2"/>
    <outline text="This is item 3"/>
  </outline>
</body>
</opml>

Becomes this:

# Zavala OPML+Markdown Example

## This would be considered a **Level 2** Heading

This is text associated with the Level 2 Heading

### This would be considered a _Level 3_ Heading

This is text associated with the Level 3 Heading

* This is an unordered list with a [link](http://scripting.com)
    * This is item 1
    * This is item 2
    * This is item 3

We are very similar in what we are doing, except I don't support using Markdown to define document structure. I use the OPML format for that.

For now, I'm checking to see if a _note property is available to determine the difference between a Heading and a List. I'm not real happy with that approach and am looking for a better way to do it.

I also don't have support for a whole bunch of things in Markdown, like Ordered Lists. I will probably expand my support more as Zavala matures. I know many times I would have liked to use Code Blocks and Block Quotes, but don't support them.

The one thing on my wish list would be that you would split document structure from text formatting. I would prefer that document structure is done in OPML and text formatting in Markdown.

As an example, an ordered list might be defined as:

<outline type="ordered_list" text="Ordered list">
  <outline text="aukland"/>
  <outline text="persephone"/> 
  <outline text="raiders"/>
</outline>

Notice that the Markdown number isn't necessary. That would make it possible to generate Markdown, HTML, or some other format with a different numbering scheme like the Harvard Outline Format. (Apologies if I got how the type property works wrong. I don't fully understand them yet.)

I'm really interested in what you come up with here. I want to make Zavala as interoperable as possible and am willing to put in some work to make that happen.

[mistersugar]

I would/will use Markdown to style outlines and blog posts. I would appreciate that. I've long used Textile for my blogging in the Textpattern CMS, but I've also learned to use Markdown in the iA Writer tool (which I used for simple outlines for a long time before I learned to use Little Outliner and now Drummer.

[tohuvabohu]

@scripting I am reading your posts about desiring feedback on features and bugs, and I do care, ha! I've been writing in Markdown for over a decade, even when I'm scratching some handwriting on the back of a napkin. I also recently started using Drummer, and I've just started trying to learn some JavaScript in the last few weeks as well. But, I just don't know enough about scripting (which I get the irony of that being your handle) to think that I have helpful feedback.

[sabre23t]

Thanks for testing/adding Markdown nodetype @scripting . I'm confident I would love it. I've been using Markdown markup in outlines since 2012 on Checkvist.com and since 2021 on Logseq.com. A few immediate comments ...

1. It would be a good choice to go with GFM flavour of Markdown.
2. Markdown allows raw html so a default of Markdown nodetype allows usage of both Markdown and HTML.
3. I find most useful in outlines are Markdown inlines tags. Bold/Italic supported using ctrl-B/ctrl-I in Drummer. Using Markdown code (`), strikethrough (\~) is as easy.
4. Supporting Markdown headings and lists are trickier because how they interacts/overlaps with parent outline hierarchy/indentations. I like the way Logseq handles this.
5. A bit more testing shows that Drummer limits of only one line of text in a single node/headline still hold. That constraint easy usage of Markdown blocks that are usually multilines. Both Checkvist.com and Logseq.com allow multiple lines text in their outline nodes/blocks.

Mmm ... seems Drummer Markdown nodetype have different convention for multiline blocks. Need to explore this more later. GTG

[PostMonsterG]

I wrote up my experience working through a possible use case involving iA Writer on the iPad here http://oldschool.scripting.com/PostMonsterG/2021/11/17/024054.html?title=discardingAnIdea

I really like the idea of Markdown support. I don't have any good suggestions for resolving the issues I ran into that wouldn't involve a major revamp of the outliner part (concord?) of Drummer.

For the idea I was exploring, I think there's just a basic impedance mismatch between "this node is a document I can edit with iA Writer" and the way Drummer views it.

I conceptualize Drummer's approach as, "each return is a node", which is natural for an outliner, but different from traditional text editors like iA Writer.

[sabre23t]

Okay a bit more testing/commenting. As @PostMonsterG said, in Drummer "each return is a node", in other words in Drummer a node can only contain a single line (multilines not allowed).

So how can we represent a multiline paragraph of text in Drummer as per GFM Example 190? How about making the mulitple lines as subs to a node/headline per screenshot below?

Unfortunately that failed. Drummer hands it over to Old School that merges all 5 lines into one paragraph, instead of two. :-(

[am1t]

@scripting I wrote a long post with Markdown to try out the option, and I can report that I like this change overall. I have, of course, captured a few thoughts and instances when I will use this.

@sabre23t I noticed, while writing the post, that the multiple paragraphs was failing. I am sure that will need a fix. For now, I used a <br/> to break into multiple paragraphs.

[scripting]

This is great stuff, what I was hoping for -- that you guys would push the edges, find the problems and even suggest workarounds. I'll be thinking about this and will report back. Thank you. ;-)

[fmcpherson]

Because the plus button defaults to type = outline, I am wondering if you might want to have a user setting for the default type the plus button uses? I can see some people who would always want new nodes to be markdown while others would be happy with outline. At a minimum I hope to see the markdown type stay, I expect to use it most for quotes.

[scripting]

@fmcpherson -- that would come much later, when that problem is presenting itself, if it does.

Right now I just want to sort out the mechanics of how the Markdown text is converted to HTML.

[scripting]

Okay -- I've given it some thought, considered all your comments (I hope) and I think I know what I want to do next, but I want to run it by you all first.

How many newlines?

At the end I've included the code in Old School that renders Markdown for titled posts. It gets a pointer to the parent, and it visits each of its subs. The key bit of code is in the function add. Note that it ignores outline indentation, and it generates two newlines for each line from the outline.

I think the two newlines is the source of a lot of the difficulty, and it should change to one newline per headline.

What role should indentation play?

I thought about what significance outline indentation should have on the Markdown. In the current implementation, again by looking at function add, you can see that indentation is ignored. As tempting as it is to use it for something, I think this is the right approach. You, the author, can use indentation however you like, but it isn't part of the Markdown rendering. I think this is consistent with the philosophy of Markdown.

To be clear, again, this only applies to titled posts, not singular posts.

    function subsToMarkdown (parent) { //10/30/21 by DW
        let markdowntext = "", indentlevel = 0;
        function add (s) {
            markdowntext += s + "\n\n";
            }
        function addlevel (theNode) {
            if (theNode.subs !== undefined) {
                theNode.subs.forEach (function (sub) {
                    add (sub.text);
                    indentlevel++;
                    addlevel (sub);
                    indentlevel--;
                    });
                }
            }
        addlevel (parent);
        var processedtext = "<span class=\"spMarkdownText\">" + markdownProcess (markdowntext) + "</span>";
        console.log ("getItemSubs: markdowntext == " + debugMarkdownText (markdowntext));
        console.log ("getItemSubs: processedtext == " + debugMarkdownText (processedtext));
        return (processedtext);
        }

[scripting]

@am1t -- I checked in the style sheet changes.

[scripting]

I made the change in Old School, it's no longer adding two newlines for every headline coming out of Drummer.

But -- when I skip a line, it's not starting a new paragraph.

I've been reading everything I can find trying to figure out what's the legit way to skip a line in Markdown text, and I can't get it to skip a line.

Here's the post I'm working with.

And this is the Markdown engine I'm using.

[scotthansonde]

John Gruber, the godfather of Markdown, said to force a line break, end a line with two spaces. https://daringfireball.net/projects/markdown/syntax#p

I find that totally unintuitive (not to mention not easily visible).

[scripting]

I finally have it working somewhat the way I want it to. An example.

Here's the text in a post, as I edited it.

And here's the rendering.

I didn't do any fancy stuff like put spaces in weird places. And I got what I was expecting.

Now we have a baseline to test against, I think.

And clearly there's more work to do on the CSS. ;-)

[scripting]

Comments on markdown posts?

I made a big change to the way markdown posts are rendered (see above).

If you have an experimental post, now would be a great time to try re-rendering it and..

1. Evaluate -- how does it look?

2. Does it make more sense this way, from a markdown point of view?

3. Should we leave it this way -- ie indentation is ignored, you can use it any way you like and 2. we add one newline for every headline, not two.

4. Please provide a link to the rendered page. I'd like to see how you're using it.

[akaKenSmith]

I have only one simple markdown experiment, a blockquoted poem that gains its linebreaks using the html br tag. It does not seem to have changed in appearance when I gave it a fresh render today.

http://oldschool.scripting.com/KenSmith/2021/11/19/014000.html?title=aPrivateClub

[scripting]

@akaKenSmith -- I looked at your OPML file, and noted that your use of Markdown was on an individual node not a titled post. I'm trying to work out the issues for titled posts right now.

But it was really interesting to watch you write your latest post in real time. It's the first time I've observed an instant outline in action, believe it or not. Wow.

[am1t]

@scripting I tried it with a new post, looks good to me. However, as I feared, I don't like how the outline looks, too many empty spaces. I guess that's the trade-off.

But still, why is a new node not considered as a new paragraph? Is there a technical reason?

Of course, as you said, styling is yet to be fixed. I wonder why these markdown elements need special styling? Can these not be rendered as other regular nodes?

Update: I see that the ordered list is not rendered correctly any more. It's broken in your original example post too.

[sabre23t]

My markdown experimental post today here. It renders as below, markdown left, outline right:

My immediate comments/suggestions on the renderings:

1. The post title heading in markdown nodetype add extra newline, it doesn't in outline nodetype that looks better to me.
2. I needed to add extra blank node/headline to break the post into two lists. I suggest that change of indentation automatically generate newline for markdown processing.
3. Perhaps a good design goal is to have an outline nodetype when changed to markdown nodetype, renders almost exactly the same. Noting that markdown markup allows HTML markup in it.

The markdown nodetype left, and the outline nodetype right, as in drummer:

[sabre23t]

am1t wrote:

But still, why is a new node not considered as a new paragraph? Is there a technical reason?

I think main reason is Markdown GFM Paragraphs definition that says a sequence of non-blank lines (generally) forms a paragraph. So we need blank lines to separate paragraphs in Markdown. If we don't want those blank lines a workaround is to use <p> at start of the paragraphs.

The following is a screenshot of that example of markdown nodetype in Drummer, rendered here in Old School.

[am1t]

Thank you, @sabre23t, for the clarification. I understand the need to stick with Markdown behaviour. But given that outline to Markdown text conversion is done in OldSchool (add function in code that Dave shared), in what condition should a new node in an outline not be considered as a new paragraph?

[sabre23t]

am!t wrote:

what condition should a new node in an outline not be considered as a new paragraph?

One example would be list items as per my earlier example further above.

More examples of impact of blank lines separation (and otherwise) to list items given in Markdown GFM Lists defintion. Specifically difference between tight list and loose list.

If a node/headline generate two new lines (previous behaviour of Old School) we won't be able to generate Markdown markup for tight list.

There may be other examples of Markdown constructs that can't be generated when a node/headline generate two new lines.

[scripting]

@sabre23t --

• Interesting point how this is LogSeq in reverse. Only in this context, for publishing via Old School. In other contexts it'll be exactly what LogSeq does, or as close as we can get. There the purpose will be interop. This is the usual Markdown use-case, going from written document to published document.

• Re an outdent generating an extra blank line, I saw the need for it too, but imho the cost is too high. I want to be able to say simply that what we send to the Markdown processor is exactly what you typed, and that indentation is insignificant, it's there for you to use, has no bearing on the output. I don't want to have to add "except when you outdent." So get used to typing a blank line at the end of a list, if you're indenting (which I certainly will do myself).

• This same code will be used to flow to GitHub, btw. I have a project underway that is built around that idea.

• Re the "design goal" -- I don't want to think about that at this time, way too complicated to consider. Also it would introduce another dependency when adding a rendering feature to Old School, the impact it might have on Markdown. But thanks for planting the seed.

@am1t -- Re a place where generating two newlines for a node is a problem. There must've been a problem, or I wouldn't have reverted to one newline. I wasn't taking great notes at the time, I was confused as hell, and needed other people more experienced in Markdown to help.

Summary -- I think the approach we have now is what Old School should do. One newline per outline node, indentation belongs to the author, generates nothing.

If all goes well I should have the next version ready in a couple of hours.

[scripting]

@sabre23t -- thanks for your guidance. it's good to have you around with your deep understanding of markdown.

I've just finished this morning's changes.

[am1t]

Thank you, @sabre23t -- I had no idea about tight and loose list. I was always curious why the way things were set up the way they were. I learnt something new today 👍🏽

[scripting]

I started working on some docs just now and realized I can't live with the way we've got Markdown set up.

To see what I saw, choose Open URL in the File menu in Drummer, and enter this URL.

http://scripting.com/publicfolder/misc/pageParkDocs.opml

Those are all the docs for PagePark, edited over quite a few years.

I think you can tell that the generator for this text is generating two newlines per outline line.

It would be incredibly more difficult to work on if it was just one line per.

[scotthansonde]

On the GitHub Docs I was writing, the only place the new lines made a difference was within a code block. Otherwise there were just minor aesthetic differences (blank lines or not between list items). So I could live with a per-user choice as described in the Change Notes.

[sabre23t]

Dave blogged yesterday:

[...] I can't work with the one-newline per headline rule. [...] So my current thinking is that it's going to have to be a per-user choice, and it will have to result in a new head-level attribute to communicate the choice between Drummer and Old School, via the OPML.

Instead of head-level an alternative could be communicating this at node-level. Possibly having node type markdown generating two-newlines per headline, and node type markdown1 generating one-newline per headline.

This could allow a blog.opml to have mostly markdown type nodes and only markdown1 where necessary (to allow markdown markups like tight list, continuous quotes etc).

[am1t]

I would love the per-user choice. As I mentioned earlier, this change will make me use Markdown more. I didn't like the way the outline looked with just one new line per node.

[scripting]

Markdown-in-an-outline

I think I have the Markdown-in-an-outline question sorted out, so this is an RFC.

Old School generates two newlines for every outline node, except...

• when a node has an flSinglespaceMarkdown attribute, value true, one newline is generated for its subs.

the assumption is that most often people will write in the outliner one paragraph per headline. but there are exceptional cases where you need more control over the markdown text we generate, and need to do your own double spacing, so you tell Old School to just do one.

PS: I've implemented it now. One addition, it generates an extra newline after finished with subs that just had one newline. Saves you from having to add one at the end of a single-spaced level.