Remove accommodation for hard-wrapping (design goal 7)

[eugenesvk]

Blank lines, especially in lists, is one of the more annoying features of this otherwise great improvement over markdown(s).

This is some text.
## My next heading

- one
  - two
  - three

↓

This is some text.

## My next heading

- one

  - two
  - three

As far as I understood it, it's related to this design goal:

7. The syntax should be friendly to hard-wrapping

But hard-wrapping is an annoying historical misfeature (at least for regular natural language text), since it's trying to force something that should be flexible and reader-dependent — line width — to be fixed and writer-dependent. It made (a tiny bit) more sense in the past when screen width was more similar and soft-wrapping was(?) expensive (or something), but with the current huge variations of viewing devices (especially mobile devices) it's lost even that.

Therefore, while source readability is important, I'd argue that the specific requirement without special editor modes that soft-wrap long lines is ossifying this poor historic practice. The editors should adjust its modes to make soft-wrapping great and convenient instead And if my understanding is correct that is also introduces the extra blank lines, it adds a strong reason to remove such a design goal

[jgm]

hard-wrapping is an annoying historical misfeature

I disagree. I use editors that can soft-wrap, but I still hard-wrap. There are many reasons for this. One is to ensure that the documents will look right on a wide range of editors/readers (not just one with my own custom settings). Another is because even on my editors, the soft-wrapped versions look bad. Here is a comparison with Emacs with markdown-mode and visual-line-mode:

[jgm]

Anyway, I know from experience that I will never convince anyone in the anti-hard-wrapping crowd. Djot is designed to be usable whether or not you hard-wrap; it gives you the option, and that's important to me.

[bpj]

hard-wrapping is an annoying historical misfeature

1. No it isn't. Some of us prefer physical text lines which don't exceed a certain length because it makes editing/reading/diffing[^1] easier.
2. There are many who type each sentence/clause on its own line as a variant of # 1 and to make it easier to eyeball the text, who most emphatically want their line breaks preserved.
3. In terminal Vim scrolling gets wonky if lines are too long.
4. Some of us have peripheral vision loss and need a limited line length for that reason.

All except # 2 apply to me.[^2] Don't try to force softwrapping on me or anybody. We aren't forcing hardwrapping on you!

[^1]: Yes diffing text is a thing. There are many reasons to use plaintext-based workflows.

[^2]: In fact I use both softwrapping at 36 chars to accommodate # 4 and hardwrapping at 72 chars to accommodate # 1 and # 3, and I'm pretty dependent on both!

[jgm]

Also worth mentioning: Some people have a practice of using one sentence per line. That is also supported and is good for diffability.

[eugenesvk]

Anyway, I know from experience that I will never convince anyone in the anti-hard-wrapping crowd

Definitely not with the example that ignores the core argument! Why would you use the same width when the whole point is that your readers are unlikely to have it? So your goal

documents will look right on a wide range of editors/readers

actually means the opposite — you should NOT force your width on a widely different readers (from narrow mobile phones to ultra-wide desktop monitors)!

So instead of visual improvement over the soft-wrapped version you'd have two unfixable fails

And herein lies the core difference — the soft-wrap issue can be fixed with a better implementation, while your hard-wraps can't be fixed, you force the width on all the readers, so it's strictly a worse alternative!

Here is a comparison with Emacs with markdown-mode and visual-line-mode

Don't know about Emacs, but here is a simple fix I've found with a quick Google search for Vim:

Given how extensible these editors are, I'm certain that the fix can be extended to block quotes within markdown

Djot is designed to be usable whether or not you hard-wrap; it gives you the option, and that's important to me.

Well, the option is bad since it forces useless blank lines (especially noticable in lists)

[eugenesvk]

We aren't forcing hardwrapping on you!

Of course you are. The hard wraps are literally hardcoded into the text! And also into this spec, which forces blank lines!

2. want their line breaks preserved.

But the line breaks are NOT preserved, without a blank line djot concatenates all the hard-broken lines into a single paragraph!

3. In terminal Vim scrolling gets wonky if lines are too long.

So why should this bug force a worse format???

4. Some of us have peripheral vision loss and need a limited line length for that reason.

Soft wrapping solves this as you mentioned yourself, so why bring it up?

[jgm]

With the vim option you mention:

Still not great! The ordered list looks okay. But only the numbered list, and only if it has a single digit followed by a period. If you use lettered lists, it breaks; if your list has two-digit numbers, it breaks; and this doesn't help with the block quote.

[jgm]

Oh, one should also :set linebreak so that breaks don't occur inside words. The result is then a bit better, but the issues noted above persist.

[jgm]

The issue with lists with two digit markers can be fixed with

:set briopt+=list:-1

but that is only supported starting in vim 8.2.3198 (a year ago). And it does not fix the other issues.

[jgm]

Worse, with these settings, extra indentation is added even in a code block!

There is only one space between an and embedded in this line, which should be verbatim; the reader will assume that there is a newline and indenting spaces.

[jgm]

When all common tools in which text is edited (including e.g. text boxes on the web) automatically, transparently, and reliably display format documents in a way that is optimal for readability, and when all widely used tools do word-level diffs, it may be time to reconsider this issue. Until then, design goal 7 follows from the goal of source readability.

[jgm]

And yes -- I do understand that documents with hard breaks can be hard to work with in editors/textareas with a line width that is smaller than the width to which you have hard-wrapped the text. That is a consideration one must take into account in deciding whether to hard-wrap in emails, for example.

But if I'm writing an article, I am mainly concerned with how the source looks in my own editor.

With djot you have the freedom to go either way. Freedom has a price: in this case it's the blank line before lists.

[eugenesvk]

If you use lettered lists, it breaks; and this doesn't help with the block quote

Another quick google led to this additional option https://neovim.io/doc/user/options.html#'breakindentopt' that sets the regex for lists, so you can add any list you like. Or maybe it's already added in some addon, don't know, but it's weird that you're actively trying to block the light in the end of this soft-wrapping issue tunnel while ignoring the fact that your hard-wrapping solution doesn't (and more importantly — can't) even offer that while the downside is a bit more severe than a lack of a hanging indent in some cases

There is only one space between an and embedded in this line, which should be verbatim; the reader will assume that there is a newline and indenting spaces

No he won't? since the newline indicator (e.g., special arrows on the right hand side in your screenshots) isn't present and he already expects all lists to have hanging indents that he intentionaly set up? (also, this is another thing that could be solved by adding scope conditions to the rule)

reliably display format documents in a way that is optimal for readability

Strange, this condition is broken for hard-wrapped docs

Until then, design goal 7 follows from the goal of source readability

It doesn't because the readability suffers in the hard-wrapping case, and in a way that can't be fixed via some tool improvement

with a line width that is smaller

or bigger, using more precious vertical space by wasting the available horizontal space is also an issue

That is a consideration one must take into account in deciding whether to hard-wrap in emails, for example.

Except it's a global consideration, you could read any source document on a different device, including the source to an article on your website which you host via GitHub pages

But if I'm writing an article, I am mainly concerned with how the source looks in my own editor.

Your own editor on which device??? Are you really saying that you've designed your spec for a user that only uses one?

With djot you have the freedom to go either way. Freedom has a price: in this case it's the blank line before lists.

'Freedom' sounds nice, but offers no guidance: you could just as well design the spec for the price to be paid by the hard-wrappers by requiring line-continuation symbols, or by the parser with increased complexity (e.g., have special cases for lists or something)

[jgm]

sets the regex for lists

But as noted, this just isn't reliable. The regex will be triggered by all sorts of things that aren't lists, because it won't be aware of special contexts like code blocks. You'd have to implement all the logic of djot in the editor to do this reliably. To this you reply: well, the person using the editor will see the line continuation marker and infer that extra space was added, because the editor was treating this as a list. OK. I don't want to do those inferences to mentally factor out spaces when I'm editing/reading text.

or bigger, using more precious vertical space by wasting the available horizontal space

Using all of the available horizontal space isn't always good for readability. Generally you don't want to go beyond 12 words per line. So yes, if someone opens up my document in a 132-character wide editor window, I'm perfectly happy if it doesn't use all of that line width, and I suspect they'll be happy too. You might say, well, they can just change the width or their editor window. But maybe they're switching between documentation and code, and they want the wider width for the code? You might say, well, there's a special option in my editor to set the wrap width depending on the file type. That could be a solution, but how many people are going to have everything configured that way?

Are you really saying that you've designed your spec for a user that only uses one?

Well, no. The spec is designed so that somebody who wants to hard-wrap can do that. I outlined one scenario in which it would make sense to do so. But if I'm editing an article, I'll prefer it hard-wrapped both in my local editor (where hard-wrapping will make the source maximally readable for me) and in a editing box on GitHub (if I ever bothered to edit it there).

the parser with increased complexity (e.g., have special cases for lists or something)

Well, that's what we did with commonmark. My reasons for not liking the solutions we arrived at there are explained in "Beyond Markdown." (It's not just about parsing complexity, but also about consistency and uniformity.) But it sounds as if you'll prefer commonmark to djot. Fine.

I think we both understand the considerations on both sides, and weigh them differently. You're not going to convince me, and I'm not going to convince you. Let's draw this to a close.

[eugenesvk]

(by the way, emacs, as expected, also has this kind of functionality, for example, in this plugin https://github.com/brentonk/adaptive-wrap-vp)

regex will be triggered by all sorts of things that aren't lists, because it won't be aware of special contexts like code blocks.

Well, the 10. item was a list, it's just that for some reason you believe that it's ok for this list item to be visually worse and overflow in the spirit of being "verbatim". I disagree, I need to be able to read code blocks as well — also for visual consistency precisely to avoid the confusion whether there are extra spaces or not

But maybe there are more real examples where blocked code should indeed wrap differently, so

You'd have to implement all the logic of djot in the editor to do this reliably.

No you won't, you get this information for free from a language syntax scheme — it defines scopes (e.g., the block scope) which you could then use as an additional limiter to the regex. Granted (though I don't know much about this niche of the editor extension functionality), the combination of syntax scope and the indent regex is currently (at least without any plugins) not supported

I don't want to do those inferences to mentally factor out spaces when I'm editing/reading text.

But you're not removing the mental factoring, you're just replacing it — instead of a consistent visual experience, where a 10. long line is always soft-wrapped with a hanging indent, you want to have to mentally factor out spaces outside of blocks while mentally not factoring them out inside

all of the available ... opens up my document in a 132-character wide editor

Ok, instead of this strawman about "all" and "132", let's cover the more realistic annoyances:

• if someone opens up my forced 80-char document in a 100-character wide editor window, I'm perfectly happy to waste the extra 20 characters because I know better

(also an added illustration of the non-email hardships)

• if someone opens up my forced 80-char document in a 60-character wide editor window, I'm perfectly happy to
  • either make half the line disappear (if no soft-wrapping is set)
  • or make all the sentences cut mid-screen due to hard-wraps @80 preventing proper reflows

Not sure why you're happy with such a result, it's much worse than the lack of a couple of spaces in the hanging indent, it's also that can't be fixed by the user automatically unlike with said hanging indents, where a plugin can exist

I suspect they'll be happy too.

Or more likely they'd be annoyed by the visual degradation described above and the disrespect of their settings that as you correctly mention already exist

That could be a solution, but how many people are going to have everything configured that way?

• it could be already configured for them in any good code editor via the installed language plugin
• more than the number of people who agree with your exact specifications on how wide an editor should be

I outlined one scenario in which it would make sense to do so. But if I'm editing an article, I'll prefer it hard-wrapped both in my local editor (where hard-wrapping will make the source maximally readable for me) and in a editing box on GitHub (if I ever bothered to edit it there).

You don't need to bother with GitHub's editing box, your own local editor doesn't have a fixed width because you can use your own local editor on a laptop screen and your same own local editor on a wider desktop monitor! So again, your scenario only makes sense in a rather narrow case of:

• one user (or identical width preferences) +
• one display width +
• never resizing the editor (what if you want to have some side-by-side comparison and need to have the line width a bit more narrow? Well, you'll get the screenshot above with ugly wraps)!

Why punish the broader use cases because of this?

My reasons for not liking the solutions we arrived at there are explained in "Beyond Markdown."

I've read it before, just re-read the 5. Lists and blank lines , and there is no good explanation, just a statement

How to fix lists and blank lines We should require a blank line between paragraph text and a list. Always. That means, even in sublists. So, to create a tight list with a sublist, you’d write:

Also, it seems this isn't true anymore

So we opted for the second option, mitigating the damage with an ugly heuristic (we only allow an ordered list to interrupt a paragraph when the list number is 1).

This is unordered list, it interrupts a paragraph. The numbered list indeed interrupts only when it starts with 1., though it's a much better compromise to support the hard-wrapper than your current djot design

consistency and uniformity

Am I correct that this is only broken for the non 1. ordered lists? And we pay for this with a loss of tightness everywhere.

But it sounds as if you'll prefer commonmark to djot. Fine.

This issue is about a specific djot shortcoming, not an overall comparison, there are also many design decisions that improve over CM

I think we both understand the considerations on both sides, and weigh them differently.

Respectfully disagree, in your first response you've ignored the issue completely, then for some reason you thought that a global issue is context specific ("emails") or that wider can only mean "132-char" max wider. You also weren't aware of a plugin solution to an issue important enough for you to include it as a design principle

You're not going to convince me, and I'm not going to convince you. Let's draw this to a close.

Well, that's on you, I'm remain open to good arguments