Enable better formatting of the text of requests

[RichardTaylor]

A WhatDoTheyKnow user has requested the ability to add formatting, such as a smartly indented numbered list, to their request.

Presumably providing this feature would involve allowing users to write a rich text or HTML email and for the Alaveteli system to send it to public bodies in that format, and display it on the site.

There is I think I value in sites looking smart and tidy, and people want their requests to be readable by the public body and those reading them on the website.

Other tickets on improving the formatting / presentation of content:

3050 Better formatting of prominence reasons

3420 Make HTML format emails for Alaveteli (I don't think this ticket deals with request correspondence)

1528 Improve/fix HTML rendering of tables

[garethrees]

Not massively keen on dealing with HTML emails in correspondence, but we could potentially parse basic markdown (e.g. lists, bold, italics, quotes) for display?

[RichardTaylor]

@garethrees Are you envisaging mark-up which could be sent to the public body; and perhaps quoted back by them, that would be both human readable and result in a pretty display on Alaveteli sites?

I can just about envisage something like that working for lists starting with say a *; but I would have thought any more involved mark-up would interfere with readability for the public body?

Encouraging simple concise requests is probably the best way to keep the site looking smart and readable, there's an issue aimed at tackling excessively long requests at #3387

[garethrees]

I can just about envisage something like that working for lists starting with say a *; but I would have thought any more involved mark-up would interfere with readability for the public body?

Yeah, I'd say we'd only want to parse lists, bold (**a bold phrase** or __a bold phrase__) and italics (*an italics phrase* or _an italics phrase_).

I wouldn't be keen on doing anything more than that for the reasons you mention.

[jpwarren]

How about a site-operator selectable subset of MarkDown able to be turned on/off for different areas? Default could be for a limited set of lists, bold, italics, and operators can choose to add more MD features if they see a need?

Annotations is one area I'd really like to see MarkDown support, so emphasis can be added, better formatting of cited legislation, and blockquotes.

[RichardTaylor]

A WhatDoTheyKnow.com user has requested a feature enabling tables to be included in outgoing correspondence, they wanted to send a message "formatted as a Table with my comments against each of the Case Officers "Cost Exceeds Limits" justification[s]".

[mdeuk]

Expanding a little from my +1 in the other ticket:

Being able to present tables, use bullet points, and carry out basic text formatting would be incredibly helpful - even if it were only enabled in certain aspects, e.g. annotations, or on a per-user basis.

It would also be useful for #3075 - updating help pages as it stands is a bit of a 'dark art' and is somewhat reliant on the admin having a good grasp of HTML concepts - fine, perhaps when a developer is making the changes, but oftentimes these changes (at least on WDTK) are carried out by non-developers, so keeping it simple can be an enabler. 😃

[garethrees]

Rails 6 (#5159) includes built-in rich text editing, so this might be something we can explore. I like the idea of trying it out on some less prominent areas of the site first (comments, profiles, etc) to get a grasp on any issues, and then considering it for correspondence.

[RichardTaylor]

A WhatDoTheyKnow user has reported seeking to use whitespace to indent items in a list they included in correspondence. They found that the system removed their attempted formatting, making the correspondence less clear than expected.

The user has proposed adding a mechanism for including an indented list in outgoing correspondence.

This report, and proposal, fits with the idea noted above of enabling basic formatting eg. lists.

[RichardTaylor]

Further useful comments, suggestions and insight from the user previously referred to:

It wasn't a complete surprise; the preview showed the message stripped of leading whitespace. However, the whitespace was displayed correctly in the editor, including when I returned to it after viewing the preview, so I wasn't sure whether it was being stored correctly but displayed without whitespace, or being stripped of whitespace internally.

You are correct to assume that I initially pasted in my reply from elsewhere (a simple plaintext text editor; no formatting etc., indentation introduced manually with leading spaces). However, after viewing the preview and seeing that the leading whitespace had been removed, I also tried restoring the whitespace by hand in the editor on the website. That didn't seem to work either.

Regarding the GitHub issue, I think my suggestion is slightly simpler than any proposed there. Implementation of any level of formatting syntax presumably opens a whole can of worms about formatting for web display vs the formatting of the emails delivered to the authorities. All I would propose is ensuring that the text submitted exactly matches the text in the form's edit box upon submission, with no alterations and including any whitespace. It can then be left up to users how to format their text within the traditional constraints of plaintext emails, but with the freedom to manipulate whitespace to enforce indentation where appropriate.

[WilliamWDTK]

We've just had an email from a user who wants to either upload a PDF, or add formatting to the reply, as they have written a long and comprehensive request for an internal review, including quotes from both the authority concerned, and the ICO. The user felt, and I would be inclined to agree, that additional formatting was necessary to produce a sufficiently readable message.

The user mentioned that:

Standard mark down doesn't seem to be accepted.

They have sent us the PDF that they would be attaching, and I thought it worth considering whether we could upload it to files.wdtk, but that's an aside from this issue of formatting. I have written in my draft reply to allude to why we might not want to be hosting user-uploaded attachments, as well as referencing this issue here.

[garethrees]

Rails does at least have Action Text built in now, which could be used to add minimal rich text formatting. I think it is more worth considering these days. The original resistance against formatting was having to parse HTML replies, but we do that 99% [1] of the time anyway.

[1] Conjecture! I have not actually analysed this yet.

[WilliamWDTK]

Would the formatting be lost? Presumably we'd be sending markdown/some rich text format that has been parsed into HTML to the authorities, and then converting it back, losing the formatting the the process. Is that what is being vaguely considered?

[garethrees]

I guess we'd do that initially yeah, but this is unlikely to be worked on any time soon. The above comment was more just "a new thought/capability that isn't reflected on the issue" rather than any implication that further exploration will happen.

[RichardTaylor]

Note one-off experiment involving manually uploading a PDF at:

https://www.whatdotheyknow.com/request/public_rights_of_way_gis_data_14#outgoing-1317023

[garethrees]

We've had a pro user get in touch saying that this might be their top feature request. The key formatting features in the example they provided privately are:

• Bold
• Lists (unordered and ordered)

Rails does at least have Action Text built in now, which could be used to add minimal rich text formatting. I think it is more worth considering these days. The original resistance against formatting was having to parse HTML replies, but we do that 99% [1] of the time anyway.

Just thinking on this, there's no reason why we couldn't do this independently – create HTML outgoing mail (which gets formatted nicely in the web UI and in the mail that we send to authorities), and then reparse what we get back as plain text (without https://github.com/mysociety/alaveteli/issues/4003).

If we implemented this we'd open ourselves to new vectors of bugs and abuse of course – borked html tags, people using all bold, etc.