Plain content approach

[martintoreilly]

Problem

We do not extract all the article text from this test article from davidwolfe.com. We are missing.

• Text in <h4> elements
• Text in <blockquote> elements that is not in a child <p> element.

Our list of "leaf nodes" from which we extract text content is insufficient. These are currently <p>s and <li>s (and <ul>s and <ol>s that are turned into <p>s earlier).

[martintoreilly]

Problems generating plain content

In the current approach we identify "leaf" nodes and convert their contents to plain text. We currently consider the following HTML elements to denote "leaves":

• p elements
• li elements

Note that Readability.js replaces any div elements that just hold "inline" content with a corresponding p element, so we capture these in our p processing.

In the breaking test document we have a h4 header elements, which we fail to convert to plain text. Looking further at the HTML standard, there are several more elements like this we are currently not catching.

[martintoreilly]

Problems generating plain text

In the current approach we identify "paragraph" nodes in the plain_content output and extract their contents as plain text. We currently consider the following HTML elements to denote "paragraphs":

• Explicit p elements
• div elements (these are turned into ps by Readability.js)
• ul and ol list elements, which we turn into ps

For ul and ol list elements we do some tweaking of the content of child li elements to represent them as a string of the format * item 1, * item 2, * item 3,.

In the breaking test document we have a blockquote element containing both explicit p elements and an implicit paragraph that is just a string within the blockquote element. We currently fail to extract this string. In general, we would fail to extract any text whose immediate parent was not a p, ul or ol.

[martintoreilly]

New approach for plain content generation

The new approach will remove all html tags that are not in a specified whitelist of block-level elements, replacing most of the removed elements with their plain text content, but dropping some blacklisted elements completely (e.g. img, video, audio, script).

To do this we will implement the following algorithm:

1. Start at the top-level "readability" div element we have wrapped the article HTML in.
2. Navigate to the next "leaf" node (terminal child element) under the current element.
3. Process the node: a. If the node is a Text element, keep it. b. If the node is not a Text element and is a blacklisted element, remove it c. If the node is not a Text element, is not a blacklisted element, and has special handling rules, apply these. d. If the node is not a Text element, is not a blacklisted element, and has no special handling rules replace it with the node's innerText. This is a representation of the layout with all styling removed. In our case, we will have ensured all children are text, so we can also just grab the textContent. Both of these are defined in the HTML standard. In practice, we will probably use Beautiful Soup's get_text(" ", strip=True) function, stripping training and leading whitespace from child Text elements before recombining them with a single space between them into a single Text element.
4. Navigate to the node's parent.
5. If the parent node is a blacklisted element, remove the node.
6. If the parent node is a whitelisted element and the previously processed child node is both the only child node and a Text element, wrap this child Text element in a p paragraph element.
7. If the parent has unprocessed children, goto step 2.
8. If the parent has no unprocessed children, goto step 4.

Element lists

Block-level whitelist

article aside blockquote caption colgroup col div dl dt dd figure figcaption footer h1 h2 h3 h4 h5 h6 header li main ol p pre section table tbody thead tfoot tr td th ul

Blacklist for complete removal

These elements will be completely removed, along with all their children. Q: Should we replace these with a " content removed" placeholder? We were discussing doing this for MathML, though I've decided to treat this the same as other embedded content and just remove this for now.

• Forms: button datalist fieldset form input label legend meter optgroup option output progress select textarea
• Images: area img map picture source
• Other media: audio track video
• Other embedded content: embed math object param svg
• Other interactive: details dialog summary
• Scripting: canvas noscript script template
• Data: data link time
• Formatting: style
• Unwanted text content: nav
• Unhandled layout: br hr

Elements with special handling

• q: A quote. We will wrap the innerText quote marks ("s).
• sub and sup: Sub- and super-script. We will prefix the innerText with _ for subscript and ^ for superscript.

Undecided

• iframe: This content will be rendered and visible in the original article, so I feel we should really render it. It seems the srcdoc attribute can contain HTML source, which takes precedence over the src URL, so we should probably include this somehow if present. If the iframe only includes a URL src, I guess we should fetch the document ourselves if it is not fetched automatically by the GET request.

Remaining elements

These elements will be replaced with their innerText (concatenated text representations of all their children, with sensible whitespace rules for concatenation). a abbr address b bdi bdo cite code del dfn em i ins kbs mark q rb ruby rp rt rtc s samp small span strong u var wbr

Notes on classification of elements:

• abbr: Abbreviation or acronym. The title element can contain the expanded version of the abbreviation (e.g. <p>The <abbr title="Web Hypertext Application Technology Working Group">WHATWG</abbr> started working on HTML in 2004.</p>). We could present this as textContent (title) (i.e. <p>The WHATWG (Web Hypertext Application Technology Working Group) started working on HTML in 2004.</p>). However, this is unlikely to be displayed in the main browser rendering, so I'd suggest we just replace with innerText.
• bdi and bdo: Sets the text direction of an element, overriding any value inherited from parent elements. We suggest just using the innerText here as it's pretty much the only inline element we can't get rid of with genuinely no consequence. In this case text direction will be inferred from the parent dir attribute and other cues, which I think we can live with. Also, for this project all sites are english so we should not see any issues in our immediate use case.
• br: Readability.js wraps text separated by multiple brs in ps, so we can safely remove any remaining brs. In the long run we may want to handle this ourselves.
• del and ins: These indicate edits to the document, but the content will usually be displayed (with dome formatting to indicate an insertion or deletion). I think for us we should just keep the text either way as it is almost certainly visible to a user in a browser, so I think we should just replace with innerText.
• form: We remove all form elements and their contents.
• hr: Horizontal rule. This is really a paragraph level thematic break (e.g. scene or topic change). We should really ensure preceding and following text blocks are wrapped in a block-level element (wrapping them in a section or p ourselves if not). However, this has the same complexity as wrapping text blocks separated by multiple brs, so for now I'd suggest we don't, but we may want to consider doing this later. It may be Readability.js does this, so we should check.
• ruby: Used to provide ruby annotations, often representing additional representations of some base text, such as pronunciation guidance, translations or alternate character sets. Replacing the ruby element with the text contents of it's child rb, rt, rtc, rp elements is the standard fallback for user agents who cannot render ruby annotations and is what we will achieve by replacing all these elements with their innerText.
• s: Marks content as no longer relevant. The text is generally still displayed so I think we should just replace with innerText.

[martintoreilly]

@evelinag ☝️Please could you take a look at my plan for plain content generation above? ☝️

[sgibson91]

Adding new site tests

Plain content rules

One block level element should only contain either: 1) text; 2) or block level elements.

Any other scenarios fail tests.

In scenarios where we find paragraph, line of text, paragraph, line of text - these should be wrapped into a paragraph. Can be visually inspected.

Process

Run ExtractArticle.js to extract title, byline and content etc. of article. Then make minimal changes to the content to generate plain text.