spencermountain / wtf_wikipedia

a pretty-committed wikipedia markup parser
https://observablehq.com/@spencermountain/wtf_wikipedia
MIT License
779 stars 129 forks source link

'Chronological' output within a section #88

Open fractalien opened 6 years ago

fractalien commented 6 years ago

The parse method nicely yields arrays of sentences and links, but I see no way of knowing which lists were at which position. Is this loss of information deliberate, am I missing something, or was this just never an issue for anybody? A simple solution would seem to allow disabling the parsing of lists to somehow include them in the sentences.

spencermountain commented 6 years ago

Good point, maybe these should all be objects in a chronological array. On vacation, will think about this when im back

niebert commented 6 years ago

I assume it is more like a tree with children in a chronlogical order, in which the parse elements appear in the source wiki article. The article itself is the root node in an Abstract Syntax Tree AST The wiki article is split up in top level sections ==My Section Title ==

niebert commented 6 years ago

Started to propose a solution for the order problem of content element see GitHub Wiki

spencermountain commented 6 years ago

hey, this ordering-within-sections situation has been improved now with paragraph support. Unfortunately, (and to after considerable work) I learned how all templates, references, and tables can have an infinite number of newlines within them. This means that they will remain on Section for the medium-term. Images, lists, and sentences are now stored on Paragraph, so will now support a closer-to-truth ordering.

spencermountain commented 6 years ago

all the output methods will work normally, and should provide some better ordering out of the box cheers

niebert commented 6 years ago

Thank you Spencer, excellent, try to embed and integrate in [https://niebert.github.io/Wiki2Reveal Wiki2Reveal] cheers

fractalien commented 5 years ago

I'm sorry in case I'm missing something obvious, but how would I get the "better ordering out of the box"? Calling .json() on a section in 7.2.9, I still get lists after paragraphs, not in them. (Trying with this article.) Is there an accessor method to get both sentences and lists from a paragraph object?

spencermountain commented 5 years ago

hi @fractalien no you're right, this is still not possible.

fractalien commented 5 years ago

Thanks!

Edit: I thought I had found a workaround to my problem, but that wasn't one. The article's API-sourced markup fails to put a newline after the list (bug), yielding text containing "SunMercury". Given that the newer REST API seems not to offer a full-article markup output, I am between a rock and a hard place:

  1. Applying Wtf(…).json() to the erroneous API result obviously doesn't magically un-break the text.
  2. When using Wtf.fetch(…).json(), I get the lists separated from sentences, out of order.

I tried applying Wtf twice, essentially Wtf(Wtf.fetch(…).markdown()).json() to get proper parsing, but no separation between sentences and lists. Unfortunately, this leaves me with the whole markup intact as the alleged text of a singular section. Will have to dig further...

spencermountain commented 5 years ago

ha! this is very clever. i wish things were easier!