search

Raku / RakuDoc-GAMMA

Community review of the RakuDoc standard
Artistic License 2.0
2 stars 1 forks source link

Amend Rakudoc to use new features #50

Open finanalyst opened 1 week ago

finanalyst commented 1 week ago

@thoughtstream @lizmat @zag I have finally got my RakuAST-RakuDoc-Render working. It can now render both the renderer consistency file - rakudociem-ipsum.rakudoc and the RakuDoc specification source.

Some of the examples of the RakuDoc specification were written in an ad hoc manner because the features had not been fully implemented (eg. numbered headings and items).

The RakuDoc-GAMMA file needs amending to use RakuDoc directly.

In addition, some extra eyes on the renderings above would be appreciated. Known flaws (CSS and styling mainly):

zag commented 1 week ago

that's cool news about the RakuAST-RakuDoc-Render progress! i'm quite tied up with other tasks at the moment, but it's great to see this moving forward. thanks for keeping in the loop!

finanalyst commented 1 week ago

@zag I saw your podlite web work. It's looking good too.

thoughtstream commented 4 days ago

Outstanding work, @finanalyst!

Just a few points (for a sufficiently large value of “few”):

  1. The code block immediately after: The general syntax of =place is should have an :allow<R> (so that the R<URL> etc. are rendered as replaceables)

  2. In the code block immediately after: For example, to specify a para block in delimited form: there are two highlighting errors. On the first line the :meta<data> is highlighted oddly, as :meta\<data> (seemingly mistaking the a<data> for a formatting code). This also occurs on the first code block of the subsequent section **extended form. See also y<firstvalue>, w< B C I >, n<Experimental data>, etc. later in the rendering.

  3. Secondly, on the final line of the same code block, the =end para is not highlighted (I would have expected it to be).

  4. In the code block of the section Abbreviated form, the trailing non-para text is highlighted oddly. Random words are highlighted in cyan, or blue.

  5. In the section Aliases, immediately after: This would produce: The rendering of the disclaimer and the three bullet points seems...infelicitous. Though this might be a bug in the original source, not in the rendering.

  6. In the section Aliases, the final code block does not seem to be rendered correctly. (Lots of unclosed A<B, which doesn’t seem right).

  7. Throughout the document: many (maybe most) of the rendered code blocks seem to be visually outdenting their first lines. For example:

    sub MAIN(
        Str $first,    #= the first CLI parameter
        Int :$verbose, #= an integer to indicate output, 0 = no output
    )

=begin data :key<Virtues>
    Laziness
    Impatience
    Hubris
    =end data

    say 'The three virtues are:';
    say $=data<Virtues>;

  8. For some rendered examples, the relative indenting of text and bullet lists seems wrong. For example:

        The three suspects are:

. Happy
. Sleepy
. Grumpy

  9. In the section Multi-paragraph lists, the highlighting is mistaking the apostrophe in Let's for the start of a Raku string, which produces very odd highlighting.

  10. In the section Numbered definitions, I would argue that the autogenerated numbers on the definitions ought to be rendered the same as the definition text (i.e. bold)

  11. In the section Formulae, immediately after: ...may either be rendered “accurately”: The formulae are not in fact rendered accurately (or at all!)

  12. In the section Formatting codes, in the first table, the Instruction and HTML columns shouldn’t be actually rendered (they should show the raw RakuDoc and HTML equivalents). This is both a bug in the original file (see below) and an error in rendering (the renderer should escape the literal < and > in the plaintext HTML examples so that their meanings don’t leak through into the HTML rendering). As for the original RakuDoc source, the table:

    Inline markup   Instruction   Mnemonic              HTML equivalent     Markdown equivalent
=============   ===========   ===================   ===============     ===================
Basis           B<text>       "B for Basis"         <b>text</b>         **text**
Important       I<text>       "I for Important"     <i>text</i>         *text*
Unusual         U<text>       "U for Unusual"       <u>text</u>         __text__
Strikethrough   O<text>       "O for Overstrike"    <s>text</s>         ~~text~~
Superscript     H<text>       "H for High text"     <sup>text</sup>     ^text^
Subscript       J<text>       "J for Junior text"   <sub>text</sub>     ~text~
Code            C<text>       "C for Code"          <code>text</code>   `text`

    should be something like:

    Inline markup   Instruction   Mnemonic              HTML equivalent       Markdown equivalent
=============   ===========   ===================   ===============       ===================
Basis           C«B<text>»    "B for Basis"         C«<b>text</b>»        C«**text**»
Important       C«I<text>»    "I for Important"     C«<i>text</i>»        C«*text*»
Unusual         C«U<text>»    "U for Unusual"       C«<u>text</u>»        C«__text__»
Strikethrough   C«O<text>»    "O for Overstrike"    C«<s>text</s>»        C«~~text~~»
Superscript     C«H<text>»    "H for High text"     C«<sup>text</sup>»    C«^text^»
Subscript       C«J<text>»    "J for Junior text"   C«<sub>text</sub>»    C«~text~»
Code            C«C<text>»    "C for Code"          C«<code>text</code>   C«`text`»

  13. In the section Links, the first two indented L<...> examples are RakuDoc code blocks and should be rendered literally, (e.g. L< dealing with the filesystem | type/IO.Path.* >), not converted to HTML <a> constructs (e.g. <a href="type/IO.Path.txt"> dealing with the filesystem </a>)

  14. In the section Space-preserving text, the first code block is being misrendered, and loses the all-important newline after the S<, which makes the example wrong. Also the highlighting of the (incorrect) S<dot is weird.

  15. In the section Unicode and HTML references, the rendering of the final code block is not rendering the alternate display texts in the examples, making the examples unfathomably miraculuous. :-)

  16. In the section Indexing terms, the third and fourth code blocks are misrendered (possibly due to the use of :allowed B<> markup within them).

That’s quite a bit of nit-picking, I know, but it doesn’t detract from the overall achievement, which is extremely impressive. Especially, the table-handling, which is awesome!

finanalyst commented 3 days ago

@thoughtstream Thank you for the attention to detail. I noticed a number of the problems and addressed some of them. The indenting was/is problematic. But I think I have solved most of them. There were also some problems with the RakuDoc source itself that only came to light when I began rendering the source as opposed to the workarounds we used to illustrate the code.

Please have a look at the newest version of the rendered RakuDoc text. (I'll handle your comment on R<> for the next rendering)

The highlighting of metaoptions on RakuDoc blocks is a known flaw of the Rainbow highlighter and an issue has been raised. @patrickbkr has said he will deal with it in due course. I'll also have a go in due course.

It should be born in mind that the Raku highlighters (Patrick's Rainbow and Liz's deparse-highlighter) are the first ones that differentiate categories in RakuDoc, so kudos to them. In addition, most of the snippets in this source illustrate RakuDoc. All the other Raku documentation sources (some 700 files) illustrate Raku. Hence the priority is to get the Raku syntax right.

I have noticed some oddly rendered snippets for RakuDoc rendering, eg the example in the section on multi-paragraph lists. I'll look at this to see whether it is a Rainbow question or a RakuDoc source. [update] Its an issue where ' is used as a possessive, but Rainbow is parsing it as the start of a string. @patrickbkr (I'll raise an issue for this). I think that string delimiters in RakuDoc blocks need to be turned off. Problematic. [update 2] I think this needs to be handled by the renderer, not the highlighter. The highlighter does not 'know' whether a snippet is Raku or RakuDoc, but the renderer does. So, I will filter out string delimiter and string tokens and treat them as text in a RakuDoc snippet. [update 3] The above will not work because the tokeniser is ignoring the inner structure of the string

I think this may take a few iterations to get right.

patrickbkr commented 3 days ago

The highlighting of metaoptions on RakuDoc blocks is a known flaw of the Rainbow highlighter and an issue has been raised. @patrickbkr has said he will deal with it in due course. I'll also have a go in due course.

I think I'll need some more guidance on this to understand what the desired outcome is.

Given there is some =code block with an :allow(B). Should the highlighter generate Raku code tokens as usual and sprinkle RakuDoc tokens in between? I'd say that's the right thing to do when highlighting the RakuDoc file itself.

How is a pipeline to render some RakuDoc file meant to handle this? At one point it'll need to convert that block of Raku code to a marked up form. Will it first treat the content as plain text, find and remove the RakuDoc markup, pass the result to Rainbow and apply the removed RakuDoc markup afterwards? Will it pass the content unchanged to Rainbow and somehow try to make sense of the RakuDoc tokens the resulting token list contains?

[update 2] I think this needs to be handled by the renderer, not the highlighter. The highlighter does not 'know' whether a snippet is Raku or RakuDoc, but the renderer does. So, I will filter out string delimiter and string tokens and treat them as text in a RakuDoc snippet. [update 3] The above will not work because the tokeniser is ignoring the inner structure of the string

Unsure this will help, but are you aware, that you can use Rainbow to parse strings/RakuDoc/comments/Regexes directly? There are grammar rules for each of them. They are always listed below a comment # Starters ----- in here. You'll need to write a custom tokenize() function calling the respective rule. If this is in any way interesting, ping me, I can help get you started.

finanalyst commented 3 days ago

@patrickbkr I've found raising an issue on sourcehut quite difficult to navigate, so I'll respond here.

I was not aware that I could parse a RakuDoc string directly. That would be great. Please let me know how to start.

finanalyst commented 3 days ago

@thoughtstream I now have had the chance to go through your flaws. Appologies for responding without looking carefully. You have discovered a number of distinct errors. I think I have fixed 1,7, and 14 [update: other errors fixing slowly]

Here is a more careful analysis. Numbered as you did

thoughtstream commented 3 days ago

@finanalyst, that's already looking much better. (And I didn't detect any newly introduced bugs either :-)

Great work by you and @patrickbkr!