Pod documentation is incomplete and out of date

finanalyst commented 1 year ago

Problem or new feature

The Pod document was first written when two Pod->HTML renderers were being developed. Pod::To::HTML was chosen to be used with the Raku documentation site, but Pod::To::BigPage implemented some POD better.

Pod::To::BigPage is now long broken. Pod::To::HTML does not implement things like P<>.

The Raku Documentation site now uses Raku::Pod::Render, which is undocumented in the Pod document. It also implements more of the original Pod spec.

The Pod document here omits examples that would not work on either Pod::To::HTML or Pod::To::BigPage.

Raku::Pod::Render also allows for some cool new custom blocks.

Suggestions

I have rewritten parts of the Pod document, and changed the title to Rakudoc. Since it humanly very difficult to understand the changes without seeing the rendering, the new suggested version of the document can be seen on line as new-raku Rakudoc

Since there are numerous changes suggested here, I'll incorporate feedback and raise a PR here.

TODO

There are two extra changes coming

numbered lists and headers
:allow meta data for Code blocks.

finanalyst commented 1 year ago

:allow has been partially implemented, but highlighting is breaking :meta<tab> on =for name type blocks.

patrickbkr commented 1 year ago

I think the Pod documentation shouldn't focus on Raku::Pod::Render alone. There are other Pod implementations out there. From the top of my head:

I think in the long run we should make sure Pod is a standard that is simple enough to make it reasonably possible for multiple implementations to implement it entirely. I'd hate that we gather a large corpus of Pod documentation (in the official Raku docs as well as module documentation) and then have the different viewers fail to produce equivalent output.

That one standard we all agree on should be what is documented. Annotations that tell when parts are not yet implemented are fine.

@lizmat, @finanalyst, @zag, @Altai-man Currently lizmat is working on the RakuAST parts for Pod, finanalyst is working on Raku::Pod::Render and zag is working on Podlite / rakupod-rakudoc-blocks. I'm unsure if Altai-man or jnthn wrote the Comma Pod viewer. Maybe now is a good time to think about if we actually want to support Pod in its entirety or if we should think about stripping some bits.

Just an anecdote: I remember shimmerfairy trying to rework Pod processing in Rakudo. She called the project Supernova. I think she bit her teeth out on reworking the table parsing.

lizmat commented 1 year ago

Yeah, table parsing is still looking at me :-)

FWIW, I think the way forward is to deprecate the current Pod:: object architecture, and focus on the RakuAST::Doc:: object as the source to render from. This will e.g. allow rendering documentation without the need to create bytecode, and thus run any code (apart from BEGIN and perhaps CHECK phasers).

patrickbkr commented 1 year ago

FWIW, I think the way forward is to deprecate the current Pod:: object architecture, and focus on the RakuAST::Doc:: object as the source to render from. This will e.g. allow rendering documentation without the need to create bytecode, and thus run any code (apart from BEGIN and perhaps CHECK phasers).

Apart from the legacy Pod::, are there features in the Pod specification itself that are worth considering for stripping in your opinion? I ask, because the Rakudo parser is not the only "source of truth" for Pod content. Podlite has its completely separate parser written in JS and AFAIK Comma also has a parser separate from Rakudo, originally converted from the Rakudo parser to a Java one, a separate codebase since then.

lizmat commented 1 year ago

I don't think so.

The only thing that I think really needs to go is the way $=pod is populated. It's not needed in 99.9% of the cases when running a program.

finanalyst commented 1 year ago

I am ambivalent about the documentation page. I think other ways of rendering POD / Rakudoc should be documented here too, to show how practically Rakudoc can be used.

There will come a time when the page will need to be re-written again, to distinguish between what should be in Rakudoc, and what should be in a Renderer.

For example, the metadata. Some metadata is handled by the Rakudo compiler, eg., :allow<B I>, but other metadata, such :numbered is not. So, :numbered needs to be handled by the Renderer.

The original S26 specification for POD6, now Rakudoc, was never completely implemented by Pod::To::HTML, and the Pod documentation page was deficient.

Raku::Pod::Render started as a way to upgrade Pod::To::HTML, but branched out. It now implements more of S26 than Pod::To::HTML.

So, my view is - for the moment - to enhance the Pod page, and distinguish clearly between what is possible in different renderers.

finanalyst commented 1 year ago

The new-raku pod shows a new pod (but the numbered examples are not working yet). Also checkout the TOC list. Previously, the TOC list did not distinguish between 1/2/3 level headers. Now lower levels are indented to show the structure of the document better.

zag commented 1 year ago

I agree with @patrickbkr, the documentation should be written in accordance with the specification. It's a great challenge. Changes like this indicate implementation flaws, not specification complexity. 😀

I'm implementing Podlite (and before that Perl6::Pod) in strict accordance with S26. Even extensions like =Markdown, =Diagram, and =Toc, which are implemented in Podlite, adhere to the specification, even though they're not part of it. And it's great that s26 allows for small possibilities to create extensions. Raku Pod is wonderful in this way.

Let's keep Named blocks, which are implemented differently in various implementations outside the specification and official Pod documentation. thank you

coke commented 1 year ago

Please note that S26 is not "the specification" at this point, roast is, same with any of the other (retronymed) "speculations".

lizmat commented 1 year ago

Having noted that, but sadly the tests in roast don't really test the pod syntax, but what winds up in $=pod. Which, in a lot of cases, is suboptimal.

FWIW, I think we need to make an exception for pod6 / rakudoc at this point and keep S26 as leading until we've found a way to actually test syntax rather than implementation.

coke commented 1 year ago

make an exception for pod6 / rakudoc

👍

finanalyst commented 1 year ago

The problem is that S26 is a bit out of date. For example, it specifies :formatted in terms of FormatCodes, such as B and U. But most HTML now has the possibility to format, eg. Headings, using different Fonts, and different Font weights (not just normal and bold). So, in my re-writing of the Pod document, I have subsumed :formatted into :template, and allowed all blocks to be able to receive a different template.

Another example is that although custom blocks are allowed by S26, it does not specify how they should be shown in the TOC. As I began to implement Custom blocks, which are extraordinarily useful for a non-documentation website, I found I had to have a technique to put them into the TOC structure, hence I have proposed :toc / :toc and :headlevel(xx).

Further I think that the ability to have custom FormatCodes is super useful, but there is a need to have a way to attach meta data to FormatCodes. S26 has X< sss | skssk>, so I have implemented an extension of S26 that uniformly applies the idea to all new Format Codes.

finanalyst commented 1 year ago

However, I realise that I have introduced a number of extensions as I have worked to get all of S26 working for the new site.

Perhaps a better way would be to have as much of S26 as possible at the beginning of the document. All rendering examples are as specified in S26 and there is no mention of any of the renderers at all.

At the end of the document, there a section on each of the different Renderers, such as Pod::To::HTML, Pod::To::Text, Raku::Pod::Render, Podlite etc.

Each renderer has a section, indicating how it is installed, how it is used, any extensions from the S26 specification, examples (or links to example pages, or screenshots).

finanalyst commented 1 year ago

I have rewritten a version of the Rakudoc document, including several new sections of S26.

I have not included some, such as =alias and =finish.

I have included all the standard metadata, including :formatted, :nested and :margin. Except to be honest, I did not understand the :margin description entirely.

The problem is that the formatting of S26 got screwed up.

I have also removed all Raku::Pod::Render extensions and mentions to the bottom of the file, as suggested in the previous post.

See the PR for the concrete changes

finanalyst commented 1 year ago

@zag @lizmat @coke @patrickbkr @Altai-man Please take a look at the new version of pod on new-raku.

@zag the suggestion would be for you to add Podlite into the Render section. I think your work does need more visibility

zag commented 1 year ago

@finanalyst, thanks for your suggestion, but just to clarify, Podlite is not a just renderer - it's actually a completely new implementation of Raku Pod from scratch in TypeScript. So, it might not be appropriate to mention it in the Renderer section. thank you

zag commented 1 year ago

@finanalyst Could you please clarify the current status of doc/Language/pod.rakudoc document? Is it an updated version of the s26 specification that was mentioned earlier? Is it a guide for writing documentation using Raku Pod? Or is it something else?

thank you

finanalyst commented 1 year ago

@zag That is a very good question, and I don't have an answer.

@lizmat, @coke @patrickbkr would you please comment.

Here are some of my thoughts and perspective.

As Raku developed, there was a progression from initial architecture documents, and explanations, then specifications as developers began to write implementations of the language. Simultaneously, tests were written that any compiler of the language would need to fulfill. The process was very iterative. Specifications changed as tests were added, and tests changed as it became clear Raku (Perl 6) would work better if things were different.

For every part of Raku - that I am aware of - except Rakudoc (also known as POD6), it is the tests that now define the language. The documentation files describe the language that passes the tests.

Rakudoc/POD6 has been left behind. Language/pod.rakudoc did not document everything that was used to write the documentation files.

For example, =begin code :allow< C B > starts a code block in which C<> and B<> markup is allowed. I have found it very difficult to get this to work with the non-Raku syntax highlighter. If you look at my new-raku page for pod, it works, but the system fails in the operators.rakudoc page.

The :allow syntax was not documented in language/pod (lower case L for the url to the html page, uppercase for the rakudoc source), but it is specified in S26.

Another issue is =input and =output. They are clearly documented in language/pod. However, in the Test suite, it is expected that they are processed as Block::Code, and Rakudo (the current best compiler of Raku) generates Pod::Block::Code for all three of =code, =input, and =output. However, S26 indicates that there should be a visible difference between each of the three. But if the compiler does not output them properly, they cannot be rendered differently.

Here, we have a situation where the Test suite is clearly inadequate. Hence, the remark above that the Roast is not what defines Rakudoc. Furthermore, it would seem from a comment of @lizmat that tests related to Rakudoc actually only reflect what Pod::To::HTML produced, not what it should produce.

As I have tried to implement more aspects of the S26 specification, some of which are needed, I have found other limitations not in S26. For example, how to treat a custom block in the Table of Contents. While S26 does allow for Custom blocks, there is no specification about how these should specified. A aspect of modern HTML (which did not exist when S26 was written) is that a great deal of the rendering of a web page is split between HTML, CSS, and Javascript (including JQuery), to say nothing of React / Vue etc. I have resolved this problem for Raku::Pod::Render using the concept of 'plugins'. I have written Custom blocks that use third-party APIs, such as the Leaflet maps. The idea in Rakudoc of metadata associated with Blocks makes this easy, but it is not specified.

A very useful block that is in S26, but is not documented in language/pod, but is used in language/pod is =nested. In addition, =para is specified and is documented, but it is treated as a Named Pod::Block, it is included in the Table of Contents. Clearly a paragraph should not be in the TOC, so I have inferred that by default both =para and =nested should not be included, while all other Named Pod blocks should be in the TOC.

Another really good idea in S26 is =config which is a directive and not a Block. It provides meta tabs automatically to other blocks. I have implemented this in Raku::Pod::Render. But its power will be seen when I have finished work on implementing numbered items and numbered headings. The intention is that it will be possible to make =item numbered in some lexical scope by default. Currently, we only have un-numbered items.

Whilst there are some things in S26 that really should be available to Rakudoc, I believe some things should be left unimplemented. For instance, I see no need at all for =finish or =alias, for :nested or :formatted.

I should point out that I am not the only innovator here. Pod tables currently work differently to S26, and there are ideas to make Pod tables much more flexible.

There was another problem. The two previous renderers, namely Pod::To::HTML and Pod::To::BigPage both broke when handling 'standard' pod. For example, Pod::To::HTML does not handle P<> markup, while Pod::To::BigPage broke when handling N<>. Raku::Pod::Render manages most things now (but still chokes on :allow). The result is that some parts of language/pod do not have examples, eg. of footnotes.

When I re-wrote Language/pod.rakudoc as in the PR, my first intention was to keep to the original format, But I found all sorts of problems. In addition, a number of the extensions I had made for Raku::Pod::Render are not in S26 either. But I wanted to include things like numbered lists and headers, and =config.

There is the reasonable suggestion that the 'standard' features are described, and 'extensions' are described later. This is why I have put the Raku::Pod::Render documentation at the end.

This is a very interesting time for Rakudoc, and the specification. The reason is that Rakudo can now create AST versions of rakudoc files. This will allow for a faster renderer, but also better granularity. For example the AST version of a rakudoc file containing =input output and =code contains each of them as separate types.

patrickbkr commented 1 year ago

Rakudoc as both, a specification and implementation, was basically frozen for the last few years. I believe there is no point in treating S26 as some untouchable, perfect definition. Working on the implementation and using Rakudoc does help improving the Rakudoc specification. So now is the perfect time to improve the standard. I believe we should actively strive to define Rakudoc, both in tests and documentation. The previous comments already surfaced several needed additions, removals and changes to the current state of Rakudoc. I think we should be brave and go ahead with doing those changes. There is not enough usage of Rakudoc yet that would justify putting a strong emphasis on backwards compatibility. So I'd say: "Go, make Rakudoc something better and write the tests so we have a well defined standard in the future."

About how to document the different Rakudoc implementations:

language/pod should document what we define as the new Rakudoc standard. This documentation should be suitable for writing Rakudoc for modules to be displayed on raku.land, in Comma, or via the rakudoc CLI command. It should not document the behaviors of those tools, but simply stick to the specification. That's what a specification is for.
Individual implementations should be documented separately. The obvious place for their documentation is in their modules. (Sadly raku.land doesn't support Rakudoc yet, but that's a different topic.)
The Rakudoc flavor (not the implementation) used by docs.raku.org should be documented on docs.raku.org. I'd be fine with both, having that somewhere on language/pod clearly separated in a section called "docs.raku.org Rakudoc flavor" (akin to e.g. "GitHub Markdown flavor") or on a separate page that is prominently linked.

zag commented 1 year ago

I believe there is no point in treating S26 as some untouchable, perfect definition.

Yes, I agree with you.

I think most of the suggestions have to do with problems of incomplete s26 implementation and the resulting difficulties at the rendering level.

However, I believe the best approach is to first implement Pod as much as possible and have real-world usage examples before making any changes in specification.

Let me give you some examples:

if you remove :allow:
- you cannot insert a non-ASCII character using E<>
- can't significant text in verbatim blocks in man pages ( yes, documentation can be exported to formats other than HTML)
  - if you remove =alias, you can't maintain templates in Pod for complex documents. For example, you have a template:
```
The use of A<PROGNAME> is subject to the terms and conditions
laid out by A<VENDOR>, as specified at A<TERMS_URL>.
```
    And then overwrite variables:
```
=alias VENDOR      4D Kingdoms
P<template>
=alias VENDOR      Some other
P<template>
```

There are specific areas for the Raku language in s26 that may require updates for now, and I see areas where there is potential for improvement overall.

But I think it's necessary to start with the implementation. That's why I'm interested in keeping track of what @lizmat is doing, and I sincerely wish her success.🤞🍀

yes, a set (aka "The Rakudoc flavor ") of Named blocks, config params, and agreements to structure and maintain docs.raku.org look sensible.

thank you

zag commented 1 year ago

@finanalyst, I also have no use cases for :margin too (and =finish, and :formatted). Maybe it would be helpful to refer to the S26 design documents to get a better understanding of the ideas behind it. 😀 Otherwise, they can be removed... thank you

lizmat commented 1 year ago

FWIW, =finish should NOT be removed. It is also a debugging tool, specifically in test files, where you want to ignore the rest of a file without too much trouble.

finanalyst commented 1 year ago

@zag Are there any extensions you think are needed for Rakudoc (aka POD6)?

I like the use case you suggested for =alias (and the use case for =finish given by @lizmat)
Do you think :like is needed?
Do you think :nested is needed? There is a =nested block, which is useful, but making :nested applicable to all blocks seems unnecessarily generic.
S26 defines a D<> as a definition FormatCode in addition to =defn. I see a need for =defn, but I do not see a need for D<>. How would it differ from a C<>, or B<>?

@patrickbkr the idea of having a Rakudoc flavour for docs.raku.org is a great solution, whilst keeping language/pod as generic as possible.

I hope to create a slide-deck about Rakudoc, indicating what is implemented, what is tested, what is documented in language/pod, and what is in S26, and some extensions that would be useful.

My suggestion for extensions to S26 would be

all Format codes to have metadata in the same way X<> and L<> have.
line numbers available for blocks in the file where they are defined.
standard meta data relating to Table of Contents, eg :toc/ :!toc and :headlevel(x), which I have found useful.
for all pod-blocks to be embeddable, eg., =table inside a cell of an outer table =table and B<> inside a =defn
:column-header( <rownumber>/<row-number range> ), :column-align (syntax to be decided to define left/mid/right per column, :row-label like column-header, and :row-align
specify that Custom blocks are considered to be like =head for Table of Contents, but that =nested, =code, =input, =output, & =para are not included by default in TOC.

lizmat commented 1 year ago

Let me respond on this from the RakuAST point of view:

all Format codes to have metadata in the same way X<> and L<> have.

The RakuAST::Doc::Markup class has these attributes.

    has str  $.letter;     # the letter of Markup: A..Z
    has str  $.opener;     # opening sequence: < << «
    has str  $.closer;     # closing sequence: > >> »
    has str  $.separator;  # separator inside meta: for deparsing mainly
    has List $.atoms;      # any contents of this markup (str | nested markup)
    has List $.meta;       # any meta info (e.g. url | lemma | original value)

This implies that this is already supported in RakuAST.

line numbers available for blocks in the file where they are defined.

This should be covered by the origins framework, allowing one to retrieve file / line number of any RakuAST::Node object that originated from parsing source with a grammar (as opposed to manually building a RakuAST tree).

standard meta data relating to Table of Contents, eg :toc/ :!toc and :headlevel(x), which I have found useful.

The Rakuast::Doc::Block class has these attributes:

    has str  $.type;         # the type (e.g. "doc", "head", "item", etc)
    has int  $.level;        # the level (default "", or numeric 1..N)
    has Hash $.config;       # the config hash (e.g. :numbered, :allow<B>)
    has Bool $.abbreviated;  # bool: true if =item rather than =begin item
    has List $.paragraphs;   # the actual content

I think all necessary meta-data can be stored in here.

for all pod-blocks to be embeddable, eg., =table inside a cell of an outer table =table and B<> inside a =defn

The paragraphs attribute is a List of objects. It is agnostic to its contents. In the case of "type eq 'table'" it either contains RakuAST::Doc::Row object, or a string (if it is a row-divider).

The RakuAST class has these attributes:

    has str  $.column-dividers;  # a string with the column dividers seen
    has      $.column-offsets;   # array with start-points of columns in row
    has      $.cells;            # a List of cells
    has Bool $.multi-line;       # columns are multi-line

Note that currently the cells are strings only, but in fact they could contain anything, even another RakuAST::Doc::Block object (this allowing for embeddable tables).

:column-header( / ), :column-align (syntax to be decided to define left/mid/right per column, :row-label like column-header, and :row-align

The config attribute is a hash, it can contain anything, so all of this is just agreeing on semantics on what is found in the config.

specify that Custom blocks are considered to be like =head for Table of Contents, but that =nested, =code, =input, =output, & =para are not included by default in TOC.

Again, that could be specified in the config, could it not? Just need to agree on the semantics :-)

finanalyst commented 1 year ago

Following this discussion, it seems the content of language/pod needs to be reworked and split, rather than just updated (which is what I attempted with the PR).

Rakudoc is a slang of Raku and the RakuAST representation (which @lizmat just showed) provides far more information than the $=pod output. The $=pod representation does impose some limitations on how Rakudoc could be rendered. RakuAST means that future renderers will be able to do far more than the simple ones now.

A renderer transforms the markup into some format, such as HTML, Markup, even Text. Pod::To::HTML is an example of a render (others are mentioned above).

So what needs to be defined more clearly, I think, is the minimum adequate set of blocks, directives and metadata that a renderer must accept, and their semantics - that is a description of what they should do. And a description about to handle non-minimum markup.

Following @patrickbkr language/pod (may be language/rakudoc ?) should contain information about the minimum set, without examples. A separate page on doc.raku.org should contain examples of the minimum set of blocks. directives, and metadata of the flavour of Rakudoc that is used to render it and explicitly describe extensions. This would then allow users to see how a specific renderer treats the minimum set.

This marks a change from language/pod, which contained both the specification and examples of things.

This distinction between specification/semantics of a minimum set and examples of how a renderer does this should also make it easier for

the development of new renderers. The documentation for the renderer would contain a page showing the output for the minimum set, and the extensions it introduces;
the development of Rakudoc editors (such as Podlite - is this correct @zag ), which need to be able to manipulate the blocks, metadata and directives.

There seems to me a separate issue about testing. Would I be right in thinking that the Roast tests would only need to verify that there is enough information provided by the compiler for a renderer to meet the minimum set.

But perhaps there also needs to be a separate Rakudoc source, eg Test/rakudoc-test.rakudoc that a renderer, in contrast to a compiler, should be able to render. (Test/ would also contain a text file for the P<> markup).

finanalyst commented 1 year ago

Another question arises from @lizmat 's comment. What will be the canonical output of a Raku compiler? Would it be $=pod called from within a program, or the AST representation of a string slurped from a Rakudoc source?

Is it anticipated that a Renderer should adhere to the raku --doc=Format CLI format, where a class called Pod::To::Format exists in the environment with a render method. If so, what would be passed to the render method? Currently, the first argument is a pod block tree (which is what is provided by $=pod).

lizmat commented 1 year ago

FWIW, I think the use of $=pod should be deprecated in favour of a renderer that takes a RakuAST tree. Why? Because the RakuAST approach is way more flexible, and currently even based on $=pod, is about 6.5x as fast.

coke commented 1 year ago

I'd be happy to make these changes in the raku/doc test suite, at least.

finanalyst commented 1 year ago

closing this issue as the topics are covered in the RakuDoc revision process

Raku / doc