search

Raku / problem-solving

🦋 Problem Solving, a repo for handling problems that require review, deliberation and possibly debate
Artistic License 2.0
70 stars 16 forks source link

POD6 to RakuDoc revision process #375

Closed finanalyst closed 1 year ago

finanalyst commented 1 year ago

With the move to RakuAST, it seemed a good time to review the POD6 specification/documentation. There is more detail on this in a blog post I wrote on DEV.

There is also a minute by the Raku Steering Council about the process for the revision The aim is to find a way to get a timely change to a standard, whilst making the process transparent and open to the community.

We have just started the BETA period, which will last to 1 August. Damian Conway, who wrote most of the design of POD6, is taking an active part in this process.

Around the beginning of August, an issue will be opened in Raku/docs with the new RakuDoc document, and a link here. Hopefully a more stable version of the document will be available for the Raku community to comment on by that time. The GAMMA period will last to 1 November to give enough time for everyone to comment and for us to reach a consensus about the new specification.

The current state of the document is still a bit fluid because the structure has been changed to make parts of it clearer, to document things that existed before in POD6 but were unknown, RakuAST offers more that just $=pod and there are typos.....

But some things are beginning to clarify. For (a somewhat trivial) example, the markup language will be called RakuDoc (note the camel case) and the file extension will be .rakudoc (which it has been for some time)

For those who are interested, the latest iteration in HTML can be seen at new-raku rakudoc

Please hold back from making comments / suggestions yet. Everyone will have a chance to comment in August. The online version is DEFINITELY going to change in some way, so your red flag might disappear.

The document is quite long, and hopefully explains why RakuDoc offers more possibilities than just documenting code.

CIAvash commented 1 year ago

I know you said hold back from making comments, but something that was always missing in Rakudo and probably roast, is not recognizing formatting codes in declarator blocks:

#| Some B<subroutine>
sub foo {}

I think it should be supported(along with L<> and others), since they are docs as well.

I don't know the new RakuDoc supports this though.

finanalyst commented 1 year ago

@CIAvash FYI (since I didn't mention this explicitly above) @lizmat is also one of the group working on the BETA stage, so the above reference means your comment has been acknowledged.

JJ commented 1 year ago

Just my 2c

  1. I don't think a new POD6/Rakudoc should be a priority in any sense of the word. After all, Raku supports Slangs. Creating a new Slang in the ecosystem would be a better use of overall resources.
  2. I don't think this is solving any actual problem. As a matter of fact, the main problem is the fact that you need to run a program to parse Pod6. This has prevented it from having support in GitHub, for instance, despite all the work several members of the community put in. If you don't have POD6 support in GitHub, people are not going to write READMEs or others in Pod6. If that's not the case, there are not going to have enough examples to ask for support from GitHub (or others). Not that we would get it anyway, since see above, but still.
  3. Main problem with documentation is the existence and quality. There are 67 issues targeted for closure 4 days ago, with only 4 actual issues closed.

Everyone is entitled to spend their energy and resources whichever way they want. There's a whole ecosystem to pursue any kind of project you want, and Raku fortunately has the support for it. Involving the community, however, is already a resource-consumption item by itself, not to mention if it was actually accepted and it needed implementation in the core.

My position, then, is thanking you a lot for your passion and contributions, but firmly against any redefinition of Pod6 or inclusion of a RakuDoc along Pod6 in the core (which would be even worse)

Hope there's no issue with this comment, since it's not about the specifics, but about the general idea.

lizmat commented 1 year ago

Creating a new Slang in the ecosystem would be a better use of overall resources.

Agree that this should be possible. But in order to be able to do that in any user-friendly and future-proof manner, is still a long way off. I gave up on my attempts to even make the RakuAST pod parsing a slang (fwiw, the legacy pod parsing is most definitely NOT a slang).

As a matter of fact, the main problem is the fact that you need to run a program to parse Pod6.

More technically: parsing for Pod6 runs the BEGIN blocks and any compile-time actions. I have been thinking a lot about that, and see the following solutions:

Both have their pros and cons. The con of the second approach, is that some kind of code parsing would still be needed to handle configs. So I'm leaning towards the first option.

Main problem with documentation is the existence and quality

Agree that is a problem. But this effort is not about the content.

Everyone is entitled to spend their energy and resources whichever way they want.

I know that @finanalyst and @thoughtstream are not primarily interested in helping with the documentation content problem. And to be honest, at the moment I'm not that interested in it either, other than making sure any additions to Raku have at least a bare documentation nugget (case in point: Format and Formatter classes). So I don't think there's a lot of energy lost in that area.

against any redefinition of Pod6

This is not a redefinition of Pod6: it's a clarification / codification of many of Pod6's implicitly accepted features:

or inclusion of a RakuDoc along Pod6

RakuDoc = Pod6++. There will be no side-by-side in 6.e when RakuAST completely lands in core. Legacy features, such as $=pod, will continue to be available, but now fed from RakuDoc compatible statements.

@JJ I hope this will alleviate some of your concerns.

finanalyst commented 1 year ago

@JJ Some of the changes needed to move POD6 to RakuDoc were to meet long-standing issues. I started on this journey in response to an issue that you created, namely the ability to allow for templates rather than hard-coding HTML for each block type. That led to a close examination of the original design of POD6, which has not changed in RakuDoc. That led to new possibilities being recognised.

Once we have RakuDoc locked down, I will be writing a new renderer somewhat like Raku::Pod::Render, which will process RakuAST and not $=pod.

The difference is that $=pod needs to have the whole program compiled, and that takes a long time, so it is necessary to cache files. RakuAST becomes available earlier, so its unlikely caching will be needed.

Further, there are two use cases for POD6 and RakuAST, one interacts with code, which is useful for IDEs but leads to behaviour that Github does not like. The other use case is for standalone documentation, which is what is needed in README's.

I think what @lizmat is suggesting is that the -doc prevents the code-oriented patterns whilst allowing the standalone patterns.

In a real sense, we are not modifying POD6 so much as updating it to take into account long-term issues that are preventing POD6 from being more widely used.

@JJ whether you are happy with this or not, the RakuDoc revision has come about because of your energy and the work you put into the documentation suite.

patrickbkr commented 1 year ago

@JJ

If you don't have POD6 support in GitHub, people are not going to write READMEs or others in Pod6.

I do see several reasons why GitHub will probably never turn into a very good Raku module documentation viewer:

I guess for similar reasons GitHub doesn't support JavaDoc either.

But now that our module index sites have evolved more, I see no reason why someone would primarily view Raku distributions on GitHub instead of visiting some module index website. So I see a different way forward that I believe would push adoption of RakuDoc more than GitHub support: Have raku.land (or some other module index site) render module documentation and have a working CLI module documentation viewer. For me personally raku.land already is the main entry point to modules. I only go from raku.land to GitHub if I'm specifically interested in the version history. This parallels how I use metacpan.org vs GitHub to view Perl module documentation.

As a matter of fact, the main problem is the fact that you need to run a program to parse Pod6.

I agree this is a problem that we should solve some day, but I don't think this is the most important one. (I do understand that this is blocking GitHub support.)

@lizmat

  • an internal compiler flag that would inhibit any BEGIN time actions. This could e.g. be set whenever --doc is specified.
  • create a separate RakuDoc::Grammar that would treat code as "ambient" blocks as described in S26.

I suspect decalarator blocks will prove to be a pain point here as they reference a piece of code. I think the first option has more chance to keep declarator blocks working. I love the example in S26 and dream to one day have a renderer that supports this.

@finanalyst @lizmat

I think what @lizmat is suggesting is that the -doc prevents the code-oriented patterns whilst allowing the standalone patterns.

I really hope we can have a --doc that gives us both. People have been writing modules that modify the Raku compiler (thus depending on BEGIN). Examples are Red and OO::Monitors. We do need to execute BEGIN blocks for those to work. We shouldn't break that. But: We could have a separate --doc-safe or some such that gives us safety by not executing any user code (but would break in the aforementioned cases). Then platforms like GitHub can use --doc-safe, while raku.land could go down the sandbox + timeout + --doc route.

zag commented 1 year ago

@patrickbkr

I do see several reasons why GitHub will probably never turn into a very good Raku module documentation viewer:

Yes, you totally right. While there may be some challenges in using GitHub as a comprehensive Raku module documentation viewer, I encourage you not to give up hope. At the moment Github ran a closed beta preview of the new UI, and I'm currently working on support of pod6 for it. Technology and platforms are constantly evolving 😁

thank you

lizmat commented 1 year ago

I love the example in S26 and dream to one day have a renderer that supports this.

I don't see a reason why a RakuAST based renderer wouldn't be able to do that. I think all of the hooks are already there.

finanalyst commented 1 year ago

Reason it could not be implemented before is that Rakudo concatenated output input and code into pod-code-block so a renderer could not distinguish between the three. I did try. Not much point in changing this now.

RakuAST keeps the distinction, so a RakuAST renderer will do this immediately.

On Tue, 4 Jul 2023, 13:59 Elizabeth Mattijsen, @.***> wrote:

I love the example in S26 https://github.com/Raku/old-design-docs/blob/master/S26-documentation.pod#L384-L406 and dream to one day have a renderer that supports this.

I don't see a reason why a RakuAST based renderer wouldn't be able to do that. I think all of the hooks are already there.

— Reply to this email directly, view it on GitHub https://github.com/Raku/problem-solving/issues/375#issuecomment-1620207210, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACYZHFWK2JWDEPHKFO5QBLXOQHUDANCNFSM6AAAAAAZ3SZZ2I . You are receiving this because you were mentioned.Message ID: @.***>

lizmat commented 1 year ago

@CIAvash https://github.com/rakudo/rakudo/commit/5f488afb56 implements opt-in Markup support for declarator docs. See the commit description for more information.

jubilatious1 commented 1 year ago

Is RakuDoc (a.k.a POD6) 'Literate Programming'?

https://www.perl.com/pub/tchrist/litprog.html/

http://www.literateprogramming.com/knuthweb.pdf

http://www.literateprogramming.com/

thoughtstream commented 1 year ago

RakuDoc is not identical to literate programming.

However, it was influenced by that concept (i.e. stole ideas and mechanisms from it) and, with the introduction of placement links and the :hidden metatag, RakuDoc can certainly achieve the same code-in-one-order-with-documentation-in-another effects as classic literate programming.

jubilatious1 commented 1 year ago

I'll comment in a 'food-for-thought' manner... .

TL;DR RakuDoc/POD6 as a literate-programming Markdown replacement will have a ready supply of Data Science users.

I believe the concept of literate programming stretches back many decades, with various implementations in the meanwhile. I cited a Knuth paper earlier. I recall studying a bit about WEB and Noweb, and know a little about the difference between tangling and weaving.

All these concepts have been turned on their head with the advent of Data Science. The primary objective of literate programming today is to produce Data Science Reports, not code documentation.

In the R-programming language, this has meant the original implementation in Sweave (or S-weave, i.e. for the S programming language from IBM). That implementation was extended to the R language as well, and I recall using it with a TeX back-end.

All this work took a gigantic leap forward with 1) the advent of Markdown, and 2) release of the knitr (i.e. "knit"-R) package. Two of the terms used in that community are knitting and/or purling a document/script. Check out this PDF: "knitr: A General-Purpose Tool for Dynamic Report Generation in R".

Looking at the literature, and tools created, it seems that the focus now is NOT in creating a language that produces stand-alone static documentation, but either 1) generating reports, or 2) running code (with plaintext serving as inline documentation). See: "Using knitr and pandoc to create reproducible scientific reports".

So maybe we're looking at this issue backwards. There IS dissatisfaction in the Data Science community with Markdown, and (truth-be-told), a number of Extended Markdown Syntaxes have been developed, including CommonMark, GitHub Flavored Markdown (GFM), Markdown Extra, MultiMarkdown, and R Markdown.

My question to those reading this far: is RakuDoc/POD6 better than Markdown (and assorted flavors)?

If so (to address JJ's point), I would say full steam ahead. Markdown suffers from being anything-but-WYSIWYG. RakuDoc/POD6 doesn't seem to have that problem.

However, I've never been able to insert "code-fences" into RakuDoc/POD6 and get code output. Here knitr is a much more robust solution. See: Chunk Options, specifically the section on Code Evaluation. I think @patrickbkr is asking for the same capabilities in executing code (although I'm confused as to how RakuDoc/POD6 code output would be formatted inline).

All this...seems like a lot of work. But maybe less than anticipated, because R-programmers have moved on to the Rmarkdown package, which uses a Pandoc backend.

So why can't RakuDoc/POD6 use a Pandoc backend as well? For those who don't like to follow links, Pandoc (written in Haskell), inter-converts approximately 50 document formats.

https://github.com/jgm/pandoc

2colours commented 1 year ago

For what it's worth, I don't think Markdown has ever replaced (or will replace, for that matter) javadoc, doxygen kind of means of documentation. That's the kind of thing we failed to properly utilize with Raku POD so far.

Also, I think the sole fact that there is a need for so many "flavors" shows that Markdown is not a great fit. I suppose part of the reason for that is that these communities (also) didn't want to inter-depend and thought they knew better what they needed for themselves. In the light of this, I think it's apparent that designing something with Raku in mind, both syntactically and semantically, yields a better tool, and the question should be posed the other way around: what's so great about creating yet another semi-compatible Markdown dialect that one should give up on all POD stuff?

librasteve commented 1 year ago

I am a contrarian on POD6 and have refrained from chipping in on this thread since my plan has been to side step POD6 anyhow and didn't want to rain on the parade. But, then again, maybe this is a chance to get what I want.

My beef is that, coming from a Data Science and wider language perspective (Python in particular)...

  1. POD6 is something new to learn and I would rather use MarkDown (due to laziness)
  2. Notebooks in general and Jupyter in particular are the "standard" way to achieve literate programming

[Now I would slightly prefer doc embedded in code as opposed to code embedded in doc ... but I also see the value in using vanilla Jupyter for the vast number of possible converts to Raku]

So, my ideal is a tool that lets me adorn my Raku source with MD and then has bidirectional compatibility with .ipynb format. I would like my [Comma]IDE to be able to selectively hide doc &/ code please.

If this can be achieved with pandoc ... and my superficial reading suggests that pandoc may be a neat way to/from .ipynb, then count me in!

Edit: I am not advocating MD as a replacement for POD6, but I am hoping for a simple POD6 tag set that makes it possible to use MD in a .raku script (so legal, but not preferred) in a way that can round trip to an .ipynb ... and that's just a softish request since I guess I can continue to sidestep POD6 altogether.

2colours commented 1 year ago

That's why I would be curious if @antononcube has something to say about this, as somebody who has worked towards literate programming, notebook features and conversion between POD and Markdown.

On the other hand, I personally don't like how much this topic "gets owned" by the data science/literate programming discourse. If we think of either the old POD from Perl 5, or the name RakuDoc itself, it shines through that one of the main use cases (if not the main use case) would be the "case 1" of finanalyst's description, @leont's 2022 conference presentation also mostly discusses the eventual RakuDoc from this perspective. And indeed, one can see that we are not doing a lot with it; we barely even generate a README.md from it (I think that's about all App::Mi6 does in this regard). Compare that to even good old doxygen, for example. And then, unsurprisingly, people barely use it, even the most proficient module authors. I really don't think markdown is an answer to that problem in any sense, and at the same time, I think it's more just a beneficial side effect if RakuDoc turns out to be a powerful tool for code generation or notebook description...

finanalyst commented 1 year ago

POD6 was never fully implemented as originally intended. RakuDoc will be implemented fairly quickly and will be customisable from the start.

Also, the tools needed to generate MD and epubs as well as HTML should follow more easily too.

The customisation allows for Graphviz and Latex equations (Collection plugins already exist for these). This will allow for far more flexible documentation needs, for example showing the equation the code is implementing.

On Sun, 23 Jul 2023, 12:57 Márton Polgár, @.***> wrote:

That's why I would be curious if @antononcube https://github.com/antononcube has something to say about this, as somebody who has worked towards literate programming, notebook features and conversion between POD and Markdown.

On the other hand, I personally don't like how much this topic "gets owned" by the data science/literate programming discourse. If we think of either the old POD from Perl 5, or the name RakuDoc itself, it shines through that one of the main use cases (if not the main use case) would be the "case 1" of finanalyst's description https://new-raku.finanalyst.org/language/rakudoc, @Leont https://github.com/Leont's 2022 conference presentation https://youtu.be/2mkIZesnDl8?t=1110 also mostly discusses the eventual RakuDoc from this perspective. And indeed, one can see that we are not doing a lot with it; we barely even generate a README.md from it (I think that's about all App::Mi6 does in this regard). Compare that to even good old doxygen, for example. And then, unsurprisingly, people barely use it, even the most proficient module authors. I really don't think markdown is an answer to that problem in any sense, and at the same time, I think it's more just a beneficial side effect if RakuDoc turns out to be a powerful tool for code generation or notebook description...

— Reply to this email directly, view it on GitHub https://github.com/Raku/problem-solving/issues/375#issuecomment-1646821058, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACYZHGQ67MEKTJZ36UKZMLXRUGRRANCNFSM6AAAAAAZ3SZZ2I . You are receiving this because you were mentioned.Message ID: @.***>

jubilatious1 commented 1 year ago

I'm not advocating for a change to the POD6/RakuDoc specification, except adding the option to create reports with inline code output such as tables and graphs.

@2colours there isn't a reason to add another Markdown flavor. If POD6/RakuDoc is a better specification it will win out. But you raise a fair point: can POD6/RakuDoc replace both Doxygen and Markdown? Right now R-programmers write their documentation with roxygen2, and their Data Science reports with knitr or rmarkdown. If POD6/RakuDoc is more powerful than Doxygen I don't see why POD6/RakuDoc can't replace both Doxygen and Markdown for Raku module authors.

@librasteve I've used Jupyter notebooks, but never liked the closure model. It seemed you could be at the bottom of a document working with variable x, and then introduce a code snippet containing another variable x at the top. The document would chug along merrily without balking at the inconsistency. Has this changed? In knitr/rmarkdown the two different xs will be understood as the same variable x, which is what you want in a single, cohesive report document.

@finanalyst I've used Graphviz with R (external to the knitr/rmarkdown environment), but I'm pretty sure those packages generate HTML/PDF graphs for inclusion in documents. And math equations: there are Tex/LateX tools available for PDF, but I recall the HTML equations are displayed with MathJax.

@thoughtstream I'm sorry I couldn't get your links to work. But maybe my comments are too far afield? It seems reading the Pandoc repo that we would need some Haskell and/or Lua expertise to get Pandoc conversion from POD6/RakuDoc to other documentation formats.

Thanks everyone for the lively discussion!

jubilatious1 commented 1 year ago

From the Perl5 world:

https://metacpan.org/pod/Pod::Pandoc

https://pod-pandoc.readthedocs.io/en/latest/Pod-Pandoc.html

https://github.com/jgm/pandoc/issues/176

https://github.com/jgm/pandoc/issues/593

https://github.com/jgm/pandoc/issues/6488

finanalyst commented 1 year ago

@jubilatious1 @librasteve @2colours @thoughtstream @lizmat

RakuDoc update.

As explained in the first post in this thread, RakuDoc is currently being revised. @jubilatious1 the links by @thoughtstream were to a private repo, an on-line version can be found at new-raku. This is still not the final version, although close.

Most of the revision has been completed, and we are working our way through an issue related to markup in code blocks, and specifically whether RakuDoc instructions should be handled by the Raku parser, or the RakuDoc renderer.

Although most of RakuDoc can be rendered by Raku::Pod::Render, there are quite a few features of POD6 and now RakuDoc that have yet to be implemented. For example, the AUTHORS section right at the start (of the RakuDoc document) will appear at the end of the document, even though it is written at the start. The AUTHORS block is a semantic block, and it can be hidden from the rendered version, but placed elsewhere in the rendered version. In other words, the place where a section is written does not need to be where it is rendered, which was what @thoughtstream was referring to.

The RakuDoc document includes some significant explanations and examples that are missing from the POD6 document.

As mentioned in the first post, the design and wording of RakuDoc will be open to community discussion from August 1. I think we are on schedule for that date. Community discussion is slated to last until November.

@jubilatious1 When I mentioned GraphViz, Equations, and Leaflet maps, I meant they can be easily exposed using Custom blocks in RakuDoc.

Some of the examples on the same web page as the examples above are suffering bit-rot because they were written in compliance with the POD6 document, and they will need a bit of adjusting to bring them into compliance with RakuDoc.

@librasteve I understand that you may not like POD6, aka RakuDoc. I also sympathise completely with the pain of a learning curve for a new language. My pain is significantly higher now because I have to implement the additions to RakuDoc (some of which were in POD6, but not realised in a renderer).

Unlike you, I do not like MarkDown. It is too simple and constrained for content rich documents, eg. the Raku Documentation suite. I wanted the custom blocks I cite above for my own website. I put some websites together using award winning CMS systems, and I had to write the plugins for Maps for one of them. I found that their reliance on MarkDown as the underlying text component was inflexible. I also found that writing custom plugins was convoluted too. MarkDown does not have any extensibility specified, which is why there are so many flavours of it.

@jubilatious1 RakuDoc was not written for Literate programming, but it borrows from its concepts. From what I have read, RakuDoc could be used to accomplish the goals set out for literate programming. A challenge for the future will be to take some of the examples in the literate programming specification and show how to accomplish them in RakuDoc - perhaps by using some custom blocks.

As for a pandoc translator, that too sounds like an interest project for the future.

lizmat commented 1 year ago

Closing in lieu of #380

jubilatious1 commented 9 months ago

@codesections some commentary would be helpful, in light of:

https://www.codesections.com/blog/weaving-raku/

Can "Technical Reports" be generated with Raku, Markdown, POD6, RakuDoc?

Thx.

antononcube commented 9 months ago

Can "Technical Reports" be generated with Raku, Markdown, POD6, RakuDoc?

Yes, although I am not sure what "technical report" generation means.

  1. Markdown, Org-mode, and Pod6 documents can be used to do Literate Programming.
  2. Documents can be generated via document templates.
  3. Jupyter notebooks can be used to write interactively notebooks (with mixed text, graphics, and code) and export them to Mardkown, PDF, etc.
  4. Mathematica notebooks can be used to do Literate Programming.

Large fractions of the functionalities above are demonstrated and discussed in the TRC-2023 talk "Integrating Large Language Models with Raku".

jubilatious1 commented 9 months ago

@antononcube I'm looking for the Raku/POD6/RakuDoc equivalent of the following (RMarkdown example):

RMarkdown_outline

https://learning.nceas.ucsb.edu/2023-04-coreR/session_04.html

jubilatious1 commented 9 months ago

@antononcube RMarkdown example with output:

RMarkdown_example_with_output

https://learning.nceas.ucsb.edu/2023-04-coreR/session_04.html

antononcube commented 9 months ago

@antononcube RMarkdown example with output: [...]

This is the primary use case of for implementing "Text::CodeProcessing".

As for Pod6 notebook (interactive) solutions:

As for notebook solutions in general:

Here is a screenshot of a Jupyter notebook using a Raku kernel:

(It is a fragment of the LLM-workflows notebook I mentioned above.)

antononcube commented 9 months ago

@antononcube I'm looking for the Raku/POD6/RakuDoc equivalent of the following (RMarkdown example):

RMarkdown_outline

https://learning.nceas.ucsb.edu/2023-04-coreR/session_04.html

Please see the Pod6 text and code chunks tests here of the package "Text::CodeProcessing".

finanalyst commented 9 months ago

@antononcube @jubilatious1 This thread was closed when the gamma revision repo was set up. It would be better to take this discussion there. I have some specific thoughts about the question raised here, but I would prefer to keep the discussion with the RakuDoc discussion. Is this OK with you?