Closed finanalyst closed 1 year ago
I know you said hold back from making comments, but something that was always missing in Rakudo and probably roast, is not recognizing formatting codes in declarator blocks:
#| Some B<subroutine>
sub foo {}
I think it should be supported(along with L<>
and others), since they are docs as well.
I don't know the new RakuDoc supports this though.
@CIAvash FYI (since I didn't mention this explicitly above) @lizmat is also one of the group working on the BETA stage, so the above reference means your comment has been acknowledged.
Just my 2c
Everyone is entitled to spend their energy and resources whichever way they want. There's a whole ecosystem to pursue any kind of project you want, and Raku fortunately has the support for it. Involving the community, however, is already a resource-consumption item by itself, not to mention if it was actually accepted and it needed implementation in the core.
My position, then, is thanking you a lot for your passion and contributions, but firmly against any redefinition of Pod6 or inclusion of a RakuDoc along Pod6 in the core (which would be even worse)
Hope there's no issue with this comment, since it's not about the specifics, but about the general idea.
Creating a new Slang in the ecosystem would be a better use of overall resources.
Agree that this should be possible. But in order to be able to do that in any user-friendly and future-proof manner, is still a long way off. I gave up on my attempts to even make the RakuAST pod parsing a slang (fwiw, the legacy pod parsing is most definitely NOT a slang).
As a matter of fact, the main problem is the fact that you need to run a program to parse Pod6.
More technically: parsing for Pod6 runs the BEGIN
blocks and any compile-time actions. I have been thinking a lot about that, and see the following solutions:
--doc
is specified.RakuDoc::Grammar
that would treat code as "ambient" blocks as described in S26. This grammar would then be used whenever --doc
was specified.Both have their pros and cons. The con of the second approach, is that some kind of code parsing would still be needed to handle configs. So I'm leaning towards the first option.
Main problem with documentation is the existence and quality
Agree that is a problem. But this effort is not about the content.
Everyone is entitled to spend their energy and resources whichever way they want.
I know that @finanalyst and @thoughtstream are not primarily interested in helping with the documentation content problem. And to be honest, at the moment I'm not that interested in it either, other than making sure any additions to Raku have at least a bare documentation nugget (case in point: Format
and Formatter
classes). So I don't think there's a lot of energy lost in that area.
against any redefinition of Pod6
This is not a redefinition of Pod6: it's a clarification / codification of many of Pod6's implicitly accepted features:
=finish
being a directiveor inclusion of a RakuDoc along Pod6
RakuDoc = Pod6++. There will be no side-by-side in 6.e when RakuAST completely lands in core. Legacy features, such as $=pod
, will continue to be available, but now fed from RakuDoc compatible statements.
@JJ I hope this will alleviate some of your concerns.
@JJ Some of the changes needed to move POD6 to RakuDoc were to meet long-standing issues. I started on this journey in response to an issue that you created, namely the ability to allow for templates rather than hard-coding HTML for each block type. That led to a close examination of the original design of POD6, which has not changed in RakuDoc. That led to new possibilities being recognised.
Once we have RakuDoc locked down, I will be writing a new renderer somewhat like Raku::Pod::Render, which will process RakuAST
and not $=pod
.
The difference is that $=pod
needs to have the whole program compiled, and that takes a long time, so it is necessary to cache files. RakuAST becomes available earlier, so its unlikely caching will be needed.
Further, there are two use cases for POD6 and RakuAST, one interacts with code, which is useful for IDEs but leads to behaviour that Github does not like. The other use case is for standalone documentation, which is what is needed in README's.
I think what @lizmat is suggesting is that the -doc
prevents the code-oriented patterns whilst allowing the standalone patterns.
In a real sense, we are not modifying POD6 so much as updating it to take into account long-term issues that are preventing POD6 from being more widely used.
@JJ whether you are happy with this or not, the RakuDoc revision has come about because of your energy and the work you put into the documentation suite.
@JJ
If you don't have POD6 support in GitHub, people are not going to write READMEs or others in Pod6.
I do see several reasons why GitHub will probably never turn into a very good Raku module documentation viewer:
I guess for similar reasons GitHub doesn't support JavaDoc either.
But now that our module index sites have evolved more, I see no reason why someone would primarily view Raku distributions on GitHub instead of visiting some module index website. So I see a different way forward that I believe would push adoption of RakuDoc more than GitHub support: Have raku.land (or some other module index site) render module documentation and have a working CLI module documentation viewer. For me personally raku.land already is the main entry point to modules. I only go from raku.land to GitHub if I'm specifically interested in the version history. This parallels how I use metacpan.org vs GitHub to view Perl module documentation.
As a matter of fact, the main problem is the fact that you need to run a program to parse Pod6.
I agree this is a problem that we should solve some day, but I don't think this is the most important one. (I do understand that this is blocking GitHub support.)
@lizmat
- an internal compiler flag that would inhibit any BEGIN time actions. This could e.g. be set whenever
--doc
is specified.- create a separate
RakuDoc::Grammar
that would treat code as "ambient" blocks as described in S26.
I suspect decalarator blocks will prove to be a pain point here as they reference a piece of code. I think the first option has more chance to keep declarator blocks working. I love the example in S26 and dream to one day have a renderer that supports this.
@finanalyst @lizmat
I think what @lizmat is suggesting is that the -doc prevents the code-oriented patterns whilst allowing the standalone patterns.
I really hope we can have a --doc
that gives us both. People have been writing modules that modify the Raku compiler (thus depending on BEGIN
). Examples are Red
and OO::Monitors
. We do need to execute BEGIN
blocks for those to work. We shouldn't break that. But: We could have a separate --doc-safe
or some such that gives us safety by not executing any user code (but would break in the aforementioned cases). Then platforms like GitHub can use --doc-safe
, while raku.land could go down the sandbox + timeout + --doc
route.
@patrickbkr
I do see several reasons why GitHub will probably never turn into a very good Raku module documentation viewer:
Yes, you totally right. While there may be some challenges in using GitHub as a comprehensive Raku module documentation viewer, I encourage you not to give up hope. At the moment Github ran a closed beta preview of the new UI, and I'm currently working on support of pod6 for it. Technology and platforms are constantly evolving 😁
thank you
I love the example in S26 and dream to one day have a renderer that supports this.
I don't see a reason why a RakuAST based renderer wouldn't be able to do that. I think all of the hooks are already there.
Reason it could not be implemented before is that Rakudo concatenated output input and code into pod-code-block so a renderer could not distinguish between the three. I did try. Not much point in changing this now.
RakuAST keeps the distinction, so a RakuAST renderer will do this immediately.
On Tue, 4 Jul 2023, 13:59 Elizabeth Mattijsen, @.***> wrote:
I love the example in S26 https://github.com/Raku/old-design-docs/blob/master/S26-documentation.pod#L384-L406 and dream to one day have a renderer that supports this.
I don't see a reason why a RakuAST based renderer wouldn't be able to do that. I think all of the hooks are already there.
— Reply to this email directly, view it on GitHub https://github.com/Raku/problem-solving/issues/375#issuecomment-1620207210, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACYZHFWK2JWDEPHKFO5QBLXOQHUDANCNFSM6AAAAAAZ3SZZ2I . You are receiving this because you were mentioned.Message ID: @.***>
@CIAvash https://github.com/rakudo/rakudo/commit/5f488afb56 implements opt-in Markup support for declarator docs. See the commit description for more information.
Is RakuDoc (a.k.a POD6) 'Literate Programming'?
https://www.perl.com/pub/tchrist/litprog.html/
RakuDoc is not identical to literate programming.
However, it was influenced by that concept (i.e. stole ideas and mechanisms from it)
and, with the introduction of placement links and the :hidden
metatag,
RakuDoc can certainly achieve the same code-in-one-order-with-documentation-in-another effects
as classic literate programming.
I'll comment in a 'food-for-thought' manner... .
TL;DR RakuDoc/POD6 as a literate-programming Markdown replacement will have a ready supply of Data Science users.
I believe the concept of literate programming stretches back many decades, with various implementations in the meanwhile. I cited a Knuth paper earlier. I recall studying a bit about WEB
and Noweb
, and know a little about the difference between tangling and weaving.
All these concepts have been turned on their head with the advent of Data Science. The primary objective of literate programming today is to produce Data Science Reports, not code documentation.
In the R-programming language, this has meant the original implementation in Sweave
(or S
-weave, i.e. for the S
programming language from IBM). That implementation was extended to the R language as well, and I recall using it with a TeX
back-end.
All this work took a gigantic leap forward with 1) the advent of Markdown, and 2) release of the knitr
(i.e. "knit"-R
) package. Two of the terms used in that community are knitting and/or purling a document/script. Check out this PDF: "knitr: A General-Purpose Tool for Dynamic Report Generation in R".
Looking at the literature, and tools created, it seems that the focus now is NOT in creating a language that produces stand-alone static documentation, but either 1) generating reports, or 2) running code (with plaintext serving as inline documentation). See: "Using knitr and pandoc to create reproducible scientific reports".
So maybe we're looking at this issue backwards. There IS dissatisfaction in the Data Science community with Markdown, and (truth-be-told), a number of Extended Markdown Syntaxes have been developed, including CommonMark, GitHub Flavored Markdown (GFM), Markdown Extra, MultiMarkdown, and R Markdown.
My question to those reading this far: is RakuDoc/POD6 better than Markdown (and assorted flavors)?
If so (to address JJ's point), I would say full steam ahead. Markdown suffers from being anything-but-WYSIWYG. RakuDoc/POD6 doesn't seem to have that problem.
However, I've never been able to insert "code-fences" into RakuDoc/POD6 and get code output. Here knitr
is a much more robust solution. See: Chunk Options, specifically the section on Code Evaluation. I think @patrickbkr is asking for the same capabilities in executing code (although I'm confused as to how RakuDoc/POD6 code output would be formatted inline).
All this...seems like a lot of work. But maybe less than anticipated, because R-programmers have moved on to the Rmarkdown package, which uses a Pandoc backend.
So why can't RakuDoc/POD6 use a Pandoc backend as well? For those who don't like to follow links, Pandoc (written in Haskell), inter-converts approximately 50 document formats.
For what it's worth, I don't think Markdown has ever replaced (or will replace, for that matter) javadoc, doxygen kind of means of documentation. That's the kind of thing we failed to properly utilize with Raku POD so far.
Also, I think the sole fact that there is a need for so many "flavors" shows that Markdown is not a great fit. I suppose part of the reason for that is that these communities (also) didn't want to inter-depend and thought they knew better what they needed for themselves. In the light of this, I think it's apparent that designing something with Raku in mind, both syntactically and semantically, yields a better tool, and the question should be posed the other way around: what's so great about creating yet another semi-compatible Markdown dialect that one should give up on all POD stuff?
I am a contrarian on POD6 and have refrained from chipping in on this thread since my plan has been to side step POD6 anyhow and didn't want to rain on the parade. But, then again, maybe this is a chance to get what I want.
My beef is that, coming from a Data Science and wider language perspective (Python in particular)...
[Now I would slightly prefer doc embedded in code as opposed to code embedded in doc ... but I also see the value in using vanilla Jupyter for the vast number of possible converts to Raku]
So, my ideal is a tool that lets me adorn my Raku source with MD and then has bidirectional compatibility with .ipynb format. I would like my [Comma]IDE to be able to selectively hide doc &/ code please.
If this can be achieved with pandoc ... and my superficial reading suggests that pandoc may be a neat way to/from .ipynb, then count me in!
Edit: I am not advocating MD as a replacement for POD6, but I am hoping for a simple POD6 tag set that makes it possible to use MD in a .raku script (so legal, but not preferred) in a way that can round trip to an .ipynb ... and that's just a softish request since I guess I can continue to sidestep POD6 altogether.
That's why I would be curious if @antononcube has something to say about this, as somebody who has worked towards literate programming, notebook features and conversion between POD and Markdown.
On the other hand, I personally don't like how much this topic "gets owned" by the data science/literate programming discourse. If we think of either the old POD from Perl 5, or the name RakuDoc itself, it shines through that one of the main use cases (if not the main use case) would be the "case 1" of finanalyst's description, @leont's 2022 conference presentation also mostly discusses the eventual RakuDoc from this perspective. And indeed, one can see that we are not doing a lot with it; we barely even generate a README.md from it (I think that's about all App::Mi6 does in this regard). Compare that to even good old doxygen, for example. And then, unsurprisingly, people barely use it, even the most proficient module authors. I really don't think markdown is an answer to that problem in any sense, and at the same time, I think it's more just a beneficial side effect if RakuDoc turns out to be a powerful tool for code generation or notebook description...
POD6 was never fully implemented as originally intended. RakuDoc will be implemented fairly quickly and will be customisable from the start.
Also, the tools needed to generate MD and epubs as well as HTML should follow more easily too.
The customisation allows for Graphviz and Latex equations (Collection plugins already exist for these). This will allow for far more flexible documentation needs, for example showing the equation the code is implementing.
On Sun, 23 Jul 2023, 12:57 Márton Polgár, @.***> wrote:
That's why I would be curious if @antononcube https://github.com/antononcube has something to say about this, as somebody who has worked towards literate programming, notebook features and conversion between POD and Markdown.
On the other hand, I personally don't like how much this topic "gets owned" by the data science/literate programming discourse. If we think of either the old POD from Perl 5, or the name RakuDoc itself, it shines through that one of the main use cases (if not the main use case) would be the "case 1" of finanalyst's description https://new-raku.finanalyst.org/language/rakudoc, @Leont https://github.com/Leont's 2022 conference presentation https://youtu.be/2mkIZesnDl8?t=1110 also mostly discusses the eventual RakuDoc from this perspective. And indeed, one can see that we are not doing a lot with it; we barely even generate a README.md from it (I think that's about all App::Mi6 does in this regard). Compare that to even good old doxygen, for example. And then, unsurprisingly, people barely use it, even the most proficient module authors. I really don't think markdown is an answer to that problem in any sense, and at the same time, I think it's more just a beneficial side effect if RakuDoc turns out to be a powerful tool for code generation or notebook description...
— Reply to this email directly, view it on GitHub https://github.com/Raku/problem-solving/issues/375#issuecomment-1646821058, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACYZHGQ67MEKTJZ36UKZMLXRUGRRANCNFSM6AAAAAAZ3SZZ2I . You are receiving this because you were mentioned.Message ID: @.***>
I'm not advocating for a change to the POD6/RakuDoc specification, except adding the option to create reports with inline code output such as tables and graphs.
@2colours there isn't a reason to add another Markdown flavor. If POD6/RakuDoc is a better specification it will win out. But you raise a fair point: can POD6/RakuDoc replace both Doxygen and Markdown? Right now R-programmers write their documentation with roxygen2, and their Data Science reports with knitr or rmarkdown. If POD6/RakuDoc is more powerful than Doxygen I don't see why POD6/RakuDoc can't replace both Doxygen and Markdown for Raku module authors.
@librasteve I've used Jupyter notebooks, but never liked the closure model. It seemed you could be at the bottom of a document working with variable x
, and then introduce a code snippet containing another variable x
at the top. The document would chug along merrily without balking at the inconsistency. Has this changed? In knitr
/rmarkdown
the two different x
s will be understood as the same variable x
, which is what you want in a single, cohesive report document.
@finanalyst I've used Graphviz with R (external to the knitr
/rmarkdown
environment), but I'm pretty sure those packages generate HTML/PDF graphs for inclusion in documents. And math equations: there are Tex/LateX tools available for PDF, but I recall the HTML equations are displayed with MathJax.
@thoughtstream I'm sorry I couldn't get your links to work. But maybe my comments are too far afield? It seems reading the Pandoc repo that we would need some Haskell and/or Lua expertise to get Pandoc conversion from POD6/RakuDoc to other documentation formats.
Thanks everyone for the lively discussion!
@jubilatious1 @librasteve @2colours @thoughtstream @lizmat
As explained in the first post in this thread, RakuDoc is currently being revised. @jubilatious1 the links by @thoughtstream were to a private repo, an on-line version can be found at new-raku. This is still not the final version, although close.
Most of the revision has been completed, and we are working our way through an issue related to markup in code blocks, and specifically whether RakuDoc instructions should be handled by the Raku parser, or the RakuDoc renderer.
Although most of RakuDoc can be rendered by Raku::Pod::Render
, there are quite a few features of POD6 and now RakuDoc that have yet to be implemented. For example, the AUTHORS section right at the start (of the RakuDoc document) will appear at the end of the document, even though it is written at the start. The AUTHORS block is a semantic block, and it can be hidden from the rendered version, but placed elsewhere in the rendered version. In other words, the place where a section is written does not need to be where it is rendered, which was what @thoughtstream was referring to.
The RakuDoc document includes some significant explanations and examples that are missing from the POD6 document.
As mentioned in the first post, the design and wording of RakuDoc will be open to community discussion from August 1. I think we are on schedule for that date. Community discussion is slated to last until November.
@jubilatious1 When I mentioned GraphViz, Equations, and Leaflet maps, I meant they can be easily exposed using Custom blocks in RakuDoc.
Some of the examples on the same web page as the examples above are suffering bit-rot because they were written in compliance with the POD6 document, and they will need a bit of adjusting to bring them into compliance with RakuDoc.
@librasteve I understand that you may not like POD6, aka RakuDoc. I also sympathise completely with the pain of a learning curve for a new language. My pain is significantly higher now because I have to implement the additions to RakuDoc (some of which were in POD6, but not realised in a renderer).
Unlike you, I do not like MarkDown. It is too simple and constrained for content rich documents, eg. the Raku Documentation suite. I wanted the custom blocks I cite above for my own website. I put some websites together using award winning CMS systems, and I had to write the plugins for Maps for one of them. I found that their reliance on MarkDown as the underlying text component was inflexible. I also found that writing custom plugins was convoluted too. MarkDown does not have any extensibility specified, which is why there are so many flavours of it.
@jubilatious1 RakuDoc was not written for Literate programming, but it borrows from its concepts. From what I have read, RakuDoc could be used to accomplish the goals set out for literate programming. A challenge for the future will be to take some of the examples in the literate programming specification and show how to accomplish them in RakuDoc - perhaps by using some custom blocks.
As for a pandoc translator, that too sounds like an interest project for the future.
Closing in lieu of #380
@codesections some commentary would be helpful, in light of:
https://www.codesections.com/blog/weaving-raku/
Can "Technical Reports" be generated with Raku, Markdown, POD6, RakuDoc?
Thx.
Can "Technical Reports" be generated with Raku, Markdown, POD6, RakuDoc?
Yes, although I am not sure what "technical report" generation means.
Large fractions of the functionalities above are demonstrated and discussed in the TRC-2023 talk "Integrating Large Language Models with Raku".
@antononcube I'm looking for the Raku/POD6/RakuDoc equivalent of the following (RMarkdown
example):
https://learning.nceas.ucsb.edu/2023-04-coreR/session_04.html
@antononcube RMarkdown example with output:
https://learning.nceas.ucsb.edu/2023-04-coreR/session_04.html
@antononcube RMarkdown example with output: [...]
This is the primary use case of for implementing "Text::CodeProcessing".
As for Pod6 notebook (interactive) solutions:
As for notebook solutions in general:
Here is a screenshot of a Jupyter notebook using a Raku kernel:
(It is a fragment of the LLM-workflows notebook I mentioned above.)
@antononcube I'm looking for the Raku/POD6/RakuDoc equivalent of the following (
RMarkdown
example):
https://learning.nceas.ucsb.edu/2023-04-coreR/session_04.html
Please see the Pod6 text and code chunks tests here of the package "Text::CodeProcessing".
@antononcube @jubilatious1 This thread was closed when the gamma revision repo was set up. It would be better to take this discussion there. I have some specific thoughts about the question raised here, but I would prefer to keep the discussion with the RakuDoc discussion. Is this OK with you?
With the move to RakuAST, it seemed a good time to review the POD6 specification/documentation. There is more detail on this in a blog post I wrote on DEV.
There is also a minute by the Raku Steering Council about the process for the revision The aim is to find a way to get a timely change to a standard, whilst making the process transparent and open to the community.
We have just started the BETA period, which will last to 1 August. Damian Conway, who wrote most of the design of POD6, is taking an active part in this process.
Around the beginning of August, an issue will be opened in Raku/docs with the new RakuDoc document, and a link here. Hopefully a more stable version of the document will be available for the Raku community to comment on by that time. The GAMMA period will last to 1 November to give enough time for everyone to comment and for us to reach a consensus about the new specification.
The current state of the document is still a bit fluid because the structure has been changed to make parts of it clearer, to document things that existed before in POD6 but were unknown, RakuAST offers more that just
$=pod
and there are typos.....But some things are beginning to clarify. For (a somewhat trivial) example, the markup language will be called
RakuDoc
(note the camel case) and the file extension will be.rakudoc
(which it has been for some time)For those who are interested, the latest iteration in HTML can be seen at new-raku rakudoc
Please hold back from making comments / suggestions yet. Everyone will have a chance to comment in August. The online version is DEFINITELY going to change in some way, so your red flag might disappear.
The document is quite long, and hopefully explains why RakuDoc offers more possibilities than just documenting code.