Rakudoc in third-party modules and some related topics

[vrurg]

This is a meta-issue as it is about to cover a few topics I stumbled upon while writing docs for my last module. Perhaps it would make sense to spawn more specialized issues later, but for now it feels to me like having more sense in providing general overview of the situation.

It is also to be noted that I've got an answer to at least one of my question, but it was part of an undocumented conversation. Therefore it'd be included here as well. Other answers could be lurking out there too, but they're not known to me.

I would try to structure this from the perspective of a module developer, possibly not very experienced with documenting in Raku.

Where To Start?

Turns out we don't have a manual on how to write manuals. Something I'd rather expect to be part of module development section in the Raku documentation, or be linked to from the section. What I would like to see there is:

1. Where to start in general, including such topics as recommended sections and styling in general
2. How to organize the docs from distribution perspective: file locations, pros and cons of using dedicated .rakudoc, or inlining into .rakumod, etc.
3. What tools to use.

I'd say this all is better be a single piece since, as a rule, writing docs is the final stage of project development.

Docs In An Installed Distribution

Giving further development to the item 2 from from above, there are cases when having documentation inlined into a .rakumod is not considered a good idea. My own latest example is LibXML::Class::Manual which is not a module and better not be. Other cases may include documenting classes which are declared within a module but one wants them to be available for a documentation reading tool by their full names:

unit module Foo;

class Bar {...}

And then:

$ rakudoc Foo::Bar

Our current Raku documentation states that .rakudoc are to be placed under doc/ directory of distribution source tree. But the directory is never included into precompiled installation. Thus, zef install Foo and then wiping zef's cache would leave one with no access to these docs whatsoever.

Though, for the clarity, even having the cache preserved is of little help because one has to know where is it located before using it. Moreover, if they don't know about the cache existence then it's not much different from not having it in first place.

The Reading Tool

For those coming from Perl background, having rakudoc is quite natural expectation. To be frank, I don't know what is the situation with documentation readers in other languages, but having one should be considered an advantage. From this point of view I tend to consider having the rakudoc as the standard.

Either way, there is no alternative to it presently. But in its current form it is barely useful too.

First of all, it requires a document to follow its demands with regard to the structure and it is very rigid in these demands. In particular, it has to be =TITLE, not =head TITLE, not =head1 TITLE. But then again, why =TITLE and =SUBTITLE, after all? My own preference is man-styled docs with =NAME, =SYNOPSIS, =DESCRIPTION, etc. In large due to keeping in mind that this how they should be used too.

The other requirement is to have :kind, :subkind, and :category keys on =begin pod – but what are they? What for? Seems like this is something apparent to The Raku Documentation project participants, but they are an undocumented mystery to me.

It is understandable that since rakudoc is not part of Rakudo and, apparently and definitely, of the Raku language, I cannot impose any requirements on it. But without an alternative implementing more democratic rules I'd have these question open. It is irrelevant to me wether they'd be answered by rakudoc or somebody picks up the task and implements a "competitor".

And then, once again, access to .rakudoc in precompiled distros...

Tooling?

Having some kind of standard tool or tools to deal with documentation, starting from verifying it and all the way down to producing output formats and even publishing them – it's a dream. Let it be here, just in case...

BTW, with RakuAST implementing this part would become more feasible.

TOC?

I'm not even certain as to what to ask about here. Some documents (the above mentioned manual, in particular) tend to grow big. Having TOC would help a lot in navigating them. Even if it's not clickable since having a document overview already eases search for necessary information.

Some docs are so tiny that TOC in them would be an overkill.

Having TOC as a part of Rakudoc standard seems to be an advantage. Making it available for opt-in/opt-out would be great too.

Rakudoc Cross-linking

L<> tag is giving more questions than answers. The first and the biggest problem is when it comes down to multi-format output (HTML, Markdown, etc.) where there could not only be different file extensions but even different directory structure – if there is any directories at all like when everything is thrown into a single PDF, for example. There is a real-life case of https://raku.land where links in READMEs are very fragile and often just do not work.

What aspects of linking we can take as axioms?

• anchors (explicit or implicit)
• inter-document
• cross-document
• the already mentioned multi-format output with different file extensions and URI schemas

A typical question of linking would look like: in a doc for module Foo::Bar::Baz I need to link to a paragraph in Foo::Fubar documentation so, that it would work in a local HTML; on a web page; on a version control service provider like GitHub, GitLab, whatever else is one's choice; etc.

Perhaps this question doesn't have a comprehensive answer, but at the moment there is barely anything reliable but WWW URLs. Not even relative paths are sufficiently good for the task.

[vrurg]

My own two cents.

First, about TOC. Best is to have =toc so one can chose where exactly they want to see it. This is best in cases when a kind of preface is required.

Speaking of links, @finanalyst has mentioned during our last RSC meeting that we can introduce, say :id configuration key, making it possible to create reliable links. The problem in general for now is when we have, say =head My Ideas there is no way to link to it in a way that when it gets renamed to =head Our Ideas the old L<> tags would still point out to the section.

Another matter to cover is that limiting links with only certain kinds of Rakudoc elements (like headings) make it harder for both the writer and the reader: "See section L, where you would find a list with an item of interest." Nah!

Look at the method L<foo|#method-foo> down below.

...

=begin item :id<method-foo>
B<C<method foo(...)>>

Hope you found what you're looking for here!
=end item

Having :id available for any (or almost any) kind of Rakudoc block would be very beneficial for docs readability.

Another point is cross-module/-document linking. Speaking of the Foo::Bar::Baz -> Foo::Fubar example, best if one can simply use L<Foo::Fubar|#some-id> and the rest would be taken care of by the tools. Moreover, I'd make the pipe | optional and have it just as `L<Foo::Fubar#some-id> in this particular case. Others would still require the separator.

[vrurg]

Having to always define :id for any potential anchor is too much. Therefore some rules of implicit anchor naming must be defined. But they have to be clear and standard for any output format. Whatever is eventually generated in the output, anchor name in Rakudoc would only depend on what Rakudoc itself states. So, one would know that =head Our Ideas can be referenced as #our-ideas at any time.

It feels to me that these rules are to be coded in RakuAST parser. Tools would simply pick up what they're offered with. This should also simplify producing links for something like L<Our Ideas> as the corresponding object would already have the target anchor name in it.

[finanalyst]

I gave a 10 min presentation to the Core summit about changes needed in Rakudoc. As a result of the 3hours of discussion, I have rewritten the Rakudoc document / specification, making it backwards compatible, but also explaining a lot more about certain features of Rakudoc (aka POD6), such as metadata, the difference between code-oriented Rakudoc and page-oriented Rakudoc, directives vs blocks, etc.

A first draft of the Rakudoc document is currently being reviewed by @lizmat As soon as she is through, it will be sent for a BETA review to people who have been implementing Rakudoc in various ways. After the BETA review and incorporating changes, the document will be placed somewhere (as a Raku/Doc PR) for a wider community review. Also I will release a series of blogs about the changes to explain the rationale behind them.

As to the requests above (this is not an exhaustive list of the revisions):

• An original unimplemented specification in S26 (the speculation about POD6) was P<:toc> which would place a TOC where the author wanted. This will be a part of the revised Rakudoc.
  • There is a problem in that for a large project like Raku documentation the designer will want to place the ToC in a standard place, such as a sidebar.
• The idea of an :id<some-custom-anchor> will be in the Rakudoc revision, and will be applied to all blocks.
  • It is for the author to guarantee that the id is unique, a conflict is resolved by the browser or renderer.
  • If a =head block exists, then it automatically generates an anchor (as mentioned above), which is guaranteed to be unique. However, a custom :id will also generate an anchor at the same place. This way a document author can create an easy to remember anchor for their own documents.
  • The ability to add custom metadata already exists by using the =for directive. So if a =head1 This is a header exists and the id needs to be customised, then the new code will be:

    =for head1 :to<easy_anchor>
    This is a header

    A compliant renderer will be expected to generate both the this_is_a_header and the easy_anchor anchors.

• The need for a way to link by Module was noted at the Core summit with a new schema for L<> and P<> markup, namely mod://Some::Module. The syntax will be included in the Rakudoc revision, but the semantics have yet to be fully nailed down. It will link to the rakuland site though.
• About the :kind :subkind metadata in Raku documentation files, this is a poorly documented feature of POD6. This data is used by the Renderer of the Raku documentation to compile cross document indexes and secondary files. Only some metadata is specified by the new Rakudoc revision, and if it is not in the specification, then it is ignored, unless a customised renderer is used.
• About how to create documentation, and ways to view any module's documentation, this is indeed a need. There is a part of the META6 specification that includes a 'doc' directory, but this not used or recognised yet.

In addition, the move to RakuAST will change Rakudoc considerably. It will no longer need to be accessed via the $=pod mechanism, but will be primarily found using an .AST call on a file.

Hopes this explains some of the progress that is on its way.

[patrickbkr]

• The need for a way to link by Module was noted at the Core summit with a new schema for L<> and P<> markup, namely mod://Some::Module. The syntax will be included in the Rakudoc revision, but the semantics have yet to be fully nailed down. It will link to the rakuland site though.

The specification shouldn't dictate, which platform module links are resolved to. Once raku.land has gained Rakudoc support it makes sense for docs.raku.org to link to raku.land (as that's currently the dominant platform). But other documentation providers should be free to link somewhere else. E.g. a command line documentation reader should try to resolve and display module links itself.

@finanalyst I guess this is what you meant. But I wanted to be sure.

[finanalyst]

@patrickbkr yes, you're right about retaining a generic interface, but what I meant about 'the semantics have yet to be fully nailed down' is to work out how to do this. The reference to rakuland is more about an initial default. Because Rakudoc must be customisable, every renderer will have to be configurable in some way. I don't want to define how a renderer should be configured. However, by defining the behaviours of some blocks, say the way numbering works with =head and =item, Rakudoc is being specific about defaults, allowing Renderer developers to provide other behaviours. The =config directive to change behaviours. oO ( maybe we need to have a metadata option like :mod-prefix that by default is set to eg rakuland, then if a document writer wants a different behaviour, they could put

=config Markup-L mod-prefix 'https://another-site.eg/dist?'

) end of speculation Would this suit?

[patrickbkr]

I think documentation software will fall in one of several categories:

• Software to display the official Raku documentation which may contain links to ecosystem modules -> hard code raku.land or let the user decide via some configuration on the website
• Generic ecosystem module documentation viewers (raku.land, the rakudoc CLI tool) -> They should open links in their own interface.
• Software to create or display documentation potentially unrelated to Raku. (podlite?) -> Won't have module links.
• A company may have internal Raku code and documentation which they view in a private installation of e.g. raku.land. -> Configure that installation to resolve links to that private installation itself.

Following the above categories I see two cases:

• the reader wants to change where such links go to (i.e. by setting some option in the viewer program / website)
• the software presenting the documentation decides

@finanalyst Can you come up with an example where it would make sense for the mod-prefix to be specified in the documentation itself?

[vrurg]

I don't think there must be any suggestion/defaults on the Rakudoc side as to where module links must point at. Neither implicit, by the standard; nor explicit by document creator. In the latter case there is always an option of inserting the link manually, as a URL, since it's about to be rigidly hardcoded anyway.

This has to be 100% on the reader side.

[finanalyst]

@vrurg I'm having difficulties reconciling your two comments:

best if one can simply use L<Foo::Fubar|#some-id> and the rest would be taken care of by the tools. Moreover, I'd make the pipe | optional and have it just as `LFoo::Fubar#some-id in this particular case

and recently

don't think there must be any suggestion/defaults on the Rakudoc side as to where module links must point at. Neither implicit, by the standard; nor explicit by document creator. In the latter case there is always an option of inserting the link manually, as a URL, since it's about to be rigidly hardcoded anyway.

The original desire seemed to be that in some documentation an author could say eg

This to-json method is similar to the L<to-json method in JSON::Fast|mod://JSON::Fast#to-json>

The renderer would have default mechanism to find JSON::Fast and point at the documentation.

Personally I don't like the suggestion purely because it leaves too much to the tooling, but I could see it would be useful.

But if such a functionality is wanted, how should a Renderer operate to become compliant?

[vrurg]

The renderer would have default mechanism to find JSON::Fast and point at the documentation.

I have combined the "renderer" and actual person-reader into a single term "reader". The point is that there is language side (the Rakudoc standard), the writer side (module/document author), and the "reader" side. Or, the other way around, "reader" is the one who reads Rakudoc sources.

Not the best definitions, but OK for the purpose of defining where the final link is produced.

Personally I don't like the suggestion purely because it leaves too much to the tooling, but I could see it would be useful.

Dynamic or semi-dynamic things like URLs are better be left off to the tools. As much as we should do is to provide recommendations, but nothing more. Patrick has provided very good examples as to when the final URL can differ from our understanding of it. My other point would be that changing a standard is not a particularly fast paced process. Not to mention the need for the tools to update afterwards. But in case of a disastrous disappearance of the site, hardcoded in the standard, we better let tool developers to react as quick as they can.

[finanalyst]

The three terms are needed:

• reader - a human reading documentation about a module or functionality. The reader will want to use a method that is most convenient. There are three current useful methods:
  • HTML rendered pages (such as the Raku documentation suite), but also in pages rendered by Podlite.
  • MarkDown rendered pages, eg. REAME.md
  • An IDE, such as Comma
• developer / document author - a human writing software or describing how to use software, which the reader will want to use. The writer uses Rakudoc, which is neutral about the output format.
• renderer - the software tool that converts Rakudoc instructions into a form useful to the reader. The renderer for the Raku documentation site is now Raku::Pod::Render.

So the suggestion is to include in the standard for the minimum Rakudoc the schema for L<> markup called mod://, in addition to the very common https://| http://. The aim is for mod:// to be followed by the name of a Raku module.

That part seems easy. The question is how to specify what a Rakudoc B should do when implementing that link? There may be multiple renderers (I am already planning a new renderer to work directly on RakuAST because Raku::Pod::Render works using $=pod).

It seems to me that a default behaviour should be indicated in the Rakudoc documentation. This default can be changed in the future.

However, this default can be changed / customised.

I am ABSOLUTELY against hard-coded solutions. I wrote Raku::Pod::Render because Pod::To::HTML hard coded everything. The bits that are now put into Mustache templates were my first PRs to Pod::To::HTML.

What I am trying to say is that even if there is maximum flexibility, there should be some reasonable defaults so that the first use of a Render can be as easy as possible for a new user.

[ugexe]

Something to consider when trying to create a URI scheme for this is that users should be able to link to a specific version like mod://JSON::Fast:ver<42>#to-json. The < and > would make it annoying to embed in L<>.

[CIAvash]

That has a solution, one can use L<<>> or L«», unless you're referring to something else.

[finanalyst]

@ugexe the Rakudo grammar is actually good at handling this sort of bracing. For instance C< L<> > passes L<> to the handler for C.

[CIAvash]

As long as the delimiters are balanced it'll work, unless the <> part is in the URL section(I don't know if that's a bug):

=begin pod
L<mod://JSON::Fast:ver<42>#to-json>
# Pod::FormattingCode.new(type => "L", meta => [], config => {}, contents => ["mod://JSON::Fast:ver<42>#to-json"])

L<test|mod://JSON::Fast:ver<42>#to-json>
# Pod::FormattingCode.new(type => "L", meta => ["mod://JSON::Fast:ver<42"], config => {}, contents => ["test"])

L<<test|mod://JSON::Fast:ver<42>#to-json>>
# Pod::FormattingCode.new(type => "L", meta => ["mod://JSON::Fast:ver<42>#to-json"], config => {}, contents => ["test"])

L«test|mod://JSON::Fast:ver<42>#to-json»
# Pod::FormattingCode.new(type => "L", meta => ["mod://JSON::Fast:ver<42>#to-json"], config => {}, contents => ["test"])
=end pod

[finanalyst]

@CIAvash Good catch. I would say the second case is a bug. Have you tried this with RakuAST?

[CIAvash]

@finanalyst no, not sure how to do that. I'm still on Rakudo 2023.04. That doesn't support RakuDoc, does it?

[finanalyst]

@CIAvash RakuAST for RakuDoc is very bleeding edge. Using your tests in a file called ttt.rakudoc, I get.

$ RAKUDO_RAKUAST=1 raku -e "'ttt.rakudoc'.IO.slurp.AST.say"
RakuAST::StatementList.new(
  RakuAST::Doc::Block.new(
    type       => "pod",
    paragraphs => (
      RakuAST::Doc::Paragraph.new(
        RakuAST::Doc::Markup.new(
          letter => "L",
          opener => "<",
          closer => ">",
          atoms  => (
            "mod://JSON::Fast:ver",
            RakuAST::Doc::Markup.new(
              letter => "",
              opener => "<",
              closer => ">",
              atoms  => (
                "42",
              )
            ),
            "#to-json",
          )
        ),
        "\n\n"
      ),
      RakuAST::Doc::Paragraph.new(
        RakuAST::Doc::Markup.new(
          letter => "L",
          opener => "<",
          closer => ">",
          atoms  => (
            "test|mod://JSON::Fast:ver",
            RakuAST::Doc::Markup.new(
              letter => "",
              opener => "<",
              closer => ">",
              atoms  => (
                "42",
              )
            ),
            "#to-json",
          )
        ),
        "\n\n"
      ),
      RakuAST::Doc::Paragraph.new(
        RakuAST::Doc::Markup.new(
          letter => "L",
          opener => "<<",
          closer => ">>",
          atoms  => (
            "test",
          ),
          meta   => (
            "mod://JSON::Fast:ver<42>#to-json",
          )
        ),
        "\n\n"
      ),
      RakuAST::Doc::Paragraph.new(
        RakuAST::Doc::Markup.new(
          letter => "L",
          opener => "«",
          closer => "»",
          atoms  => (
            "test",
          ),
          meta   => (
            "mod://JSON::Fast:ver<42>#to-json",
          )
        ),
        "\n\n"
      ),
    )
  )
)

So @ugexe was correct: the inner <> causes problems.

The interesting thing is that both of the first two cases in your example are wrong, the first one subtly. Rakudo is parsing the <> after the ver as a FormatCode with letter ''.

Using << >> as the bracketing pair yields the correct form (the url should be in the meta attribute.

[CIAvash]

@finanalyst hmm, don't know if that's a bug or not, but using multiple angle brackets or «» is safer for sure.

[lizmat]

So @ugexe was correct: the inner <> causes problems.

It indeed causes issues in the legacy $=pod generation. Am looking into this.

The interesting thing is that both of the first two cases in your example are wrong, the first one subtly. Rakudo is parsing the <> after the ver as a FormatCode with letter ''.

The rules according to S26 are that if you use the same delimiter inside a markup as the "outer" markup, they must balance. The easiest way to implement this, was to handle the as "letterless" markup codes. A "letterless" markup code can always just be stringified. One could argue the parser should do that for you, but I'm a little in two minds about that, as it would remove potentially valuable information.

Using << >> as the bracketing pair yields the correct form (the url should be in the meta attribute.

Different delimiters mean that you can have unbalanced as well, as they are completely ignored in markup parsing.

[finanalyst]

A "letterless" markup code can always just be stringified. One could argue the parser should do that for you, but I'm a little in two minds about that, as it would remove potentially valuable information.

If the parser doesn't stringify, then a RakuAST-oriented renderer needs to look for letterless MC. Then this needs to be documented. It could be useful if a renderer is looking to do something with :var<>, :auth<> or :api<>. That seems a long way ahead.

Different delimiters mean that you can have unbalanced as well, as they are completely ignored in markup parsing.

But that would allow unbalanced delimiters. A note in documentation would indicate that if different delimiters are used, the parser will not pick up an imbalance.

[lizmat]

But that would allow unbalanced delimiters. A note in documentation would indicate that if different delimiters are used, the parser will not pick up an imbalance.

Sometimes you need to allow for unbalanced delimiters. C« < » comes to mind.

[lizmat]

https://github.com/rakudo/rakudo/commit/e66bcd670c fixes the L<test|mod://JSON::Fast:ver<42>#to-json> case.

RakuAST::Doc::Markup.new(
  letter => "L",
  opener => "<",
  closer => ">",
  atoms  => (
    "test",
  ),
  meta   => (
    "mod://JSON::Fast:ver<42>#to-json",
  )
)

[ugexe]

Some other things to consider that crossed my mind:

Is this mod:// string is meant to be an actual URI? If so I don't think we could use :, <, or > (among others) in the host portion of it. The file URI scheme solves a similar problem (that / can be contained in the first part -- i.e. the host) by starting the scheme with a triple slash ala file:/// (thus e.g. JSON::Fast would end up in the path portion of the URI instead of host).

Another other thing is that : < > would typically be URI encoded. This might not be a problem as most tooling can probably handle that conversion, but I do wonder if having the URI not parse-able by URI grammars in their initial form will lead to problems down the road. I'm not really sure there is a good solution for this, but it is worth being aware of.

Third, while it doesn't map directly to raku syntax, it might be better to pass filters like we normally would with a URI, ala mod:///Foo::Bar?version=1&api=2&auth="blorg". This also happens to solve some parts of the previous problem since we wouldn't be passing in < and >.

Fourth, should we use a different prefix like rakumod://? That might make it easier to have something that would show up on e.g. https://en.wikipedia.org/wiki/List_of_URI_schemes since it wouldn't be using a super generic scheme mod

Finally, should this URI map to a distribution + module? For instance a distribution called Net::HTTP which provides a module Net::HTTP::Get: rakumod:///Net::HTTP/Net::HTTP::GET?version=1&auth="github:ugexe". I.e. first path portion would be the distribution, the second path portion (which could be optional) would be something contained in that distribution.

[CIAvash]

It shouldn't be an actual URI, should it? Language implementation should create the RakuDoc data and then RakuDoc renderer should create the suitable link/whatever IMO.

The old design docs mention L<doc:Module::Name>: https://design.raku.org/S26.html#Links

[finanalyst]

To respond to @ugexe , let me start with some initial ideas.

The idea behind the mod: schema is to make it easier for a documentation writer to refer to a module without needing to know where the module is located.

A link has two parts L< 'the text shown in the documentation' | 'the meta data containing the link>`. So when I say 'meta data', I mean the second bit.

The metadata can already contain three schemas: http://, https://, file://

The original suggestion was to have quite a few more, but the one that seems to have usefulness is toc:, to which we are adding index:.

The new schema requested was mod:, but :rakumod seems a better idea since I'm sure someone will want to reference a module in another language, which would increase complexity without much benefit.

Since the idea is to make it easy to write and (read the raw Rakudoc), my feeling is that the rakumod: schema should be as close to the Raku standard idiom as possible.

The question about the URI mapping is something I would call the 'semantics' of the schema (I'm probably technically wrong here). I would think that when a renderer creates the rendered version, for example an HTML <a> tab, it should create something like Foo::Bar?version=1&api=2&auth="blorg" from L<< text | mod://Foo::Bar:ver<1>:api<2>:auth<blorg> >>.

As to distribution / module parts, I think there is a question to be resolved. Where doe s the link to point at? Suppose we have a distribution ExampleModule structured as

ExampleModule/
  docs/
     README.rakudoc
     Moredetail.rakudoc
  lib/
     ExampleModule.rakumod
  README.md

Now we have L<looking at the Example module|rakumod://ExampleModule>, where does it reference? And an HTML URL is going to expect an HTML page to be served back.

This indicates that a Rakumod: schema imposes an expectation on the site that is serving the link.

[niner]

Another point for rakumod: is that it's more self-documenting. It's crystal clear that the URI is identifying a Raku module.

Since we want to use a URI to identify these modules, we must adhere to the URI syntax. Everything else would just put an unneeded burden on tool developers. As such a tool developer, I'd expect to be able to pass the actual link to a standard URI parser to get at the individual components (most of all the scheme to determine, how to link). The URI part after the initial // is the authority which is [userinfo "@"] host [":" port]. As we don't have that kind of information in rakumod links, it doesn't make sense to include the //. Thus a sensible URI would look like rakumod:Foo::Bar

As to the question of how to encode adverbs, I would suggest that we stick to URI syntax as @ugexe suggested: rakumod:Foo::Bar?version=1&api=2&auth="blorg". The reasons are:

• Since tools will already use a URI parser for processing these links, getting the adverbs will be very straight forward, without having to parse the path part yet again.
• The :version<1> syntax might be quite common, but it's by no means standard and by far not the only one. A benign alternative would be :version(1), but syntactically it could also be :version(1 + 2) or really arbitrary code. I'm pretty sure we do not want to go down that route.
• If you argue, that the above is nonsense and we'd of course not allow full Raku syntax in these adverbs, then you'll have to answer how to e.g. quote > characters contained in these fields (like auth super>>fancy>>company). The closer you stick to standard URI syntax, the more obvious the answers are.
• Even with :version<1> you'll end up with the mentioned > >> at the end of the link, which only makes it harder to read.

Just a side node: technically, we don't want those links to be URIs, but IRIs instead. After all, we'll permit pretty much all of Unicode there as usual.

[CIAvash]

Maybe it's because I haven't been involved, but I still don't understand why the tools need to parse the L<> link. Isn't it the job of Raku to provide the necessary and parsed data?

For the dist/module linking, I think it needs to be standardized, so tools know what doc to show as the main document and in what order.

[lizmat]

FWIW, in RakuAST, anything after the | in L<> will appear in the meta attribute. That it didn't so far in some cases, was a bug and that has been fixed.

[CIAvash]

Just to be clear, I mean maybe Raku should do more than extracting the link, also parse it and provide an object representing the dist/module, since Raku already supports that syntax.

[finanalyst]

@CIAvash Here's my take on your comment. (I hope I'm not stating the obvious)

Rakudoc is a markup language that is quite neutral to the way a user will view the documentation. ('quite neutral' although it leans heavily to HTML).

Rakudoc is eventually rendered to an output that the reader will view. So a Rakudoc some text L<pointer|https://docs.raku.org/language/pod> will be output into HTML as

some text <a href="http://docs.raku.org/language/pod>pointer</a>

and into Markdown as

some text [pointer](http://docs.raku.org/language/pod)

and eventually into epub in an xhml version.

Currently parses the Rakudoc into a $=pod tree as

[Pod::Block::Named.new(name => "pod", config => {}, contents => [Pod::Block::Para.new(config => {}, contents => ["some text ", Pod::FormattingCode.new(type => "L", meta => ["http://docs.raku.org/language/pod"], config => {}, contents => ["pointer"])])])]

It is the renderer that takes the pod block and turns it into html or markdown.

In the future, it won't be a POD block, but an AST tree that is used by the Render. Here is the AST tree for the same file.

RakuAST::StatementList.new(
  RakuAST::Doc::Block.new(
    type       => "pod",
    paragraphs => (
      RakuAST::Doc::Paragraph.new(
        "some text ",
        RakuAST::Doc::Markup.new(
          letter => "L",
          opener => "«",
          closer => "»",
          atoms  => (
            "pointer",
          ),
          meta   => (
            "http://docs.raku.org/language/pod",
          )
        ),
        "\n\n"
      ),
    )
  ),
)

The current conversation is exactly about standardising the distribution / module order and what is intended by it.

[CIAvash]

Yes, what I'm suggesting is that if Raku sees L<rakumod:Some::Dist::Module:ver<1.0.0>> , it should generate more data in RakuDoc/RakuAST(if there won't be a pod block). So as an example:

RakuAST::StatementList.new(
  RakuAST::Doc::Block.new(
    type       => "pod",
    paragraphs => (
      RakuAST::Doc::Paragraph.new(
        "some text ",
        RakuAST::Doc::Markup.new(
          letter => "L",
          type   => "module", # or something like that
          opener => "«",
          closer => "»",
          atoms  => (
            "Some::Dist::Module",
          ),
          meta   => (
               RakuAST::Doc::Module.new(
                   dist    => "Some::Dist",
                   module  => "Module",
                   version => "1.0.0",
               )
          )
        ),
        "\n\n"
      ),
    )
  ),
)

Again, just an example of what I mean.

And maybe even something like L<doc:language/pod> can be used to refer to official docs. (if their location is stable).

And normal links continue to work as usual.

[lizmat]

Re: RakuAST::Doc::Module: it feels to me that this should be a more generally useful object, e.g. as a result of use Some::Dist:ver<1.0.0>. So the name should probably not include the ::Doc part.

[CIAvash]

Yeah, it was just for sake of example, I didn't want to emphasize on the naming.

[finanalyst]

@lizmat if I understand what you said, then use Some::Distribution and L<xx|rakumod://Some::Distribution> could be more intimately linked. May be even get the data from the local installation???

Also, for readability I prefer rakumod:// to rakumod:. It is analogous to file:// and http://. I accept that the // serve a purpose in file: etc, but I have never seen http: without a http://

[niner]

That's because it doesn't make sense to use the HTTP protocol with something that isn't an HTTP server that's running on some host. So you will always find authority parts in HTTP URIs. You don't find them in mailto URIs like mailto:foo@bar.org or data URIs like data:just%20a%20string. You want it to be a URI? Then adhere to URI's syntax. Otherwise it is not a URI. If it just sorta, kinda looks like a URI, but really isn't, then it's just confusing.

If the Raku parser is responsible for parsing that URI and turning it into some more capable object, then what is it supposed to do with e.g. https or file URIs?

[CIAvash]

That's because it doesn't make sense to use the HTTP protocol with something that isn't an HTTP server that's running on some host. So you will always find authority parts in HTTP URIs. You don't find them in mailto URIs like mailto:foo@bar.org or data URIs like data:just%20a%20string. You want it to be a URI? Then adhere to URI's syntax. Otherwise it is not a URI. If it just sorta, kinda looks like a URI, but really isn't, then it's just confusing.

I agree with the first part. I think it shouldn't be rakumod://, but I personally don't find rakumod: confusing.

If the Raku parser is responsible for parsing that URI and turning it into some more capable object, then what is it supposed to do with e.g. https or file URIs?

As I mentioned in my comment, they won't be treated specially, the URL will be what is inside L<>.

I think Raku should parse the L<> link and create objects only when it's special. e.g. rakumod:, rakudoc: or maybe even man:(for manuals). And if that's not okay, maybe we should use a new code formatter for linking to documentations.

But if others disagree with this approach, then what makes the most sense is using the URI scheme. Examples for that were given by @niner and @ugexe. another example I would add is the Magnet URI scheme.

[CIAvash]

One way of avoiding URI scheme and creating a new code formatter can be using C<> inside L<> to make it clear that we are linking to some code, but it may get verbose:

L<C<mod:Some::Dist:ver<1.0.0>>>
L<C<mod:Some::Dist:ver<1.0.0>::Module::some_function()>>
L<C<mod:Some::Dist:ver<1.0.0>::Class.some_method()>>
L<C<mod:Some::Dist:ver<1.0.0>::Class.some_attribute>>

[niner]

That's one way, but why add this complication? Links contain labels and URIs. That's easy to understand and will certainly match expectations set up by all other important formats like HTML and Markdown. URIs are the perfect way to express such links as after all they are Universal Resource Identifiers. They really can identify anything. Adding a special rakumod scheme is exactly how our use-case is meant to be handled by URIs. All that's really unclear is what syntax to use for those adverbs.

[CIAvash]

For documentation ordering, I can only speak from my experience and the way I do it; I mostly put documentation inside the code, so based on that my suggestion would be:

1. Look at the docs directory(if that's the new place to put the docs) and use the corresponding doc
2. Use the README.rakudoc in docs directory(if the link points to the dist)
3. Use doc from the Raku files(.raku, .rakumod)
4. Use the README.md (or other readme files)

One problem with this order, might be that developer used some documentation in the code, but the full documentation is in the README.md, …

One way to solve this would be leaving it to the programmer to specify it, using the =config block.

[CIAvash]

@niner I don't know, maybe you're right, but that syntax is more familiar to Raku programmers.

Using URI scheme:

L<rakumod:Some::Dist?version="1.0.0">
L<rakumod:Some::Dist?version="1.0.0"&module=Module&function="some_function">
L<rakumod:Some::Dist?version="1.0.0"&class=Class&method="some_method">
L<rakumod:Some::Dist?version="1.0.0"&class=Class&attribute=some_attribute>

[vrurg]

Having read through the thread, I'm somewhat baffled with a couple of things.

First of all, I like the URI approach in general. BTW, speaking of rakumod: vs. rakumod:// – this can be left out for a document writer to decide, which one is preferred for them. That's not a big deal, in my view.

The use of standard URI arguments also appeals to me. But there is a fly in the ointment: are we going to support :ver<1.0+>? If so then I'd hate having to remember to write it as version="1.0%2B" since even right now I'm not sure if the code is correct, even though I was looking for it only a couple of minutes ago.

The other question is about the interpretation of the "host" part of URI. In L<rakumod:Foo::Bar> – is it about a distribution, or is it about a module? rakudist:Foo::Bar is one solution, but multiplying schemes isn't the best of the best. I would propose the following order:

1. rakumod:Foo::Bar points to a module. If it is not part of the current distribution then we expect it to be possible to locate the distro in an index of some kind.
2. rakumod:Foo::Bar?in="Foo" to use distribution "Foo".
3. rakumod:Foo?module="Foo::Bar" is the same, but it may have different meaning in a very specific case.
4. rakumod:Foo::Bar?dist is for cases when a distribution a module of the same name and we want to be explicit about linking to the distro. Useful for, say, pointing out to a download page. 'dist' could be aliased 'distribution' for purists sake. ;)

Speaking of the difference between 2 and 3, it could play a role for a rare but not impossible case when a module has versions different from those on the distribution. Since we do not prohibit such discrepancies then they are allowed. In this case rakumod:Foo::Bar?in="Foo"&ver="1.0" would be about Foo::Bar:ver<1.0> in the distribution Foo. Apparently, rakumod:Foo?module="Foo::Bar"&ver=1.2" is a link to distribution Foo:ver<1.2>.

Speaking of the advantages of using URI-complient meta, the arguments not used to select module/distribution version can be bypassed to the final link. If translated into a URL the arguments would be submitted to a web server.

[ugexe]

are we going to support :ver<1.0+>?

I wouldn't think so. There isn't really anything it could link to that makes a lot of sense.

rakumod:Foo::Bar?in="Foo" to use distribution "Foo".

The only issue here is that the distribution always exists, but modules may not. For instance: a distribution that provides all of its functionality / code inside of a bin/myapp file.

[vrurg]

are we going to support :ver<1.0+>?

I wouldn't think so. There isn't really anything it could link to that makes a lot of sense.

Not exactly. Say, I have a dependency with the same version mask and would like to link to the same version. But considering what may go wrong here, I wouldn't mind if this feature remains missing.

rakumod:Foo::Bar?in="Foo" to use distribution "Foo".

The only issue here is that the distribution always exists, but modules may not. For instance: a distribution that provides all of its functionality / code inside of a bin/myapp file.

Are links pointing to nowhere is a big surprise to you? ;) I tend to check as many links in my docs as possible. In the future we may get a linter for Rakudoc too.

Either way, we talk about documentation, not modules themselves. rakumod:Foo::Bar?in="Foo" could eventually lead me to doc/Foo/Bar.rakudoc, or to a render of it.

[finanalyst]

I am unclear what to think about the complex syntax for distributions and modules. I think that the links are not to modules themselves, but to documentation about the distribution / modules. Locations within a document are not necessarily the names of classes/roles/sub etc, but are anchors defined by =head or X<> markup. A heading might be eg a sub name, but that is a writers choice.

If a link is expected to resolve to an HTML page, then the actual page served will depend on the server site.

@ugexe are you against the + in the ver<2.1+> ? If documentation changes between versions, I can see the need for ver<2.1>, but not for +.

[vrurg]

I think that the links are not to modules themselves, but to documentation about the distribution / modules.

You're right. But to resolve a link to a module we'd need to locate its distribution. This could be especially relevant for local offline documentation browsers when they'd need to search through local caches.

The point about module version vs. distribution version could be especially confusing. :api versioning could serve even better example to get the point. Imagine a distribution Foo where Foo::Bar module provides very conservative API which version could lag behind distro's API. Say, there is something like "api": "1.5", in META6, but module Foo::Bar:api<1.1>;. In this case, having use Foo::Bar:api<1.1>; in one's sources would naturally translate into L<rakumod:Foo::Bar?api="1.1"> in the docs. The developer wouldn't care in which Foo distribution version the destination page is found as long as it documents the requested API version.

[ugexe]

@ugexe are you against the + in the ver<2.1+> ?

There is nothing inherently wrong with including it, but I do think it will increase the complexity quite a bit since it now implies tooling needs methods of resolving these types of queries in the same way raku would (i.e. use the same sort order). I suppose if such a link did exist I would expect it to go to the latest distribution that is greater than 2.1 (similar to requesting a document without a version at all probably implies the latest version)

[finanalyst]

Rather than rakumod: perhaps it should be rakudoc: ? This would signal clearly that the link is to documentation, and not to code.

It also implies that the documentation may be found in a .rakudoc or .rakumod file.

It also means that the link is to sources written in Rakudoc and not just sources written in raku, which is possible because there are non-raku implementations of POD6 and hopefully of Rakudoc.

[vrurg]

Rather than rakumod: perhaps it should be rakudoc: ?

Makes sense to me.

[dontlaugh]

On the topic of super generic urls, there is a so-called "purl" spec that enmurates the kind of features we'd need, I think https://github.com/package-url/purl-spec

[JJ]

On the topic of super generic urls, there is a so-called "purl" spec that enmurates the kind of features we'd need, I think https://github.com/package-url/purl-spec

Seems to be mainly for packages? Taking into account it uses the constant pkg as scheme.