Support vor AsciiDoc cross-document syntax <<>>

[michaelmejaeger]

This plugin is really great. I have only one issue with it: it uses the link:url[text] syntax for references. Although this is fine for external Links, it does not work for cross-references with AsciiDoc documents. The documentation of AsciiDoctor (https://asciidoctor.org/docs/user-manual/#inter-document-cross-references) recommends here using the "<<url#,text>>" syntax.

This commit tries to fix this, it seems to work well ;-).

[ratfactor]

@michaelmejaeger Hello! Thank you for the pull request.

I see three problems:

1. This completely replaces the existing link:foo[bar] syntax, which would be a breaking change and require conversion of all existing user's wikis.
2. It doesn't address the true purpose of the cross reference macro in AsciiDoc - which is to create links between sections within a document.
3. It only works with AsciiDoctor when exporting to other formats!

Problem 2

See https://asciidoc.org/userguide.html#_inline_macros (specifically Section 21.1.2. Internal Cross References):

Two AsciiDoc inline macros are provided for creating hypertext links within an AsciiDoc document. You can use either the standard macro syntax or the (preferred) alternative.

Anchors can be created with either macro syntax: [[foo]] or anchor:foo.

Creating a link to an anchor uses the xref (cross-reference macro) with either <<foo,My Foo>> or xref:foo[My Foo] syntax.

(AsciiDoctor also mentions [#foo] shorthand, but other implementations won't work with this syntax: asciidoc: WARNING: test.adoc: line 6: missing section: [#foo]. Likewise, the "natural" section title links <<My Foo>> are a very nice AsciiDoctor-only addition.)

Problem 3

As you point out, AsciiDoctor also allows "inter-document" cross references with the <<bar#foo>> syntax, but only AsciiDoctor supports this syntax!

The original Python asciidoc implementation will create the link test.html#bar#foo!

So we have a very serious portability problem with this type of link.

Correct AsciiDoc local document links

The AsciiDoc user's guide is very clear that links to other documents are to be done with the link:<target>[<caption>] syntax. (See "21.1.3. Linking to Local Documents".)

Hypertext links to files on the local file system are specified using the link inline macro.

It also mentions:

You can use the <filename>#<id> syntax to refer to an anchor within a target document but this usually only makes sense when targeting HTML documents.

Which is what the AsciiDoctor documentation explains as well. AsciiDoc does not define a standard way to link to anchors within other documents because it's a concept that doesn't make sense most/many other exported document types.

The only universal AsciiDoc method to create inter-document anchor links is to create explicit anchor IDs and use the HTML-style anchor link syntax (for use with HTML output).

Example:

= Test 1

Hello, this is test1.txt.

[foo]
== The Foo Section

Foody foo foo.

= Test 2

This is a link to link:test1#foo[The Foo Section].

I'm going to close this request for now. Please let me know if you believe I have made a mistake!

By the way: it would be great to support the universal AsciiDoc internal cross reference macros. VimWiki has internal anchors, so looking at how those are handled those might be a good starting point. :-)

[michaelmejaeger]

Hi Dave,

many thanks for the detailed answer! Here are my comments to that:

Problem 1

This is true, I did not consider that and this alone is probably a showstopper.

Problem 2

It is absolutely true, that the xref syntax is mainly for internal references. However, the link macro is also not useful for this, since it links to files and not to documents in the sense of AsciiDoc. This means that link:test1#foo[The Foo Section] as given in the example creates a link to the file test1 and not test1.adoc or test1.html if rendered to html. This is probably the reason why Asciidoctor provides another option -- the standard simply does not provide a solution here. I understood that we are on the same page here. My question is then, how to you render your AsciiDoc files? With Asciidoctor and the original asciidoc-Implementation the link-references do not work, since they (correctly) produce a link to the file with the name give (without an extension).

Problem 3

It's true that my proposal does only work with AsciiDoctor and not with the original Python implementation of asciidoc, however, the solution based on link does neither work with AsciiDoctor nor with asciidoc (at least not with version 8.6.10 I have installed).

My company is engaged with the AsciiDoc standard, so I would like to learn more here. Do you think it makes sense to try to improve the options to att inter-document references?

[ratfactor]

Yeah, you're absolutely right!

page1.adoc

<<page2#foo,Foo>>
link:page2#foo[Foo]

Exported with asciidoc (Python):

<a href="#page2#foo">Foo</a>
<a href="page2#foo">Foo</a>

Exported with asciidoctor (Ruby):

<a href="page2.html#foo">Foo</a>
<a href="page2#foo">Foo</a>

Aside

The funny thing is that I now remember that two years ago, I had performed this exact comparison and had forgotten the difference: AsciiDoctor adds the .html extension to the xref macro links in addition to giving special treatment to the # "fragment" separator.

Also two years ago, I side-stepped this entire issue by exporting all of the pages in my website without file extensions. This allows me to have "clean" URLs without any web server request manipulation and the link:foo[Foo] links work out of the box because foo is the page filename!

Link vs XRef

You're absolutely correct that link: doesn't work for links to exported documents with file extensions (.html) because no URI conversion is performed. So how about xref: links?

I thought it might be instructive to go take a look at DocBook, since that's where all of this originates from.

https://tdg.docbook.org/tdg/5.2/xref.html

The xref element forms a cross-reference from the location of the xref to the element to which it points. Unlike link and the other cross-referencing elements, xref is empty. The processing system has to generate appropriate cross-reference text for the reader.

https://tdg.docbook.org/tdg/5.2/link.html

The link is a general-purpose hypertext element. Usually, link surrounds the text that should be made “hot” (unlike xref which must generate the text), but the endterm attribute can be used to copy text from another element.

It goes on to say that link elements can have an linkend or xlink:href attribute, which allow links to act as xrefs (internal or external anchor links) with user-specified link text.

Neither of these elements seems quite right, but...

OLink

Now, where it gets really interesting is the concept of DocBook olink tags:

https://tdg.docbook.org/tdg/5.2/olink.html

Unlike link, the semantics of olink are application specific. The olink element provides a mechanism for establishing links across documents, where ID/IDREF linking is not possible and URI-based linking may be inappropriate.

Which, if I'm reading it correctly, is exactly what we want!

Here's an explanation that confirms:

http://www.sagehill.net/docbookxsl/Olinking.html

When writing technical documentation, it is often necessary to cross reference to other information. When that other information is in the current document, then DocBook provides support with the xref and link elements. But if the information is in another document, you cannot use those elements because their linkend attribute value must point to an id (or xml:id for DocBook 5) attribute value that is in the current document.

The olink element is the equivalent for linking outside the current document. It has an attribute for specifying a document identifier (targetdoc) as well as the id of the target element (targetptr). The combination of those two attributes provides a unique identifier to locate cross references. These attributes on olink are available starting with the DocBook XML DTD version 4.2.

The above resource even goes on to discuss generating relative links for HTML output.

But my excitement has been tempered by finding this:

https://discuss.asciidoctor.org/DocBook-5-olink-support-td2621.html

Asciidoctor introduces a syntax to address cross-document references. We could wire this to olink in the DocBook backend. If necessary, we can enhance the feature if necessary to cover all the requirements.

Which brings us full-circle to your original proposal to use XRefs for all inter-document links and on closer reading of the OLink page in The Definitive Guide (same link as above), this seems to be, in principal, supported by the spec:

Other elements can also behave like olink by using the common linking attributes xlink:href and xlink:role. When an element has an xlink:role="http://docbook.org/xlink/role/olink" attribute, then its xlink:href attribute is interpreted using olink semantics. That is, the part of xlink:href before the fragment identifier (#) is interpreted as equivalent to an olink targetdoc attribute value instead of a URI, and the part after the fragment identifier as an olink targetptr attribute value.

So I have to conclude that you're correct, at least as far as the AsciiDoctor implementation goes. (Though I don't love the overloading of XRef in this way. :broken_heart: )

And then:

My company is engaged with the AsciiDoc standard, so I would like to learn more here. Do you think it makes sense to try to improve the options to att inter-document references?

I'm very interested in this! I think in an ideal world, there would be "official" AsciiDoc support for OLinks via the standard macro syntax. Something like this:

olink:foo[Document Foo]
olink:foo#bar[Bar in Document Foo]

We may need to dig into the DocBook standard a bit further, but it looks like the document name (targetdoc) is required and the anchor (which is probably the targetptr attribute) is optional. :heart_eyes:

Back to Problem 1!

Either way, you're right about how this is handled in VViki. Better inter-document linking syntax is needed. :scream:

It will need to either be:

• "Opt-in" so that existing wikis are not broken
• Change the default and have an option to "opt-out" for backward-compatibility along with a big banner in the README about the breaking change

Thank you for bringing this up and I hope we can come up with a great solution that will make collections of AsciiDoc documents better than ever!

[ratfactor]

Note: searching "inter-document", and to a lesser extent, "olink", returns some good reading in the AsciiDoctor issues:

• https://github.com/asciidoctor/asciidoctor/issues?q=inter-document
• https://github.com/asciidoctor/asciidoctor/issues?q=olink

Suffice it to say that this is something that the AsciiDoctor project has been thinking about and dealing with since at least 2015.

[marmalodak]

In trying to follow this issue, I came across AsciiDoc Language at the Eclipse Foundation. Does this mean there will exist a formal spec to which both asciidoc-py3 and asciidoctor will conform?

Would this be the way forward on issue #4?

[ratfactor]

@marmalodak I also ran across that in searching for some sort of official spec, but I didn't stick around long when I failed to find an actual spec...

What I didn't realize was that the working group is quite new and is currently doing administrative things to get itself into position to start working on the language spec. From my skimming of the mailing list at https://www.eclipse.org/lists/asciidoc-wg/maillist.html and some of the recent meeting minutes (Google Docs linked from list messages), it looks like the WG is working on a budget. So they're not at the language stuff quite yet.

From my reading, this Language Dev mailing list will become active as soon as the Working Group is ready to start the fun stuff: https://www.eclipse.org/lists/asciidoc-lang-dev/index.html

I'm very interested in "official" support for inter-document linking and I've subscribed to the aforementioned mailing lists so I can bring up the issue (and perhaps propose "first-class" support of DocBook olinks) when the time comes.

Thanks for pointing me back at the Eclipse page! I do agree that an official spec would be the ideal way forward with issue #4!

@michaelmejaeger Is the Eclipse AsciiDoc Language project what your company is engaged with? (I've also realized that I didn't tag you in my rather long reply above. Sorry about that.)

[marmalodak]

I came across vviki today, so please pardon the newbie question as I haven't installed yet.

How do I render my vviki as either PDF or HTML? If I do, do vviki links translate to hyperlinks?

And thank you very much for the detailed answers.

[ratfactor]

@marmalodak The short answer is that you can use any popular AsciiDoc tool (and, importantly any DocBook tool (after conversion from AsciiDoc)) to export VViki pages as they are syntactically correct AsciiDoc documents.

HTML Output

External links will work out of the box (a link to http://wikipedia.org will work exactly as expected).

Internal links (as VViki currently uses to link internal wiki pages link:page2[Page Two]), on the other hand, won't work out of the box as they won't include the .html file extension of the exported file. :sob:

What I'm doing for HTML exporting

My own website is rendered with a little Ruby script I hacked together which uses Asciidoctor as a Ruby library so I have complete control over the export process (which also allows me to have custom page templates). So I'm exporting my pages without the .html extension. This works brilliantly when the pages are put on a webserver (example.com/page2 is a valid link) and also when browsing the files locally (at least on Linux) where my file manager and browser happily understand these to be HTML documents without the file extension.

PDF

I've imagined scenarios in which I might want PDF output of my wiki pages, but I haven't actually had the need yet. I was planning to use https://github.com/asciidoctor/asciidoctor-pdf as it seems to be the most popular option. However, I think this is where the DocBook connection might really shine as there is some serious industrial-grade DocBook support for print formats!

Single-page PDF exports are no problem. But what about internal wiki links? Well, linking between PDFs is an avenue I haven't explored. On the other hand, I have always assumed it would be possible to export all or part of a VViki wiki as book sections and/or chapters and exported as a single PDF or ebook document. What's interesting about this, though, is that links would then need to be internal cross-references. Hmmm...

Conclusion

There are a number of hacks to get around the file extension problem in HTML links:

• Export your HTML files to without the .html extension in the filename (e.g. page2 instead of page2.html). Or rename the files to remove it after exporting.
• Write a little pre-processing script that adds '.html' to all relative links before sending them to an AsciiDoc tool.
• Write a post-processing script that adds .html to relative links in the exported HTML. (I like this one least.)
• Use @michaelmejaeger's https://github.com/michaelmejaeger/vviki/tree/patch-1 branch to change VViki to use Asciidoctor's <<page2#Page Two>> cross-reference syntax (currently supported only by Asciidoctor).

The idea of PDF exporting brings up all sorts of additional questions to which I don't seem to have any good answers just yet!

[michaelmejaeger]

Hi all,

sorry for being so quiet these days, I simply had too much to do.

AsciiDoc Standardization

@ratfactor It's true that my company is engaged with the AsciiDoc Working Group at Eclipse Foundation. I asked a colleague who is better informed than me and he confirmed your impression: at the moment most discussions are about budget and formalities. Besides at the time of writing this, it is not possible to engage for topics regarding the AsciiDoc specification. However, I also subscribed the mailinglist in order to be notified when things start to get productive. Until then, we have to wait :-/.

AsciiDoc Rendering

@marmalodak Everything that @ratfactor wrote is correct. I adapted vviki to use the Asciidoctor syntax for cross-referencing. For rendering I'm currently using a simple script that I bound to a key combindation in my .vimrc:

" AsciiDoc preview
nmap <leader>v :!asciidoc-view.sh %<CR><CR>

The shell script asciidoc-view.sh then looks like the following:

#!/bin/bash

USAGE="Usage: $0 <filename>"
FILENAMEASCIIDOC=$1

if [ -z "${FILENAMEASCIIDOC}" ]
then
  echo $USAGE
  exit 1
fi

echo "Converting asciidoc file ${FILENAMEASCIIDOC} to HTML and showing in firefox..."

FILENAMEHTML="${FILENAMEASCIIDOC%adoc}html"

asciidoctor $FILENAMEASCIIDOC && firefox $FILENAMEHTML &

Of course, you can enhance the script to render all pages and not just the current one, but this suffices for my needs. I also don't publish my wiki somewhere, it's just for personal notetaking...

[ratfactor]

@michaelmejaeger Thank you for the info! Waiting for the commitee is hard, not because I'm in a hurry, but because I feel like my little wiki plugin is in an awkward state until we figure this out. I'm looking forward to seeing the first messages arrive in the AsciiDoc Language Dev mailing list.

I like your Bash script. I keep meaning to clean up the Ruby script that publishes my website. Maybe I'll have a burst of energy and free time someday. :hatching_chick:

[marmalodak]

There might be a better way to do this (maybe with autocmd on the file extension?), but this is what I have at the start of a bunch of my asciidoc files:

    ////                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        
    to use this function: yank lines 3 to 9 and then type :@"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 
    function! ConvertAdoc()                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     
        silent !asciidoctor -b html  --attribute docinfo=shared mydoc.text^M^M                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   
        silent !scp mydoc.html mylaptop:/Users/marmalodak/Documents/                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     
        execute("redraw!")                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      
    endfunction                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 
    autocmd! BufWritePost,FileWritePost mydoc.text :call ConvertAdoc()    
    ////

When I start an editing session, I follow those instructions, and then I can keep a browser open and reload fairly smoothly on a side monitor. This gives me the flexibility to customize my conversion per document, for example, if I want a PDF instead of an HTML file.

I wouldn't need the scp if I used a git repo for my docs.

[ratfactor]

@marmalodak I believe you could make this a little more convenient by throwing it into your .vimrc with a couple modifications:

function! ConvertAdoc()
    if !exists('g:autoconvert_adoc')
        return
    endif

    execute("silent !asciidoctor -b html  --attribute docinfo=shared ".expand('%'))
    execute("silent !scp ".expand('%')." mylaptop:/Users/marmalodak/Documents/")
    execute("redraw!")
endfunction

autocmd! BufWritePost,FileWritePost *.text :call ConvertAdoc()

nnoremap <leader>aa :let g:autoconvert_adoc = 1<CR>

(I have not entirely tested this.)

Then you can just type ,aa (or whatever your leader key is) to toggle the Autoconvert Adoc feature on and off.

I like the :@" trick, though. Using buffers like that is so powerful. I always have to look up the syntax when I want to interact with the buffers in Ex commands.

[ratfactor]

@michaelmejaeger If you're still interested, I have added support for the "xref" link syntax as an option. Thank you again for your pull request and bringing the issue to my attention!

From the release 1.1.0 help documentation:

g:vviki_page_link_syntax                              *vviki_page_link_syntax*

Unfortunately, AsciiDoc does not define syntax for linking across documents.
Therefore, VViki currently supports three syntax styles. All three will work
exactly the same within Vim. The differences will be noticed when you attempt
to export your AsciiDoc pages to another format (such as HTML).

To use a non-default link syntax style, set it like so:
>
    let g:vviki_page_link_syntax = 'olink'
<

(Note that VViki will recognize all supported link syntax styles as links and
follow them regardless of setting - all other features (such as syntax
concealing and creating new links) will use the chosen syntax style
exclusively.)

The available syntax options:

        Syntax        | Example           | asciidoc | asciidoctor
    ------------------+-------------------+----------+-------------
     "link" (default) | `link:foo[My Foo]`  | foo      | foo
     "olink"          | `olink:foo[My Foo]` |          |
     "xref_hack"      | `<<foo#,My Foo>>`   | foo      | foo.html

Because the need for the different syntax styles lies in the conversion of
pages to other formats (such as HTML), the above table shows the linked page
filename as exported by each of the two most popular command line tools
`asciidoc` (Python) and `asciidoctor` (Ruby) as they would export each of the
different link syntaxes.

You'll note that only one combination produces relative (wiki-internal) page
filename links with a ".html" file extension when exporting to HTML. (More
about the "olink" results in a moment.)

The author of VViki uses the default "link" syntax and actually prefers the
"extensionless" filenames. When delivered from an Apache web server, they are
recognized correctly as HTML and display correctly in all browsers, giving
so-called "clean URLs".

One could certainly write pre- or post-processing scripts to manipulate the
AsciiDoc source files to add file extensions to links as well.

DocBook (the XML format on which AsciiDoc is based) has a link syntax
specifically addressing the issue of "establishing links across documents."
The "olink" type links to a document by name rather than FILEname. It is the
author's hope that an "olink" macro will someday be recognized by major
AsciiDoc processing applications for this specific purpose. None do at this
time, so the option in VViki is purely aspirational.

As you can see, I also added support for a hopeful future "olink" style!

[ratfactor]

For anyone interested in this continuing saga, I've been following the AsciiDoc Working Group mailing list activity and this AsciiDoc reference documentation has been posted (hosted on asciidoctor.com until the WG gets its own website going for the language). Check out the xref page:

https://docs.asciidoctor.org/asciidoc/latest/macros/inter-document-xref/

Specifically, this example under the heading Navigating between source files:

= Document Title
ifdef::env-github,env-browser[:relfilesuffix: .adoc]

See the xref:README.adoc[README].

This shows two really interesting things:

1. We can use the relfilesuffix attribute (counterpart to outfilesuffix) to set the file extension used for relative links in specific environments. (This is further documented/explained on the linked page.)
2. The new preferred syntax for cross-document links (according to AsciiDoctor) is xref:<document>[description]!

Well, that doesn't sound like the correct usage of xref to me at all ("xref — A cross reference to another PART of the document" (emphasis mine)).

In fact, I made this test repo here on GitHub to try out some various link types in a README.adoc file and got some interesting results. Chiefly, xref:foo.adoc[Foo] does work for relative links on GitHub, but only with the '.adoc' file extension. Otherwise, it attempts to make an HTML anchor link to "#foo" (as you would expect).

Additionally, the <<xref:foo.adoc#,Foo>> works with or without the .adoc extension! (But don't forget the trailing # or it won't work!)

Jeez, what a mess.

Summary:

1. I remain convinced that DocBook olinks are the right way to do relative inter-document linking within a body of documents.
2. GitHub is another important AsciiDoc target/host.
3. xrefs are a total hack, but they basically work. VViki supports them (see previous post).
4. I need to figure out if the existence of relfilesuffix and outfilesuffix AsciiDoc attributes have any bearing on VViki.

[marmalodak]

There is now an empty gitlab repo at the eclipse foundation. Was it created two days ago?

Would you ever be interested in presenting vviki at an eclipse meetup?

(Not sure if this is the appropriate place to bring this up)

[ratfactor]

@marmalodak Interesting! I'm gonna bookmark that repo now. Maybe they're getting closer to actually discussing the spec? The Working Group mailing list has been slow lately, so I'm not sure what the schedule looks like.

I did see the call for Eclipse presentations. I'm not sure I've got enough material to make a coherent one.

What I'm doing right now is watching the Language Dev mailing list like a hungry lizard watches the approaching insect. :lizard: As soon as that thing goes active, I plan to pounce on it and present my argument for olink:foo[Foo]. Caught unawares, they will be helpless in the grip of my persuasive speech and will most likely capitulate out of pure fright. Later they will realize it was the right answer all along.

[ratfactor]

Pounced! :lizard:

https://www.eclipse.org/lists/asciidoc-lang-dev/msg00007.html

Oh for... I decided to re-send with plaintext flowed=false so it doesn't appear as a total mess. (Shakes fist at Thunderbird.) :angry: :rage: :angry:

This one is readable:

https://www.eclipse.org/lists/asciidoc-lang-dev/msg00009.html

[ratfactor]

And now that I've linked from there to there, hello to any mailing list folks who made it this far! :laughing:

https://www.eclipse.org/lists/asciidoc-lang-dev/msg00011.html

Hmmm. I can do better than that.

Unleash the manifesto!

https://www.eclipse.org/lists/asciidoc-lang-dev/msg00015.html

[ratfactor]

This is my final hurrah. The mailing list has convinced me that xref is the only way forward. And I'm completely okay with that. But I'd like to see much greater clarity about how relative document paths are defined, etc.

To that end, I created this:

https://github.com/ratfactor/xref.adoc

And posted this:

https://www.eclipse.org/lists/asciidoc-lang-dev/msg00067.html

I don't have a lot of hope that anything will come out of it, but at least I'll have given it a good shot.

I'll probably update the README soon to make it clear what VViki settings are currently recommended (xref:foo#[Foo] has the best support and does what you'd expect). I'm not going to remove the olink option just yet, but that would probably be a good idea to avoid confusing people.

[michaelmejaeger]

Great, I am following the discussion on the mailing list. I am excited to see the momentum that AsciiDoc gets here! Thanks or your engagement, Dave!

[ratfactor]

@michaelmejaeger Thank you, I appreciate that a lot!