spring-projects / spring-framework

Spring Framework
https://spring.io/projects/spring-framework
Apache License 2.0
56.77k stars 38.16k forks source link

Move to pure Asciidoctor for documentation [SPR-14997] #19564

Closed spring-projects-issues closed 7 years ago

spring-projects-issues commented 7 years ago

Brian Clozel opened SPR-14997 and commented

This should in fact:

Also, the documentation could use the dedicated asciidoctor plugin to switch from java to xml config in the HTML backend.


Reference URL: https://github.com/spring-projects/spring-boot/issues/7447

Issue Links:

spring-projects-issues commented 7 years ago

Dan Allen commented

I'd like to offer some insight that may help with this transition / migration.

As the Asciidoctor project lead, it's my vision to see users eventually migrate away from DocBook. While I consider it important to have the option to export to DocBook and reuse that toolchain, I don't consider this to be the most efficient or friendly route. Instead, I'm aiming for direct conversion from AsciiDoc to the target output (HTML, multi-page HTML, PDF, etc). I'm confident it can be achieved. Of note, the latest Asciidoctor PDF gets us much closer to parity with the DocBook toolchain...and progress continues.

Multi-page support has traditionally been handled by the DocBook toolchain, a philosophy inherited from AsciiDoc Python. Obviously, it's not ideal to rely on DocBook and its convoluted configuration in order to produce this format. And there's absolutely no reason to. Asciidoctor parses the document into an AST. Any engineer with a modest amount of talent and sufficient time can generate from this model exactly what DocBook produces, if not better. In fact, better is the goal.

The main issue I have with multi-page support is that it's often done with absolutely no thought. Many sites force users to step through the docs paragraph-by-paragraph because the tool mindlessly minced the original document.

What constitutes a page? This is not something a tool can decide for you. It's an information architecture problem. The author or information architect needs to define the parameters of a page and the tool needs to carry out those directives.

We could have a convention where splits occur at annotated section nodes (in AsciiDoc terminology, a role or dedicated attribute). This decision has to be made first before a good tool can be created. Again, any engineer should be able to make the tool carve up the document according to these hints, or by falling back to an intelligent convention.

There's a similar consideration for the ToC and site navigation. What belongs in those catalogs? Where does the information originate? You should design an intuitive hierarchy that's easily navigable. The user doesn't necessarily want a list of every section title. They may even want to look at the content from different perspectives, known as topic-based entries.

Once you start dividing up the document, you really need to think about linking. I actually don't think the built-in linking mechanism in AsciiDoc is sufficient for an enterprise site (not even inter-document xrefs). Those are just handy defaults. The tools are there to create a more sophisticated and more loosely-coupled mechanism.

I recommend introducing an inline macro named something like xlink that defines the parameters of a link, but doesn't concern itself with how to make it. The converter can then take that information, consider what it's making, and produce the appropriate link for the context. (It's also possible to piggyback off the default link so that previews still work, but use the additional metadata when compiling and chunking the documentation for publication).

You may also want automatic links to navigate around the hierarchy or to related topics. The hierarchy links are easy to calculate at build time. Related topics can be achieved by adding metadata (custom attributes) to the document header. If you decide on a standard set of attributes, the converter can make use of these to generate links to other sections, topics, or document roots.

The best way to support AsciidoctorJ, the Maven plugin, and the Gradle plugin is to write the code in Ruby. The Ruby API gives you access to everything, whereas the AsciidoctorJ API is still catching up. If Ruby isn't an option and you must write the extension in Java, then you definitely should be using the alpha for AsciidoctorJ 1.6.0. The AST has been radically improved in that version and gets you much closer to what the Ruby version provides.

In general, my advice is to think outside the box and consider leveraging extensions. AsciiDoc provides both a starting point and an extensible framework for composing and converting documents that fit your needs. If you find yourself having to be too specific or verbose in the AsciiDoc, that's usually a sign you should introduce an abstraction (i.e., syntax extension or declarative metadata).

spring-projects-issues commented 7 years ago

Brian Clozel commented

I've separated the current reference documentation in several docs and removed docbook from the toolchain. The multi-page support in asciidoctor can still be vary beneficial, but splitting the documentation as a first step will improve the documentation quality and make it easier to refactor it.

spring-projects-issues commented 7 years ago

Dan Allen commented

(y)

spring-projects-issues commented 7 years ago

Sam Brannen commented

Reopening this issue due to the following issues.

  1. Reference manual artifacts are no longer published to http://docs.spring.io/spring/docs/5.0.0.BUILD-SNAPSHOT/spring-framework-reference/
  2. Several warnings are emitted while executing gradle asciidoc:
asciidoctor: WARNING: could not embed image: ~/spring-framework/src/docs/asciidoc/images/singleton.png; PNG uses unsupported interlace method
asciidoctor: WARNING: could not embed image: ~/spring-framework/src/docs/asciidoc/images/prototype.png; PNG uses unsupported interlace method
asciidoctor: WARNING: could not embed image: ~/spring-framework/src/docs/asciidoc/images/aop-proxy-plain-pojo-call.png; PNG uses unsupported interlace method
asciidoctor: WARNING: could not embed image: ~/spring-framework/src/docs/asciidoc/images/aop-proxy-call.png; PNG uses unsupported interlace method
asciidoctor: WARNING: GIF image format not supported. Install the prawn-gmagick gem or convert images/DataAccessException.gif to PNG.
asciidoctor: WARNING: could not embed image: ~/spring-framework/src/docs/asciidoc/images/mvc.png; PNG uses unsupported interlace method
  1. The link to https://github.com/spring-projects/spring-framework/graphs/contributors on the first page in the generated HTML is not a clickable link.
  2. The link to https://github.com/spring-projects/spring-framework/graphs/contributors on the first page in the generated PDF is broken.
  3. No links to sections in other generated PDFs work.
  4. Elements that should be styled as bold within code blocks show up as <strong>@Autowired</strong> within the generated PDFs.
spring-projects-issues commented 7 years ago

Dan Allen commented

I want to be clear that my (y) was to the intent. I hadn't yet checked the result.

asciidoctor: WARNING: could not embed image: ...

Asciidoctor PDF does not support the GIF format by default and, apparently, interlaced PNGs (which is news to me).

Although prawn-gmagick would solve these issues (and make image processing a lot faster), it's not available for use with AsciidoctorJ PDF (since it uses native bindings). In theory, however, it's possible to get AsciidoctorJ PDF to use Java APIs to process images. That would just need to be implemented (similar to https://github.com/packetmonkey/prawn-gmagick/blob/master/lib/prawn/gmagick.rb).

The link to https://github.com/spring-projects/spring-framework/graphs/contributors on the first page in the generated HTML is not a clickable link.

The author line is not well-formed. The right way to do it would be:

The_Spring_Framework_team; contributors <https://github.com/spring-projects/spring-framework/graphs/contributors>

However, author names aren't yet shown as links in PDF. (See https://github.com/asciidoctor/asciidoctor-pdf/issues/800). Therefore, a suitable and valid workaround is:

:author: pass:m[The Spring Framework team and https://github.com/spring-projects/spring-framework/graphs/contributors[contributors]]

No links to sections in other generated PDFs work.

Once you split into multiple documents, you must use the inter-document xref syntax to produce links between those documents. See http://asciidoctor.org/docs/user-manual/#inter-document-cross-references.

Elements that should be styled as bold within code blocks show up as <strong>@Autowired</strong> within the generated PDFs.

Asciidoctor PDF does not support both formatted text and syntax highlighting in the same listing block. Due to how syntax highlighting is done for PDF (builds an AST), I haven't yet figured out how to add supplemental formatted text. To be honest, this is not a good practice anyway. The syntax highlighter is not seeing pure code, so the results are going to be mixed. It's better to use a feature such as line highlights or emphasis which annotate the listing instead of adding additional symbols to the code.

spring-projects-issues commented 7 years ago

Dan Allen commented

To solve the images issue, I just recommend saving the images so they are compatible. I was able to fix them using the following:

convert images/singleton.png images/singleton-fixed.png

and

convert images/DataAccessException.gif images/DataAccessException-fixed.png
spring-projects-issues commented 7 years ago

Sam Brannen commented

Dan Allen,

Wow. Thanks for all the tips!

We'll tackle those soon.

As for the bold sections within code listings, I understand what you're saying. That's basically legacy from our DocBook days. Using numbered callouts -- is that what you mean by "line highlights"? -- would definitely be more professional... though a bit more work at the same time. ;-)

spring-projects-issues commented 7 years ago

Dan Allen commented

You're welcome!

Yes, numbered callouts is certainly one option. Another option, which is newer, is to highlight individual lines.

[source,java,highlight=2]
----
@ContextConfiguration
@WebAppConfiguration
public class WebAppTests {
    // class body...
}
----

It's not yet working in Asciidoctor PDF, but it's on the roadmap to get working. See https://github.com/asciidoctor/asciidoctor-pdf/issues/681.

spring-projects-issues commented 7 years ago

Sam Brannen commented

Yes indeed, line highlighting sounds like a nice alternative. I wasn't aware of that feature.

And... was that mere coincidence that you picked a snippet that I authored? ;-)

It's not yet working in Asciidoctor PDF, but it's on the roadmap to get working. See https://github.com/asciidoctor/asciidoctor-pdf/issues/681.

That's not so tragic. As long as there are no adverse side-effects when generating the PDF, I think that would be totally fine: we'd just have the special highlighting in the HTML version for now.

spring-projects-issues commented 7 years ago

Sam Brannen commented

Dan Allen, your convert tip worked like a charm:

https://github.com/spring-projects/spring-framework/commit/454e61e8ac94f08bf879103382d9c422c2df162f

spring-projects-issues commented 7 years ago

Dan Allen commented

was that mere coincidence that you picked a snippet that I authored?

I have my ways ;)

As long as there are no adverse side-effects when generating the PDF, I think that would be totally fine: we'd just have the special highlighting in the HTML version for now.

And it should be available pretty soon anyway.

When converting to PDF, I recommend highlighting with Rouge (keyword: rouge). It's the present and future of syntax highlighting in Asciidoctor...and it will be integrated into the HTML soon. It's also what GitHub uses on GitHub.com.

spring-projects-issues commented 7 years ago

Sam Brannen commented

Dan Allen,

I tried the following:

:author: pass:m[The Spring Framework team and https://github.com/spring-projects/spring-framework/graphs/contributors[contributors]]

That worked in the generated PDF, and it appeared to work in the generated HTML: the actual link works fine in both. However, in the generated HTML, the following shows up at the very top of the index.html page:

contributors">

Any ideas?

spring-projects-issues commented 7 years ago

Dan Allen commented

The problem is that the value of the authors attribute is added to the \ tag in the \ as is, not sanitized. I'm trying to find a workaround, but so far I'm coming up short. Here's something, but it's not quite what we want.

= Spring Framework Reference Documentation
The_Spring_Framework_team; and <https://github.com/spring-projects/spring-framework/graphs/contributors[contributors]>
ifdef::backend-pdf[]
:author: pass:m[The Spring Framework team and https://github.com/spring-projects/spring-framework/graphs/contributors[contributors]]
endif::[]
spring-projects-issues commented 7 years ago

Dan Allen commented

I'm actually surprised that what you had before worked at all. Here's the semantic information we want to capture. Perhaps we can go from there and get it working the way we want.

= Spring Framework Reference Documentation
The_Spring_Framework team; contributors <https://github.com/spring-projects/spring-framework/graphs/contributors>
spring-projects-issues commented 7 years ago

Dan Allen commented

That breaks down to:

author_1; author_2

Where "author" is:

name <URL>
spring-projects-issues commented 7 years ago

Sam Brannen commented

Here's the semantic information we want to capture.

Agreed.

Perhaps we can go from there and get it working the way we want.

Well, we are open to suggestions! ;-)

If this is a bug, we can open a ticket against Asciidoctor.

Just let us know your thoughts.

spring-projects-issues commented 7 years ago

Sam Brannen commented

Brian Clozel,

What are your thoughts on producing multi-page HTML and a single PDF?

Is that already planned?

I ask, because multiple PDFs doesn't seem to make sense in terms of a PDF download and linking between the PDFs.

spring-projects-issues commented 7 years ago

Sam Brannen commented

I noticed the following goal in this issue's description:

not generate the multi-page variant (only the single page is linked from the project page)

Was it just an oversight to resolve the issue as is (i.e., producing multi-page output)? Or will that be addressed in a follow-up issue?

spring-projects-issues commented 7 years ago

Dan Allen commented

Was it just an oversight to resolve the issue as is (i.e., producing multi-page output)? Or will that be addressed in a follow-up issue?

My understanding was that this was just a first step towards reorganizing the source files to prepare them to be used with a future native multi-page converter. It's unclear whether that will be tracked by this issue or another.

spring-projects-issues commented 7 years ago

Dan Allen commented

If this is a bug (regarding the author information), we can open a ticket against Asciidoctor.

I forgot to mention that I've filed the following issues to track it:

I do encourage you at some point to introduce your own PDF theme (since you'll definitely want to customize things over time) as well as a custom document template for converting to HTML. This would give you a lot more control over the output.

spring-projects-issues commented 7 years ago

Sam Brannen commented

Thanks for the links to those issues. I'm subscribed to both now.

spring-projects-issues commented 7 years ago

Sam Brannen commented

When converting to PDF, I recommend highlighting with Rouge (keyword: rouge). It's the present and future of syntax highlighting in Asciidoctor...and it will be integrated into the HTML soon. It's also what GitHub uses on GitHub.com.

I tried out rouge; looks good.

But what do you mean by "soon"?

The corresponding issue is currently slated for 1.7.0, but 1.6.0 hasn't even been released yet. 🤔

spring-projects-issues commented 7 years ago

Dan Allen commented

But what do you mean by "soon"? The corresponding issue is currently slated for 1.7.0, but 1.6.0 hasn't even been released yet.

The milestone was mislabeled. I'm shooting for it to be part of 1.5.7 (since 1.5.6 is already half out the door).

You can use rouge today with Asciidoctor PDF. Just override the source-highlighter attribute. The benefit that gives you is that it's easy to add a custom theme for Rouge, whereas CodeRay has no themes. See https://github.com/asciidoctor/asciidoctor-pdf/blob/master/docs/theming-guide.adoc#source-highlighting-theme

spring-projects-issues commented 7 years ago

Sam Brannen commented

The milestone was mislabeled. I'm shooting for it to be part of 1.5.7 (since 1.5.6 is already half out the door).

Ahhhh... That's much sooner. Thanks for the clarification.

You can use rouge today with Asciidoctor PDF. Just override the source-highlighter attribute. The benefit that gives you is that it's easy to add a custom theme for Rouge, whereas CodeRay has no themes. See https://github.com/asciidoctor/asciidoctor-pdf/blob/master/docs/theming-guide.adoc#source-highlighting-theme

To start using rouge now, we would need to configure separate gradle tasks for the html5 and pdf backends, which we don't yet do. Or am I missing some trick for telling Asciidoctor to use a different source highlighting theme when generating a PDF vs. HTML?

spring-projects-issues commented 7 years ago

Dan Allen commented

Indeed. Just add the following to the AsciiDoc document header:

:source-highlighter: coderay
ifdef::backend-pdf[:source-highlighter: rouge]

But don't set the source-highlighter in the Gradle configuration. Or set it to:

'source-highlighter': 'coderay@',

The trailing @ makes it overridable.

spring-projects-issues commented 7 years ago

Sam Brannen commented

That's much more elegant than introducing competing Gradle tasks.

Cheers!

spring-projects-issues commented 7 years ago

Brian Clozel commented

I'm closing this issue as its scope expanded out of its original intent: moving our documentation to an asciidoctor-only setup. Feel free to open new issues for the remaining tasks you'd like to complete before RC1.

spring-projects-issues commented 7 years ago

Bian Jiaping commented

Spring Framework 5.0 has be out for one month, but when will the htmlsingle, PDF and epub variants of reference documentation be available? Any roadmap?

spring-projects-issues commented 7 years ago

Brian Clozel commented

See #20395 for an issue about single page variant; the new format is different from the previous multipage format. You could see it as multiple single page docs, each focused on one big part of Framework. In the meantime, PDF versions are already available here: https://docs.spring.io/spring-framework/docs/5.0.0.RELEASE/spring-framework-reference/pdf/