Asciidoc Techdocs support

[Hanyman8]

Hello,

we have our complete domunentation of microservices in adoc format. Would it be possible to add adoc format instead of just markdown using mkdocs.

One of the solution for us would be using our generated html from adoc (we are able to do it in our CI/CD)

Feature Suggestion

Asciidoc support

Possible Implementation

Context

We would like to serve our html generated from our asciidocs

[Hanyman8]

Hello, is there a way to serve our own html in the techdocs? Thanks

[tomb50]

We are also currently looking into adopting Backstage and have existing adoc documentation. We are trying to compare Backstage + referncing Antora vs what could be possible with pure Backstage Techdocs.

Serving the HTML would indeed help us, even as an interim

[snowe2010]

@tomb50 we're in the exact same position. Antora, but others want to move to backstage. Backstage is great, but being dependent on markdown is quite terrible. There are several PRs that have been closed that would have accomplished this, maybe we can take them and add the support ourselves, since it seems like they're in no rush to add it. https://github.com/backstage/backstage/pull/9385

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[Okeanos]

Yeah, so I am still interested in that.

[jarekkar]

Also interested in.

[Sroca3]

Same. Asciidoc is the default for spring-rest-docs and a majority of spring documentation is in asciidoc as far as I'm aware so it's a better option than markdown IMO.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[Okeanos]

Still relevant and waiting for a solution to this.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[ikatlinsky]

Still relevant, highly desired feature

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[Okeanos]

Would still love to see this added.

[pwright]

I also had this requirement, but worked around it by generating html with no stylesheet [1] and an antora ui that only produces body html, then renaming the html files as '.md', mkdocs was fine with the result

[1] https://docs.asciidoctor.org/asciidoctor/latest/html-backend/default-stylesheet/

[remast]

We' d also very much appreciate this.👍

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[ikatlinsky]

Still highly desired

[JanGroot]

Yup, I want it as well

[StephaneRob]

As a POC, I created a s3 proxy plugin to serve static documentation (storybook, ex_doc, ....) directly from backstage and avoid modifying the build to fit with TechDocs system.

[snowe2010]

@pwright @StephaneRob, can you both describe your solutions in a bit more detail to allow others to try and get it working as well? I think it would be highly appreciated.

[snowe2010]

Ok, I tried to do what they mentioned above, this is the solution I came up with.

build.gradle.kts

import org.asciidoctor.gradle.jvm.AsciidoctorTask // had to command click this after syncing to jump to the source and then intellij was able to find it

plugins {
    // ...
    id("org.asciidoctor.jvm.convert") version "4.0.1"
}

tasks.named<AsciidoctorTask>("asciidoctor") {
    outputOptions {
        backends("html5", "docbook")
    }

    sourceDir("docs/modules/ROOT")
    resources(delegateClosureOf<CopySpec> {
        from("docs/modules/ROOT/pages/resources") {
            include("*.png")
            include("**/*.png")
        }

        into("./resources")
    })

    options(mutableMapOf("standalone" to false))
    copyResourcesOnlyIf("html5")
    useIntermediateWorkDir()

    asciidoctorj {
        modules {
            diagram.use()
        }
    }

    doLast {
        copy {
            from("build/docs/asciidoc/html5")
            into("docs/generated-html/")
            rename("""(.*).html""", "$1.md")
        }
    }
}

My original directory structure looked like this

where all my antora docs and backstage component yamls are together.

My mkdocs.yml looks like this

docs/mkdocs.yml

site_name: 'mysite'

nav:
  - Home: index.md
  - Contributing: Contributing.md
  - Quarkus Base Docs: quarkuks-base-docs.md

docs_dir: generated-html

plugins:
  - techdocs-core

where I set the docs_dir to a generated directory.

The standalone option must be set to false for this to work.

    options(mutableMapOf("standalone" to false))

The diagrams setup allows mermaid image generation to work.

If you are generating only a single backend (backends("html5")), then the asciidoc output won't have any nested html5 folder, so you'll need to update the doLast

            from("build/docs/asciidoc")

Result

Footnotes even work properly!

Issues

For some reason, the pages that aren't the main index.md prepend their file name when searching for the resources. Therefore a path like this in the markdown

Contributing.md

<div class="imageblock">
<div class="content">
<img src="resources/image-2024-01-05-11-48-24-747.png" alt="image 2024 01 05 11 48 24 747">
</div>
</div>

results in a path like this in the generated docs in backstage:

<div class="imageblock">
<div class="content">
<img alt="image 2024 01 05 11 48 24 747" src="https://backstage.com/api/techdocs/static/docs/default/component/tax-service/Contributing/resources/image-2024-01-05-11-48-24-747.png">
</div>
</div>

while the same exact style html in the generated index.md docs looks like this:

<div class="imageblock">
<div class="content">
<img alt="image 2023 12 28 14 43 56 266" src="https://backstage.com/api/techdocs/static/docs/default/component/tax-service/resources/image-2023-12-28-14-43-56-266.png">
</div>
</div>

Clearly these resources will never be found. I'm not sure what to do about this. I don't really want to just copy this resources directory around to all the subpaths for no reason. This seems like a backstage bug.

[pwright]

@snowe2010 glad to see it worked out. TLDR; for folks rename("""(.*).html""", "$1.md") allowed Tyler to insert rendered AsciiDoc into mkdocs.

[snowe2010]

@pwright that and the standalone option must be set to false

[StephaneRob]

On my side, I wanted to serve static documentation without modifying them. For this I did the following:

1. Build a front end plugin to add an entity route. This view has a selector to choose the doc to load for a given component and an iframe to load it (backend plugin routes)
2. Build a back end plugin with a catch all route to load html, css, js, font...proxying request to s3 bucket. (with a redis cache)

I needed:

• to have a convention of where i store my static doc
• make sure that assets are loaded with the correct url to proxy request to s3 using the same catch all route than the initial html

More work but can embed any type of static doc inside backstage component.

[AlexSneedMiller]

@StephaneRob are you still able to leverage the search functionality within Backstage in your s3 POC?

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[snowe2010]

Not stale, just waiting...

[Hanyman8]

Thanks to everyone to keep this issue open. We would also still use this feature in our project.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[ddolan-lutron]

Still not stale, still just waiting...

[wanisfahmyDE]

We implemented the following workaround till this is natively supported in Backstage, This worked for almost all asciidoc features that we were previously using aside of some gotchas.

• First we used asciidoctor-reducer to aggregate all ascii docs in one file
• Then we used downdoc to convert the generated index.adoc file to markdown
• Then we used the techdocs-cli to generate the site (htmls/images/sitemap/etc) and publish it to S3

Here is how our GitHub Action looks like:

name: 'publish-techdocs-action'
description: 'A github action that builds and publishes markdown files based docs to Backstage.

inputs:
  component-name:
    description: 'Name of the component in Backstage if it differs from the repo-name.'
    default: ''
    required: false
  entity-kind:
    description: 'Kind of the entity in Backstage, can be Component, API, Domain, System, etc.'
    default: 'Component'
    required: false
  verbose-logging:
    description: 'Whether verbose logging for generating docs is enabled or not.'
    default: 'false'
    required: false
  doc-source:
    description: "Format of doc-source. Currently only 'markdown' and 'asciidoc' are supported."
    default: 'markdown'
    required: false

runs:
  using: 'composite'
  steps:
    - name: Set verbose logging env variable
      if: "${{ inputs.verbose-logging == 'true' }}"
      shell: bash
      run: |
        echo 'VERBOSE=--verbose' >> $GITHUB_ENV

    - name: Setup Node
      uses: actions/setup-node@v4

    - name: Configure AWS credentials
      uses: aws-actions/configure-aws-credentials@v4
      with:
        role-to-assume: <IAM role ARN to push to S3>
        aws-region: some_region

    - name: Setup Python
      uses: actions/setup-python@v5
      with:
        python-version: '3.9'

    - name: Setup Ruby
      uses: ruby/setup-ruby@v1
      if: "${{ inputs.doc-source == 'asciidoc' }}"
      with:
        ruby-version: '3.3'
        bundler-cache: true

    - name: Install asciidoc dependencies
      if: "${{ inputs.doc-source == 'asciidoc' }}"
      shell: bash
      run: |
        gem install asciidoctor-reducer
        npm install -g downdoc

    - name: Install techdocs-cli and markdown
      shell: bash
      run: |
        sudo npm install -g @techdocs/cli
        python -m pip install mkdocs-techdocs-core==1.*
        python -m pip install mkdocs-git-revision-date-localized-plugin
        python -m pip install mkdocs-material

    # --- doc-source: markdown ---
    - name: Generate docs site for markdown
      if: "${{ inputs.doc-source == 'markdown' }}"
      shell: bash
      run: |
        techdocs-cli generate --no-docker ${{ env.VERBOSE }}

    # --- doc-source: asciidoc ---
    - name: Generate docs site for asciidoc
      if: "${{ inputs.doc-source == 'asciidoc' }}"
      shell: bash
      run: |
        mkdir /tmp/reduced
        asciidoctor-reducer docs/index.adoc -o /tmp/reduced/index.adoc
        downdoc /tmp/reduced/index.adoc -o docs/index.md
        techdocs-cli generate --no-docker ${{ env.VERBOSE }}

    - name: Publish docs site
      shell: bash
      env:
        ENTITY_NAMESPACE: 'default'
        ENTITY_KIND: ${{ inputs.entity-kind }}
        ENTITY_NAME: ${{ github.event.repository.name }}
        ENTITY_NAME_OVERRIDE: ${{ inputs.component-name }}
      run: |
        if [[ -n $ENTITY_NAME_OVERRIDE ]]; then
          ENTITY_NAME=${ENTITY_NAME_OVERRIDE}
        fi

        techdocs-cli publish \
          --publisher-type awsS3 \
          --storage-name some_s3_bucket_name \
          --entity ${ENTITY_NAMESPACE}/${ENTITY_KIND}/${ENTITY_NAME}

We also added a devcontainer that has all the above tools installed for local development Here is how the Dockerfile looks like:

# Use an official Python runtime as a parent image
FROM python:3.11-slim

# Install necessary tools
RUN apt-get update && \
     apt-get install -y --no-install-recommends curl ruby nodejs npm

# Set up a working directory
WORKDIR /workspace

# Install asciidoctor-reducer gem
RUN gem install asciidoctor-reducer

# Install downdoc
RUN npm install -g downdoc

# Install techdocs-cli
RUN npm install -g @techdocs/cli

# Install Python dependencies
RUN pip install --no-cache-dir \
    mkdocs-techdocs-core==1.4.0 \
    mkdocs-git-revision-date-localized-plugin \
    mkdocs-material

# Command to start the container
CMD ["bash"]

and the devcontainer.json:

{
    "build": { "dockerfile": "Dockerfile" }
}

Some limitations that we discovered for features that are not supported: Admonitions

• Admontions with the paragraph syntax display a backslash \ at the end of the headline
• Admonitions with the block syntax have the problem that formatting and lists are not considered

Image caption

• Image captions are shown above the image and without a figure number.

Numbered headlines

• Numbering is not displayed

Description lists

• Title can contain a baskslash at the end
• The content of description lists can be displayed incorrectly

Equations and Formulas (STEM)

• Formulas are not rendered correctly

Hope this helps until native support comes to Backstage. Thank you @mojavelinux for providing these awesome tools.

[pwright]

@wanisfahmyDE that looks cool.. Given that you cannot accept contributions in the intermediate markdown format, are there advantages to converting to markdown rather than html as a workaround?

[wanisfahmyDE]

hey @pwright, not sure if I got your question right but the techdocs-cli generates htmls from the markdown eventually and this is what is published to S3 when running techdocs-cli publish.

Initially we tried achieving the same with pandoc and asciidoctor but this caused several issues like titles not appearing, incomplete table of contents and also broken PlantUMLs. Not to mention, maintaining a long bash script to do all these conversions. downdoc and asciidoc-reducer kind of made it a "one liner(ish)" solution

[pwright]

@wanisfahmyDE Sorry if I wasn't clear, this thread above documents the idea of just converting your adoc to html (as you might do for a docs website) and then changing all the filenames to md.

Voila! you have input files that work in backstage (because html is valid markdown).

Not 100% sure, but using asciidoctor -a data-uri <file> should also make sure all images/UML becomes embedded in your "markdown".

I actually use your approach to populate a markdown site skupper.io so I'm good with it, only suggesting this as an alternative to anyone who doesn't want to learn reducer and downdoc.

[mojavelinux]

Admontions with the paragraph syntax display a backslash \ at the end of the headline

This is hard line break. If you prefer not to use that style, pass -a markdown-line-break to the downdoc command.

See https://github.com/opendevise/downdoc/blob/main/test/downdoc-test.js#L1107-L1121

If you have other question about how downdoc works, you're welcome to ask in the project chat at https://chat.asciidoctor.org.

Naturally, not all features of AsciiDoc will be supported since Markdown is a simpler language than AsciiDoc.

[snowe2010]

@pwright yeah the html solution you initially suggested and I implemented for us in gradle has been working great.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[snowe2010]

not stale...

[thrix]

Maybe we could at least document these solutions for the teams having the majority of their docs in Asciidoc? I can contribute if anybody thinks this would be useful.

[he-mi]

A documented (and supported) solution would be great. Especially in combination with Antora.