search

bwplotka / mdox

Format your docs; autogenerate from flags or Go structs or even generate versioned website directly from markdown!
Apache License 2.0
67 stars 11 forks source link

readme

mdox

go.dev reference Latest Release CI Go Report Card Slack

mdox (spelled as md docs) is a CLI for maintaining automated, high-quality project documentation and website leveraging GitHub Flavored Markdown and git.

This project can be used both as CLI as well as a library.

Goals

Allow projects to have self-updating up-to-date documentation available in both markdown (e.g readable from GitHub) and static HTML. Hosted in the same repository as code and integrated with Pull Requests CI, hosting CD, and code generation.

Features

Usage

Formatting and Link Checking

Just run mdox fmt and pass markdown files (or glob matching those).

For example, this README is formatted by the CI on every PR using mdox fmt -l *.md command.

usage: mdox fmt [<flags>] <files>...

Formats in-place given markdown files uniformly following GFM (GitHub Flavored
Markdown: https://github.github.com/gfm/). Example: mdox fmt *.md

Flags:
  -h, --[no-]help              Show context-sensitive help (also try --help-long
                               and --help-man).
      --[no-]version           Show application version.
      --log.level=info         Log filtering level.
      --log.format=clilog      Log format to use.
      --profiles.path=PROFILES.PATH  
                               Path to directory where CPU and heap profiles
                               will be saved; If empty, no profiling will be
                               enabled.
      --metrics.path=METRICS.PATH  
                               Path to directory where metrics are saved in
                               OpenMetrics format; If empty, no metrics will be
                               saved.
      --[no-]check             If true, fmt will not modify the given files,
                               instead it will fail if files needs formatting
      --[no-]soft-wraps        If true, fmt will preserve soft line breaks for
                               given files
      --[no-]code-fmt          Reformat code snippets
      --[no-]code.disable-directives  
                               If false, fmt will parse custom fenced
                               code directives prefixed with 'mdox-gen' to
                               autogenerate code snippets. For example:

                                 ```<lang> mdox-exec="<executable + arguments>"

                               This directive runs executable with arguments
                               and put its stderr and stdout output inside code
                               block content, replacing existing one.
      --anchor-dir=ANCHOR-DIR  Anchor directory for all transformers. PWD is
                               used if flag is not specified.
      --links.localize.address-regex=LINKS.LOCALIZE.ADDRESS-REGEX  
                               If specified, all HTTP(s) links that target a
                               domain and path matching given regexp will be
                               transformed to relative to anchor dir path (if
                               exists). Absolute path links will be converted to
                               relative links to anchor dir as well.
  -l, --[no-]links.validate    If true, all links will be validated
      --links.validate.config-file=<file-path>  
                               Path to YAML file for skipping
                               link check, with spec defined in
                               github.com/bwplotka/mdox/pkg/linktransformer.ValidatorConfig
      --links.validate.config=<content>  
                               Alternative to 'links.validate.config-file'
                               flag (mutually exclusive). Content of YAML file
                               for skipping link check, with spec defined in
                               github.com/bwplotka/mdox/pkg/linktransformer.ValidatorConfig
      --[no-]cache.clear       If true, entire cache database will be dropped
                               and rebuilt when mdox is run. Useful in case
                               cache needs to be cleared immediately from GitHub
                               Actions or other CI runner cache.

Args:
  <files>  Markdown file(s) to process.

Code Generation

It's not uncommon that documentation is explaining code or configuration snippets. One of the challenges of such documentation is keeping it up to date. This is where mdox code block directives comes handy! To ensure mdox will auto update code snippet add mdox-exec="<whatever command you want take output from>" after language directive on code block.

For example this Readme contains mdox --help which is has to be auto generated on every PR:

```bash mdox-exec="mdox fmt --help"
...

This also enables auto updating snippets of code in code blocks using tools like sed. For example, below code block directive will auto update and insert lines 3 to 6 from main.go into code block.

```go mdox-exec="sed -n '3,6p' main.go"
...

Some commands might have non-zero exit codes. mdox will fail commands in such cases(otherwise errors might get formatted into markdown) but the expected exit code can also be passed as a code block directive! For example, below code block executes go --help which has 2 as its exit code,

```text mdox-exec="go --help" mdox-expect-exit-code=2
...

You can disable this feature by specifying --code.disable-directives

Link Validation Configuration

By default, mdox checks all links (both relative and remote) in passed markdown files! However, in some cases, link checks might fail even when links are working, such as with rate limiting, Cloudflare protections, or something else(like localhost links in docs).

This might lead to failed CI checks which aren't desirable. So, you can use the links.validate.config-file flag to pass in YAML configuration file for selective link checking using special validators and link regexes.

For example,

version: 1
timeout: '1m'
parallelism: 100
host_max_conns: 2
random_delay: '1s'

validators:
  - regex: '(^http[s]?:\/\/)(www\.)?(github\.com\/)bwplotka\/mdox(\/pull\/|\/issues\/)'
    type: 'githubPullsIssues'

  - regex: 'localhost'
    type: 'ignore'

  - regex: 'thanos\.io'
    type: 'roundtrip'

As seen above, mdox supports validate configuration supports a few parameters and passing an array of link validators with types and regexes. The supported configuration parameters are:

There are three types of validators:

Relative link checking is not affected by this configuration, as it is expected that such links will work.

YAML can be passed in directly as well using links.validate.config flag! For more details go.dev reference or Go struct.

Link localization

It is expected for documentation to contain remote links to the project website. However, in such cases, it creates problems for multi-version docs or multi-domain websites (links would need to be updated for each version which is cumbersome). Also, it would not be navigatable locally or through GitHub (would always redirect to the website) and requires additional link checking.

This is where the links.localize.address-regex flag comes in handy!

It ensures that all HTTP(s) links that target a domain and path matching given regex will be transformed by mdox to relative links which are relative to anchor dir path (if exists). Also, all absolute path links will be converted to relative links to anchor dir as well.

So passing in regex such as --links.localize.address-regex="https:\/\/example\.\/.* will allow mdox to transform links like https://example.com/getting-started.md/ to simply getting-started.md.

Transformation

mdox allows various types of markdown file transformation which are useful for website pre-processing and is often required when using static site generators like Hugo. It helps in generating front/backmatter, renaming, and moving files, and converts links to work on websites.

Just run mdox transform --config-file=.mdox.yaml and pass in YAML configuration.

usage: mdox transform [<flags>]

Transform markdown files in various ways. For example pre-process markdown files
to allow it for use for popular static HTML websites based on markdown source
code and front matter options.

Flags:
  -h, --[no-]help                Show context-sensitive help (also try
                                 --help-long and --help-man).
      --[no-]version             Show application version.
      --log.level=info           Log filtering level.
      --log.format=clilog        Log format to use.
      --profiles.path=PROFILES.PATH  
                                 Path to directory where CPU and heap profiles
                                 will be saved; If empty, no profiling will be
                                 enabled.
      --metrics.path=METRICS.PATH  
                                 Path to directory where metrics are saved in
                                 OpenMetrics format; If empty, no metrics will
                                 be saved.
      --config-file=<file-path>  Path to Path to the YAML
                                 file with spec defined in
                                 github.com/bwplotka/mdox/pkg/transform.Config
      --config=<content>         Alternative to 'config-file' flag
                                 (mutually exclusive). Content of Path
                                 to the YAML file with spec defined in
                                 github.com/bwplotka/mdox/pkg/transform.Config

For example,

version: 1

inputDir: "docs"
outputDir: "website/docs-pre-processed/tip"
extraInputGlobs:
  - "CHANGELOG.md"
  - "static"

gitIgnored: true
localLinksStyle:
  hugo:
    indexFileName: "_index.md"

transformations:

  - glob: "../CHANGELOG.md"
    path: /thanos/CHANGELOG.md
    popHeader: true
    frontMatter:
      template: |
        title: "{{ .Origin.FirstHeader }}"
        type: docs
        lastmod: "{{ .Origin.LastMod }}"
    backMatter:
      template: |
        Found a typo, inconsistency or missing information in our docs?
        Help us to improve [Thanos](https://thanos.io) documentation by proposing a fix [on GitHub here](https://github.com/thanos-io/thanos/edit/main/{{ .Origin.Filename }}) :heart:

  - glob: "getting-started.md"
    path: /thanos/getting-started.md
    frontMatter:
      template: |
        type: docs
        title: "{{ .Origin.FirstHeader }}"
        lastmod: "{{ .Origin.LastMod }}"
        slug: "{{ .Target.FileName }}"

  - glob: "../static/**"
    path: /favicons/**

As seen above,

YAML can be passed in directly as well using --config flag! For more details go.dev reference or Go struct.

Installing

Requirements to build this tool:

go install github.com/bwplotka/mdox@latest

or via bingo if want to pin it:

bingo get -u github.com/bwplotka/mdox

Production Usage

Contributing

Any contributions are welcome! Just use GitHub Issues and Pull Requests as usual. We follow Thanos Go coding style guide.

Have questions or feedback? Join our slack channel!

Initial Author

@bwplotka

Note: This project was a part of GSoC'21 (mentor: @bwplotka, mentee: @saswatamcode).