Tool: detect links in the documentation content that lead to HTTP 404 errors

[tbouffard]

IMPORTANT: #430 lists some broken links and provide more context

Internal links are done using Antora xref and we are validating xref on PR preview and when building the whole site.

However, we don't check that external site are valid. We should run a tool at least periodically in the whole site. It could be built and serve locally if this speedup the process, or we could retrieve the whole content from Netlify (download zip archive).

Netlify

• analytics provide 404 detection on resources requested on the site. This is an extra paid feature https://docs.netlify.com/monitor-sites/analytics/
• send logs with log drain to an external service (enterprise plan): https://docs.netlify.com/monitor-sites/log-drains/

Other solutions

• in the Antora zulip chat, the question about detecting 404 has been asked several times
  • https://antora.zulipchat.com/#narrow/stream/282400-users/topic/stable.20and.20validated.20references.20.28e.2Eg.2E.20as.20application.20help.29 --> https://github.com/wjdp/htmltest. Should be fast as it is written in Go and use a cache for external URL stored in the file system (could be shared in the Gith repo). Provide a docker image.
• used in the past at Bonitasoft for evaluation: https://linkchecker.github.io/linkchecker/ may be slower as it is written in Python 3

Strategy to test the tools

WIP list

• if allowed by the tool, use a locally built site
• start by checking a single component version
• disable the "edit this page" link ~(may be possible by setting an asciidoc attribute globally in the antora-playbook.yml, probably won't work for branches were "page-editable: true" in antora.yml)~. --> use the --hide-edit-page-links option of the "build preview" script
• ignore some urls that appear in all pages (logo bonitasoft, logo of tools)
• do not generate redirect static page. This is not for production and we don't want to test the Antora redirects generation. Such static page include canonical and link to the production site that we don't want to check.
• generate page with html extension to let the tools browse the page on the filesystem

Detected problems with htmltest --> URLs to ignore

• Get "/bonitasoft": stopped after 10 redirects --- labs/latest/index.html --> https://twitter.com/bonitasoft: lead to a authentication page when using a browser
• Non-OK status: 999 --- labs/latest/index.html --> https://www.linkedin.com/company/bonitasoft/: lead to a authentication page when using a browser
• http://localhost... used in examples