Question/Suggestion: test remote site instead of local files

[antrix]

Currently, htmltest works against locally generated html files. I've tried it with a site generated by Pelican and it works great.

I was wondering if you have any plans to support running htmltest against a remote website? So instead of taking a local path as input, it would take a remote URL as input and test pages that it crawls. To limit the test, it wouldn't crawl out of the same site, i.e. something like --skip-external but for page discovery.

[wjdp]

Hi @antrix this isn't something that's really ever been considered. I've always seen htmltest as part of a CI/CD process for a statically generated site.

Implementing this would be a large change and potentially challenging:

• Documents to test are loaded from the filesystem. This would need to be changed to pull from network.
• We don't have a listing of files so discovery of documents would have to happen while checking links. (if url starts with mywebsite.com then add to doc store)
• All link checks become external link checks.
• Likely more 😄

The codebase is quite simple at present. Adding this feature introduces a whole "second mode" of operation that could complicate it quite a bit.

I would consider a pull request for this given enough demand and a good implementation. It may have to be a fork of this project for a while before a merge is considered.

[antrix]

Thank you for sharing that. I can see the rationale to not add this feature, especially the second bullet. I hadn't realized that the implementation does discovery of documents upfront instead of doing it on the fly. That would be a big change indeed.

I'll leave the issue open for a while in case more folks in the community want to chime in.

[wjdp]

Closing