Open fralau opened 4 days ago
CC: @pawamoy, @tomchristie, @squidfunk, @razorback, @gesslar
Looks like you've put a lot of thought into this. Given the cc, I take it you're asking for some feedback?
Also, programmatically checking the resulting HTML page opens a rabbit hole.
This is exactly what I do in revision-date-localized and table-reader, and I've never looked back. The only thing I had to add was scheduling of the unit tests (weekly) so I am aware of any integrations breaking because of changes upstream (mkdocs) or downstream (plugins I want to be compatible with).
TBH, in my work I always push for trying to keep things as simple as possible (e.g. What's the simplest possible thing that might work, and why didn't you try that first?. Fun thing about open source is that it's a hobby, we can go and enjoy building awesome things. And looks like you built something that works well for this plugin. I would be a bit concerned about maintainability.. do you have tests for the test framework π ?
Thanks for your thoughtful remarks. I am indeed asking for feedback (providing of course that others wish to give it).
I am with you on doing the simplest thing; and indeed, that was my main concern: I wanted to write the simplest testing code possible. Also, I looked around to see what solutions existed, especially yours.
The one difference with most other plugins, is that Mkdocs-Macros can do by nature many different things, in many different ways (thanks to Jinja2). There is potentialy a lot to test, with many little details that could go wrong. I started from how I want to formulate the test assertions and I worked my way back to built the framework so as to support those assertions.
The reason why I didn't want to test final the HTML is that I don't know what other plugins could do to the code, not to mention the markdown extensions and the themes -- perhaps that choice was overly strict. But that simplified the framework for me, because once I produced the "rendered" ones (essentially just before returning the result of on_page_markdown()
), I have to compare file sets with strictly identical structure and names.
Hence spending some time at the start, has been an investment that should pay back in the long trange.
Maintenability is one of my concerns. The framework had to be as robust as possible, and be based on concepts or mechanisms that are not susceptible of much change in the long run. I am not overly concerned about reading the markdown files in the docs
directory, or the rendered ones in their own directory. Reading the config.yaml
is a piece of cake.
The one part that could possibly break in the future, is the log, since I made up the idea of the payload. Fortunately, it didn't do so. So, unless a problem comes up , I might continue to use that approach.
Finally, yes, there is a whole series of test cases for the test framework itself. π
In mkdocstrings-python I'm aiming for end-to-end tests (or golden tests?), where I render HTML for a single object (recursively) and compare the output to snapshots. It doesn't test compatibility with other plugins (such as mkdocs-macros), but I suppose it wouldn't be hard to extend the test cases to include some tests that also make use of mkdocs-macros (for example) to assert compatibility.
But your testing framework looks nice. Give me a bit of time to comment on it more.
Thanks for your feedback, as this is encouraging me to sift my ideas, recognize weaknesses, and find new avenues.
@pawamoy I like the term golden test, for a test that compares the the output of a (long and complex) process, with an expected output. π
End-to-end is the expected approach, though how we define it in each case might vary.
One can conceive the output of Mkdocs-Macros as an input for Mkdocs.
Indeed, the files of raw Markdown generated by Mkdocs-Macros in debug mode (enriched with YAML front-matter), should be a drop-in replacement for the original Markdown pages.
Here is a diagram:
flowchart TD
subgraph First
Source[Markdown + Jinja2 pages] --> MkDocs(Mkdocs + Macros)
MkDocs --> Targeta[HTML]
MkDocs --> Targetb["Markdown pages<br>(debug)"]
end
Targetb --> Source2
subgraph Second
Source2[Markdown pages] --> MkDocs2(Mkdocs)
MkDocs2 --> Target2[HTML]
end
From that perspective what I am really trying to test, is whether or not the plugin is rendering the page correctly with Jinja2 when it should, and not rendering it when it should not. I am not testing MkDocs itself, particularly . Hence "end-to-end" could be interpreted as "from Markdown + Jinja2 to Markdown".
From the perspective of MkDocs, that would be unit test, I guess. π€
This is one way to look at things, of course. π
This should illustrate how a test can be done with the framework:
from test.fixture import DocProject
PROJECT = DocProject('myproject')
PROJECT.build(strict=False)
# did not fail
assert not PROJECT.build_result.returncode
# ----------------
# Page 'index'
# ----------------
page = PROJECT.get_page('index')
assert page.is_rendered
VARIABLE_NAME = 'greeting'
# it is defined in the config file (extra)
assert VARIABLE_NAME in PROJECT.config.extra
# check that the `greeting` variable is rendered:
assert VARIABLE_NAME in PROJECT.variables
assert PROJECT.variables[VARIABLE_NAME] in page.markdown
@pawamoy The adage seems right: βInside every large program, there is a small program trying to get out.β β C.A.R. Hoare
I realized that the framework could perhaps be generalized for testing any doc site, or any plugin. So I made a general version (DocProject) and the features I needed specifically for Mkdocs were put into a subclass (MacrosDocProject). So far, it works.
I wonder whether I could extract the core of the test framework and turn it into a Python package for general use? I would see two use cases:
The Issue
Mkdocs-Macros needs a testing framework. This is necessary, with (according to Github) over 3500 projects depending on it, some of which are large or have themselves many dependent projects.
History
Jinja2 made this easy, and the initial version of the plugin, back in 2018, was simple. Most of the later complications derived from "real-world" considerations:
The question of how to test the final results arose immediately. I solved it by using the main tool that Mkdocs provides for that purpose:
mkdocs serve
and by watching the results in a browser. It is a quick and effective method to test anything one wishes.What is needed
Hence Mkdocs-Macros needs a testing framework, in view of Continuous Integration on GitHub.
Ideas
It is easier said than done.
In #241, I summarized the discussions prompted by @timvink, inspired from his experience on mkdocstrings. It all started from the discussion on how to make Mkdocs coexist with other plugins; we agreed we needed a hook (#237); this was done... and then the question arose of how to test the result β.
His contribution was essential, because it framed the problem. He also kindly submitted a PR (#239) based on pytest, which contained a good start.
I realized, however, that I would have to take a step back, and think this problem through.
Why it is difficult?
Examples are:
Of course the log (especially with the
--debug
option) I realized that I needed a framework for that.Why I didn't use Mkdocs
One way to solve this issue, might have been to attempt to use the Mkdocs framework itself, .
Aside of the fact that it would have required an intimate knowledge of the intricacies this framework that I don't have, I realized that using Mkdocs to test itself would risk creating assertions that are tautologies or begging the question (accidentally formulated in a way that they can't give a False answer, because they are basically the same thing expressed in two different ways).
Solution
Here is an initial description.
Principle
The best approach was to make a completely distinct test framework.
The Test Framework, executes
mkdocs build --debug
(and if required,--strict
) and then compares the following five inputs:Notes on the Log
The log is parsed into a list of log objects.
There are three types of log entries:
Each properly formatted log entry has a severity ('INFO'), an optional source ('macros'), a title ('Macros arguments') and an optional payload (any text).
Target documents
The target documents are raw Markdown documents (after Jinja2 has been rendered), to which the original YAML header has been added. They are adequate to test the result of Mkdocs-Macros, as produced by
on_page_markdown()
The framework collects parses each file and provides:
First Results
A first version of the test framework (
test/fixture.py
) has been produced.The test framework provides a single DocProject object, which contains all elements necessary to test:
on_conf
(each page is then completed by its own metadata)Making
cd
into thetest
directory, and runningpytest
launches the existing tests, on two test documentation projects:simple
module