search

fediverse-devnet / feditest

A testing framework for distributed, heterogeneous systems communicating with complex protocols, such as the Fediverse
https://feditest.org/
MIT License
31 stars 6 forks source link

Better error reporting if specified template does not exist #217

Closed jernst closed 1 month ago

jernst commented 2 months ago

feditest convert-transcript ... --template does-not-exist should report something like:

ERROR: template "does-not-exist" does not exist.

steve-bate commented 1 month ago

feditest convert-transcript ... --template does-not-exist should report something like:

ERROR: template "does-not-exist" does not exist.

This may need more discussion. The argument --template some-template does not actually specify a template although the documentation says it does. It specifies a template path that's used by the Jinja2 template loader to find the hard-coded test_matrix.jinja2 and test_session.jinja2 templates (and supporting templates).

For example, I could create a templates/steve subdirectory and specify --template-path (new name for the arg) as 'steve,default'. That would cause any template references to be searched for in the steve directory first and then in the default directory if not found in steve. I'd be able to selectively override the default report that way. BTW, those path elements are not required to be a templates subdirectory. An absolute path will be used as-is, a relative path will be interpreted as relative to the templates directory (e.g., --template-path /tmp/experiment,steve,default).

I'm guessing most users will use the default template. Those who want to customize the template will need to know jinja2 and they should understand jinja2 template loading well enough to diagnose any related issues.

jernst commented 1 month ago

From the perspective of the user who does not know (and should not need to know) how --template <name> is implemented, it doesn't really matter how <name> is mapped to what they see on screen once the report has been generated. This issue is intended to say "regardless what the mapping from <name> to internal template is, if there is no mapping for a provided argument, provide a comprehensible error message which basically says: I have no idea what template you are talking about."

I think the two main "templates" that should be exposed to the user as 1) multi-application: matrix overview, and details for each app, and 2) "I am just interested in the results for my own app", so I should be able to specify the app-specific report only.

steve-bate commented 1 month ago

Overloading the term "template" like that is going to be the worst case scenario, confusing in different ways for both developers (at least me) and non-developers.

We use jinja2 templates for producing a report from a transcript. Developers may want to customize a template, either for a full report or a single-server report. The --template-path argument supports that and the diagnostics I've added can be useful to diagnose issues with that feature. I personally use template paths to customize the default templates for reports that I use for various purposes. The template path allows me to do this locally without committing anything to the main/develop branches.

It sounds like you are talking about styles of reports. Maybe we need a --report-style argument (mutually exclusive with --template-path) for that purpose? This argument would be used internally to set the template path.

If we add this (and I think it's a reasonable idea for the long term, if we eventually have non-technical feditest users), I'd like to do that in a subsequent PR.

jernst commented 1 month ago

Sounds reasonable :-)