search

web-platform-tests / wpt

Test suites for Web platform specs — including WHATWG, W3C, and others
https://web-platform-tests.org/
Other
4.97k stars 3.09k forks source link

There should be a single source of truth for all test + subtest status values #14526

Open mdittmer opened 5 years ago

mdittmer commented 5 years ago

I can't seem to find source code/data that acts as the single source of truth for potential test and subtest status values that may appear in a test run report. (I can't find human readable documentation that includes all such values either, but that's a somewhat separate issue).

IMHO, there should be a piece of code or data file that acts as the single source of truth for what is essentially an enum with string representations of all "TestStatus" values and "SubtestStatus" values.

In addition to writing the enum down somewhere, we should, presumably, chase down all the code that gets or sets status values and have it interpret the legitimacy of a value through the single source of truth.

Depending on what form the source of truth takes (JSON, YAML, Python code, etc.) perhaps it can include metadata that serves as documentation for each value.

mdittmer commented 5 years ago

//CC @LukeZielinski

jgraham commented 5 years ago

That exists in mozlog:

https://searchfox.org/mozilla-central/source/testing/mozbase/mozlog/mozlog/logtypes.py#184-196

mdittmer commented 5 years ago

@jgraham do you think there should be consensus on these types that their values and documentation should live in web-platform-tests/wpt?

jgraham commented 5 years ago

I mean the principled way to do that is probably by reference to mozlog i.e. have documentation stating that we support the statuses that are supported by mozlog, since that's the lowest-level component here. I don't think it makes sense to make a normative claim on the values, but we can certainly document the meanings of the ones that are actually used.

mdittmer commented 5 years ago

I see. wpt consumes the mozlog Python package. Yes, I think it would be good to document: in wpt, "we support mozlog status values docmented at "; in mozlog, precise semantics of each status.