search

pytest-dev / pytest

The pytest framework makes it easy to write small tests, yet scales to support complex functional testing
https://pytest.org
MIT License
12.13k stars 2.68k forks source link

Feature request: Open test reporting #11707

Open couling opened 11 months ago

couling commented 11 months ago

The Problem

At the moment the only way to get structured output from pytest is via --junitxml or installing a plugin. The problem with junitxml is that, as a java format some information from pytest cannot be exported (eg: warnings).

This means that they only way to get complete structured output from pytest is via a plugin. What would be much better is if pytest could produce just one structured output that any tool in any language could then pick up and format, rather than requiring the formatting tool to run in the same virtual environment and process as the tests themselves.

The driver for me here is that I can't demand that multiple teams update 300+ repos to add a plugin for the sake of a CI change, and injecting plugins in a CI pipeline is a great way to make tests pass or fail in CI and not on people's local machine.

Requested Solution

I'd like to ask for pytest to implement open-test-reporting once open-test-reporting v1 has been released. This is designed to be extensible meaning that pytest can add things like warnings without worrying about backward compatibility with the base standard.

RonnyPfannschmidt commented 11 months ago

At first glance the proposed format is excessively complex, unless someone creates a high level consumer/producers lib, it's unlikely pytest touches it at all