search

WordPress / phpunit-test-reporter

GNU General Public License v3.0
29 stars 21 forks source link

Support Multiple Submissions from Hosts: Research Runner and Reporter Codebase #65

Open kittenkamala opened 4 years ago

kittenkamala commented 4 years ago

Research the existing code base and configuration for reporting test results to identify all the specific points that need to be updated in the reporter and runner code. Document that here to help facilitate a solution for supporting multiple test submissions from hosts.

kittenkamala commented 4 years ago

https://github.com/WordPress/phpunit-test-reporter/blob/master/src/class-restapi.php#L116

kittenkamala commented 4 years ago

result-set.php makes a table / template that test results get pushed to

result data is being iterated through here: https://github.com/WordPress/phpunit-test-reporter/blob/master/parts/result-set.php#L34

from the result data the post author is being grabbed here: https://github.com/WordPress/phpunit-test-reporter/blob/master/parts/result-set.php#L43

kittenkamala commented 4 years ago

In the runner, results get put in an array, json encoded and sent to the api:

https://github.com/WordPress/phpunit-test-runner/blob/master/functions.php#L106

pfefferle commented 4 years ago

What about starting with a simple env var like export WPT_HOSTING_PACKAGE=Cloud Hosting? I think it is no problem to have multiple runners for multiple hosting packages.

getsource commented 4 years ago

@pfefferle Thanks for the note! I think that makes sense. I like that it would make it really easy for hosts to add/specify another environment.

Agreed it isn't a problem! What do you think should be the next step?

On getting it to the reporter, we could roll it up into $env, a different existing field, or add a new one.

Some details on the $env option: $env is curated information in JSON about the environment where tests are run. It's currently generated in a couple of places. It doesn't contain any info from the.env file, and because it is sometimes run on a different machine, that'd probably have to be injected somehow.

pfefferle commented 4 years ago

Semantically $env would be perfect. get_env_details() would be a good place to start, but then it will be ignored for the env.json file? Perhaps the reporter.php should add the info: https://github.com/WordPress/phpunit-test-runner/blob/67347895ad59d94abec722c69753a8184daa4aa8/report.php#L47

pfefferle commented 4 years ago

What speaks against using $env is, that it is a very dominant parameter, that will also be used to group and filter the output (a hoster will be represented with one, two or more item types in a revision segment, with the name of the hosting type), so it might be better to use a more prominent field?!