search

microsoft / testfx

MSTest framework and adapter
MIT License
738 stars 255 forks source link

Show slowest tests #3495

Open Evangelink opened 2 months ago

Evangelink commented 2 months ago

Summary

Allow to display slowest tests for a given assembly.

Background and Motivation

On our internal test runs we have an option to display slowest tests. I think this would be a useful feature to provide to users.

Proposed Feature

Add a --show-slowest or --show-slowest-test option.

Evangelink commented 2 months ago

To de defined: should this be a core feature or an extension feature?

I think that because this is not linked to any external deps/service this could be a built-in feature but we could think about having it directly as some external package it we are planning to connect it to some service (AzDO?). @MarcoRossignoli if I am not mistaken, nothing prevents us (technically) to have this feature as core (built-in) feature and to potentially have some extensions that would enrich this feature (e.g. connection to some history service), right?

MarcoRossignoli commented 2 months ago

I would personally do not add "questionable" service built-in. UX falls already in that bucket but we need to show something and we're giving our taste. I would keep as an extension.

Slowest test is also a not well defined concept. Hypothetical fixture system, init feature etc...are affecting the concept of slowest? If my custom adapter group by "group of tests with a category" and I want to measure that group? Some other esoteric framework could implement different concept regarding timing, personally I would not add in the core lib but in an extension where we ship "our" idea of slowest.