Created at VS Started at

[tikolakin]

Is your feature request related to a problem? Please describe. According to the current design a test run when created will be active (in progress)

This doesn't always a desired behaviour. Quite often, we create test runs a week before actually testing starts. In that case test run duration will be not valid and won't reflect actual test execution time.

Describe the solution you'd like Currently "Started" timestamp is basically a created-at timestamp. This is not quite valid if we want to have a meaningful metric for test run duration. It will have more sense to add "Created" attribute and have it display actual created-at timestamp AND Change "Started" attribute to a timestamp of a first test status assigned to any test of a given test run

Additional context Current test run duration is not valid as it doesn't reflect actual test execution time

By fixing duration to the actual duration of test execution - we can have a reliable metric of how long test execution takes. Too short or too long....

[tikolakin]

Change "Started" attribute to a timestamp of a first test status assigned to any test of a given test run This is not correct either, as it doesn't take into account duration of a first test

[poliarush]

@DavertMik what is your decision here? namely to introduce "Started At" by setting first test case's status

[DavertMik]

Yes, to add a new column

[tikolakin]

@poliarush I am not sure about

"Started At" by setting first test case's status

being a correct approach. It might need more thoughts and design The ultimate goal is to have reliable and valid test run duration. Currently it's not valid for most of our test runs. I understand that we can avoid invalid test run duration by starting test execution just after we create test run. But in our scale it just doesn't work. We have too many test runs to manage like it.

Means we are really looking at how to get test run duration metrics that is valid generally.

[tikolakin]

but

to introduce "Started At" by setting first test case's status

it will be better then it currently is . It will be relatively correct timestamp

[poliarush]

related to https://github.com/testomatio/docler-issues/issues/396

valid and reliable is to set it fully manually when you start and end working on test cases, but there is another problem here, human factor, where people will forget to start-stop timers.

I'd say we don't have ideal solution here.

[tikolakin]

Exactly, we are looking for a solution in TMS and not people. As you mentioned a human factor is difficult and not reliable. I think we can only find optimal solution to address this metrics in manual test process.

Note, that's it's not an issue for automated test runs where we have test duration calculated by the sum of all tests durations in test run.

[poliarush]

@DavertMik

i'm also thinking on:

• give possibility to edit "Started At" if it's not correct
• auto-tracking duration for each test case (showing different metrics so users can see all of it and decide where is the truth)
  • by differences between statuses (prev test status time - current test status time)
  • by start-stop of whole run (whole run time / amount of tests)
  • by manual timers (stop time - start time)
• set manual timer when set status for test cases
• it may be calculate average between metrics?! to understand more accurate duration

[tikolakin]

by start-stop of whole run (whole run time / amount of tests) by manual timers (stop time - start time)

This is sensible and can help with the metric of test duration. Especially when the test run is not completed without interruptions. In that case, if there will be manual start/stop it would be quite helpful to record at least number of stops. Example: Test run duration - 24h, stops (interruptions) - 3

[DavertMik]

@tikolakin we discussed the duration issue

Currently we decided to use following strategy:

Run Duration = SUM(Tests Durations)

Otherwise system becomes too complicated with lots of edge cases. Also we will add better duration calculations for manual tests, so you could track time spent on each test case.

[tikolakin]

I am not sure we need to change run duration for automated tests. SUM(Tests Durations) looks wrong as it doesn't take into account time of the test run spent in queues or setup etc.

[DavertMik]

@tikolakin we can't correctly measure all the setup, queues, anyway

so in this case the only way we can calculate is

Finished At - Created At

but this number becomes invalid for:

• reruns
• parallelization (sum of testrun durations less than duration of run)
• etc

[tikolakin]

I think that for automated tests the current solution of Finished At - Created At is all fine This issue was created mainly to address manual test runs

Again, the SUM(Tests Durations) is wrong for automated test. Please don't change it.

reruns parallelization (sum of testrun durations less than duration of run) etc

Those parameters do not matter for automated tests.

[DavertMik]

Ok, thanks, we will keep it