report on added autotests

[ScharfViktor]

Task:

We need to implement the creation of a QA activity report (automated testing context) that will include information which tests we have added/deleted/changed in a certain time (should be configurable)

It should only contain information about the change to the .feature file. We do not need to provide stepImplementation info

we need include info to the report:

common info:

- Added lines to the .feature files: 391
- Deleted lines in the .feature files: 191
- Modified lines in the .feature files: 582

specific information:

-  added/deleted/changed scenario - name and counts
- number of  added/deleted/changed steps - only counts
- if we have a table of examples -> case numbers in one scenario
- commitId
- author
- date

first we need provide report for web(e2e) and ocis(acceptance) tests.

I guess we need get git log first and parse log to a readable report: web: git log --since='20 days ago' -p -- tests/e2e/cucumber/features ocis: git log --since='20 days ago' -p -- tests/acceptance/features/ | grep -v 'tests/acceptance/features/bootstrap/'

for web (need improve): https://github.com/owncloud/web/pull/11323

dummy report:

QA Activity Report for August 2024

Date: 05.08.2024

Summary

Added lines to the .feature files: 391 Deleted lines in the .feature files: 191 Modified lines in the .feature files: 582

Detailed Changes

Commit: 9090ac3bac040cdd1a6d970f3fda4b342e936cc7

Author: Viktor Scharf scharf.vi@gmail.com Date: Thu Aug 1 16:05:14 2024 +0200

Added Scenarios:

Scenario Name: User logs in with valid credentials Steps Added: 5 Examples Added: 2

Deleted Scenarios:

Scenario Name: User tries to log in with invalid credentials Steps Deleted: 4

Changed Scenarios:

Scenario Name: User changes password successfully Steps Modified: 3 Examples Modified: 1

---

Commit: e23ebf776f5fe2250e8462e1846df0f75ccfdcbe

Author: Sawjan Gurung saw.jan.grg3e@gmail.com Date: Tue Aug 2 10:05:14 2024 +0200

Added Scenarios:

Scenario Name: Admin creates a new user Steps Added: 6 Examples Added: 3

Deleted Scenarios:

Scenario Name: User views their profile Steps Deleted: 3

Changed Scenarios:

Scenario Name: User updates profile information Steps Modified: 4 Examples Modified: 2

---

Commit: i7j8k9l

or report with more details like: QA_Activity_report.md

[ScharfViktor]

@micbar @dragotin I added dummy report to description. could you please check if you're satisfied with the format

[dragotin]

We would need a CSV-File ("Comma separated values") with this information, for example like that:

# Test-Type, Date, Tests Added, Tests Changed, Test Deleted, commit-ID (if avial.) 
API Test, 2024-07-23, 2, 0, 0, 54343453
UI Test, 2024-07-12, 3, 2,0, 4afe442s

Maybe we can generate that data either out of git log, or parse from the CI run? I dont know.

Given that we do track different test types in this file, I would for the beginning only track "tests", not steps etc. If, for example, a step is added to a scenario, that counts as "test changed".

@ScharfViktor what do you think, can that be generated?

[ScharfViktor]

@ScharfViktor what do you think, can that be generated?

thanks, I'll try to generate it using git log first

[ScharfViktor]

Creating a report for unitTest in the ocis repo has been challenging for me. I see that we have different types of unit tests and the way I've used before for other tests won't give good results.

As a simple solution I can get a count of all tests changed or added. but the result will be not exact:

sh:

# Get list of changed test
changed_test_files=$(git log --since="2024-08-01" --until="2024-08-31" --name-only --pretty=format: -- ocis-pkg/ services/ | grep '_test.go' | sort | uniq)

total_tests=0

for file in $changed_test_files; do
  path=$(dirname "$file")

  # run each test
  test_count=$(go test -json path | grep -c RUN)
  total_tests=$((total_tests + test_count))
done

echo "Total tests added or modified in the last month: $total_tests"

result I can add to csv file without commitId

vscharf:ocis scharfviktor$ sh test.sh 
Total tests added or modified in the last month: 306