CI post functions for hardware testing

[hakehuang]

from Testing CI requirements, we expect the CI to upload two things to a result repo:

1. the SHA number to a text file.
2. the qemu board testing result.

then result repo can used by all community member to create a real board testing.

see below red parts of proposed board testing process.

[nashif]

An updated workflow proposal that utilizes AWS S3 for data collection instead of a git repo. This is easier to implement and provides one single place for all test results and reports.

We will still use the test_results repo to collect data from members and community, but the central place for all raw results will be in AWS S3. Once all data is pushed to testrail, we can also put HTML test reports and other collateral in AWS S3.

Pushing data to git by CI is much more complex and can come with many conflicts.

[xuhao8210]

@nashif -- Could you please help review my few questions below ? Thanks in advance.

1. Could you please introduce what is the result bucket used for?
2. After the test results is pushed to testrail (e.g. once one day) , do we need the delete the test results in the Result Repo ?
3. About the description: “we will put HTML test reports and other collateral in AWS S3.” , will it also be implemented in github’s action to use another workflow ? And it seems that there is no the related API by Test Reail provided to export the HTML test reports in qm repo, could you please help double confirm it ? Thanks again.

[nashif]

1. Could you please introduce what is the result bucket used for?

This is where all test results will end up, raw results and reports generated from testrail for example. This is the same as the "archive on AWS S3" box.

After the test results is pushed to testrail (e.g. once one day) , do we need the delete the test results in the Result Repo ?

No.

3\. About the description: “we will put HTML test reports and other collateral in AWS S3.” , will it also be implemented in github’s action to use another workflow ? And it seems that there is no the related API by Test Reail provided to export the HTML test reports in qm repo, could you please help double confirm it ?   Thanks again.

Did you try http://docs.gurock.com/testrail-api2/reference-reports We need to find a way to download reports and archive them, make them available outside of testrail.

[hakehuang]

update user case diagram

[hakehuang]

@xuhao8210 I create two issues https://github.com/hakehuang/zephyr_ci_repo/issues/2, maybe we can use this as a trial test repo, and I will try to create GH hooks for below steps:

1. log latest hash number in a text file
2. upload qemu board test result by date

[nashif]

• AWS S3 bucket created
• Hosting on top of bucket created (https://testing.zephyrproject.org)
• Experimental CI job now uploading results and changing the versions file in the bucket, needs additional optimizations still, versions file is a json file: https://testing.zephyrproject.org/daily_tests/versions.json

next steps:

• synchronize matrix to build one commit
• adapt sanitycheck to support only running a group or simulators/emulators in CI

considerations:

• All results will be in one file for all platforms running in CI, not 1 per platform.

[hakehuang]

https://github.com/hakehuang/zephyr_ci_repo/actions/runs/84702849

@nashif I meet an issue when I try to create a 3 qemu builds. can we apply for more space?

[hakehuang]

• AWS S3 bucket created
• Hosting on top of bucket created (https://testing.zephyrproject.org)
• Experimental CI job now uploading results and changing the versions file in the bucket, needs additional optimizations still, versions file is a json file: https://testing.zephyrproject.org/daily_tests/versions.json

next steps:

• synchronize matrix to build one commit
• adapt sanitycheck to support only running a group or simulators/emulators in CI

considerations:

• All results will be in one file for all platforms running in CI, not 1 per platform.

I check the json file

[
  "zephyr-v2.2.0-1791-g8d984b336e",
  "zephyr-v2.2.0-1794-g670358c039",
  "zephyr-v2.2.0-1797-ga7e243bea4",
  "zephyr-v2.2.0-1803-g6e3f86876e"
]

do we want the list keep growing? Besides, what is the API to upload build report and this version json file?

[nashif]

@nashif I meet an issue when I try to create a 3 qemu builds. can we apply for more space?

not sure what are you trying to do there, we do not need to run sanitycheck in actions, this is currently not tested and not supported and requires alot of setup and resources. We will run sanitycheck as before in shippable, this part is step 1 and I am taking caring of. We will only use GH actions for all other interactions and processing of results and uploading them to testrail.

[nashif]

do we want the list keep growing? Besides, what is the API to upload build report and this version json file?

I do not see any reason why not, this is the manifest of all versions. This is preliminary and we could tag the latest version in the json file.

Uploading results and changing this file and other files will all happen using AWS S3 through the GH action. So it is normal file operations.

[nashif]

@hakehuang see this https://github.com/zephyrproject-rtos/test_results, this is the job creating the version file and uploading results from CI.

[hakehuang]

@nashif I meet an issue when I try to create a 3 qemu builds. can we apply for more space?

not sure what are you trying to do there, we do not need to run sanitycheck in actions, this is currently not tested and not supported and requires alot of setup and resources. We will run sanitycheck as before in shippable, this part is step 1 and I am taking caring of. We will only use GH actions for all other interactions and processing of results and uploading them to testrail.

I understand now, Thanks