search

packit / packit-service

Packit provided as a service
https://packit.dev
MIT License
34 stars 48 forks source link

Packit API enhancement - make /testing-farm/results parametrizable, expose ref #2221

Open danmyway opened 12 months ago

danmyway commented 12 months ago

Description

Gathering data for test results from the Testing Farm is very time demanding and tedious with growing number of pipelines. One needs to go over each check, click the respective hyperlink and scroll through the results. We have our own way to easily parse the results from the result page/artifact URL, however getting to the pipeline_id and or URL is the pain point.

One way to get to the result page/pipeline ID "faster" is to parse the Packit's API /testing-farm/results and/or /projects/{forge}/{namespace}/{repo_name}/prs namespace.

  1. The issue with the /testing-farm/results is, that it contains all the projects' testing-farm requests handled by packit with no way to parametrize the query. Being able to parametrize the query to narrow the results to a specific project would speed up the data gathering immensely.

  2. The other option, however not exclusive to the testing-farm namespace enhancement is exposing ref and possibly any type of timestamp for the /projects/{forge}/{namespace}/{repo_name}/prs response body. Specifically inside the tests field.

Benefit

If proven working, we could provide the artifact URL gathering and reporting functionality to packit itself. This could help to resolve https://github.com/packit/packit-service/issues/2205 Additionally, there could be /packit report feature provided in the future to report the test results by Packit in a comment.

❯ tesar report -f tesar_latest_jobs_20230920143852 
Reporting for the requested tasks:
AlmaLinux OS 8.6.20220901 x86_64 tier0/non-destructive COMPLETE
Rocky-8-EC2-Base-8.8-20230518.0.x86_64 tier0/non-destructive COMPLETE
+--------------------------------------+----------------------------------------+-----------------------------+--------+
| UUID                                 | Target                                 | Test Plan                   | Result |
+--------------------------------------+----------------------------------------+-----------------------------+--------+
| 2ae608e4-3456-47ca-93b5-1dbf2c420b97 | AlmaLinux OS 8.6.20220901 x86_64       |                             |        |
|                                      |                                        | plans/tier0/non-destructive | FAILED |
| 87ae5111-27ea-4718-9c5a-ab1e03e01718 | Rocky-8-EC2-Base-8.8-20230518.0.x86_64 |                             |        |
|                                      |                                        | plans/tier0/non-destructive | FAILED |
+--------------------------------------+----------------------------------------+-----------------------------+--------+
❯ tesar report -f tesar_latest_jobs_20230920143852 -l2 --short
Reporting for the requested tasks:
AlmaLinux OS 8.6.20220901 x86_64 tier0/non-destructive COMPLETE
Rocky-8-EC2-Base-8.8-20230518.0.x86_64 tier0/non-destructive COMPLETE
+--------------------------------------+----------------------------------------+-----------------+---------------------------------------------+--------+
| UUID                                 | Target                                 | Test Plan       | Test Case                                   | Result |
+--------------------------------------+----------------------------------------+-----------------+---------------------------------------------+--------+
| 2ae608e4-3456-47ca-93b5-1dbf2c420b97 | AlmaLinux OS 8.6.20220901 x86_64       |                 |                                             |        |
|                                      |                                        | non-destructive |                                             | FAILED |
|                                      |                                        |                 | simultaneous_runs                           | ERROR  |
|                                      |                                        |                 | convert_successful_report                   | ERROR  |
|                                      |                                        |                 | failures_and_skips_in_report                | ERROR  |
|                                      |                                        |                 | successful_report                           | ERROR  |
|                                      |                                        |                 | clean_cache                                 | ERROR  |
|                                      |                                        |                 | c2r_version_latest_or_newer                 | ERROR  |
|                                      |                                        |                 | c2r_version_older_no_envar                  | ERROR  |
|                                      |                                        |                 | c2r_version_older_with_envar                | ERROR  |
|                                      |                                        |                 | data_collection_acknowledgement             | ERROR  |
|                                      |                                        |                 | disable_data_collection                     | ERROR  |
|                                      |                                        |                 | incomplete_rollback_in_analyze              | ERROR  |
|                                      |                                        |                 | log_file_exists                             | ERROR  |
|                                      |                                        |                 | log_rhsm_error                              | ERROR  |
|                                      |                                        |                 | manpage                                     | ERROR  |
|                                      |                                        |                 | root_privileges                             | ERROR  |
|                                      |                                        |                 | smoke                                       | ERROR  |
|                                      |                                        |                 | variant_message                             | ERROR  |
|                                      |                                        |                 | config_cli_priority                         | ERROR  |
|                                      |                                        |                 | config_custom_path_custom_filename          | ERROR  |
|                                      |                                        |                 | config_custom_path_standard_filename        | ERROR  |
|                                      |                                        |                 | config_password_file_priority               | ERROR  |
|                                      |                                        |                 | config_standard_paths_priority              | ERROR  |
|                                      |                                        |                 | config_standard_paths_priority_diff_methods | ERROR  |
|                                      |                                        |                 | custom_invalid_repo_provided                | ERROR  |
|                                      |                                        |                 | custom_valid_repo_provided                  | ERROR  |
|                                      |                                        |                 | passing_activation_key_to_submgr            | ERROR  |
|                                      |                                        |                 | passing_password_to_submgr                  | ERROR  |
|                                      |                                        |                 | available_connection                        | ERROR  |
|                                      |                                        |                 | unavailable_connection                      | ERROR  |
|                                      |                                        |                 | custom_kernel                               | ERROR  |
|                                      |                                        |                 | failed_repoquery                            | ERROR  |
|                                      |                                        |                 | non_latest_kernel_error                     | ERROR  |
|                                      |                                        |                 | yum_exclude_kernel                          | ERROR  |
|                                      |                                        |                 | custom_module_loaded                        | ERROR  |
|                                      |                                        |                 | custom_module_not_loaded                    | ERROR  |
|                                      |                                        |                 | unsupported_kmod_with_envar                 | ERROR  |
|                                      |                                        |                 | force_loaded_kmod                           | ERROR  |
|                                      |                                        |                 | tainted_kernel                              | ERROR  |
|                                      |                                        |                 | logfile_starts_with_command                 | ERROR  |
|                                      |                                        |                 | modified_releasever_in_configs              | ERROR  |
|                                      |                                        |                 | modified_releasever_to_unknown_release      | ERROR  |
|                                      |                                        |                 | packages_untracked_graceful_rollback        | ERROR  |
|                                      |                                        |                 | rhsm_cleanup                                | ERROR  |
|                                      |                                        |                 | registration_start                          | ERROR  |
|                                      |                                        |                 | registration_success                        | ERROR  |
|                                      |                                        |                 | test_missing_credentials_rollback           | ERROR  |
|                                      |                                        |                 | package_download_error                      | ERROR  |
|                                      |                                        |                 | transaction_validation_error                | ERROR  |
|                                      |                                        |                 | missing_system_release                      | ERROR  |
|                                      |                                        |                 | backup_os_release_no_envar                  | ERROR  |
|                                      |                                        |                 | backup_os_release_with_envar                | ERROR  |
|                                      |                                        |                 | unsuccessful_satellite_registration         | ERROR  |
+--------------------------------------+----------------------------------------+-----------------+---------------------------------------------+--------+

Importance

high It's not blocking us to use Packit per se, but it's faster to query for results if we use our wrapper to schedule test jobs and parse the resulting request pipeline IDs.

What is the impacted category (job)?

Testing Farm tests

Workaround

Participation

lbarcziova commented 11 months ago

hi @danmyway ! Thanks for the issue. I see this could improve your situation a lot. The only thing that might cause trouble is our current DB architecture, with more complicated queries we have bumped into performance issues previously.

You can see the current DB models architecture here.

As for the 1.option, how do you propose the parametrizing? Would ref be enough here as well?

As for the second one, if we would do this in general (not only for /prs), this could enable some improvements for dashboard as well (somehow related are https://github.com/packit/dashboard/issues/196 and https://github.com/packit/dashboard/issues/257). However, we will need to have a closer look how demanding these queries would be.

lbarcziova commented 10 months ago

hi @danmyway ! What is the status of this issue for you and its current importance?

danmyway commented 7 months ago

Hi @lbarcziova, I'm sorry for the unresponsiveness, I have somehow missed this. Given there is no result retention in the GitHub GUI and getting to the results through the Packit Dashboard sifting through dozens of PRs is very time-consuming, I would say it is pretty important to me. We have a tooling ready, which could parse the testing farm results based on the request UUID, the data leading to the UUIDs exist, but there is no easy way to connect these endpoints to get the desired results nowadays.