Feature Request: Performance Summary View Widget

soyacz commented 1 month ago

Problem:

In Argus View, identifying failed test results from the current week is challenging due to the large number of tests and runs. Users must also navigate into test run details to view failed result tables, making it difficult to grasp failures at a glance. Additionally, important tested versions can be buried in the results.

Proposed Solution:

A widget will be developed to simplify the view of weekly test results, emphasizing failures and presenting failure details upfront, without requiring the user to open test run details. It will also provide better visibility into both passed and failed tests.

Widget Features:

Sectioned View by SUT Version:
- The widget will be divided into sections based on the System Under Test (SUT) version, such as Scylla, Scylla Manager, or Scylla Driver.
- Each section will represent a version tested during the current week and will display test run results segregated into passed and failed lists.
Highlighting Missed or Unstarted Tests:
- Tests that were not executed within the last 7 days or failed to start (e.g., due to infrastructure issues like AWS capacity) and have no SUT version assigned will be highlighted at the top of the widget.
- These tests will be treated as failures and listed separately, outside of any SUT version section.
Test Run Lists:
- Each SUT Version Section will contain test runs segregated into two lists:
  - Failed List: Displays all failed test runs and their failed result tables.
  - Passed List: Displays all successfully passed test runs.
- Each test run in both lists will be clickable, allowing users to navigate to the detailed test run page if needed.
- Each test run entry will show:
  - values of selected metrics in tabular form, with columns: result table name, and rest will be selected metrics names
  - The count of error events associated with that test run.
  - Failed result tables will be displayed (collapsed by default for passed tests, visible for failed tests). of the error count and the failed result tables (collapsed by default).
Run Investigation Status:
- Any test runs marked as "IGNORED" in their investigation status will be excluded from the analysis to avoid clutter.
- Test runs that have failed result tables or error events but are manually marked as passed will still be listed in the passed list, with an indication
Widget Configuration:
- Users can configure the widget by:
  - Setting a custom widget name.
  - Selecting which tests to monitor from the available test groups.
  - selected metric values (e.g. p99's, duration of repair etc.)
- Widgets can be edited at any time to add/remove tests or change the name.

Out of Scope:

Looking back at results from previous weeks.
Showing detailed error events beyond the error count.
Configurable monitoring windows (default is 7 days).

refs: https://github.com/scylladb/qa-tasks/issues/1784

soyacz commented 1 month ago

@fruch @roydahan please let me know if I got it right and my proposed solution fits our needs.

soyacz commented 1 day ago

summary-widget.html.zip @fruch please take a look basic layout (autogenerated, mostly showing just layout as how it will look in detail will change).

fruch commented 1 day ago

summary-widget.html.zip @fruch please take a look basic layout (autogenerated, mostly showing just layout as how it will look in detail will change).

looks like a good start, I would rather see it with less clicks failures should be open to begin with.

It's kind of missing showing all the types (read/mixed/write) side by side.

@roydahan @juliayakovlev, care to take a look ?

soyacz commented 1 day ago

summary-widget.html.zip @fruch please take a look basic layout (autogenerated, mostly showing just layout as how it will look in detail will change).

looks like a good start, I would rather see it with less clicks failures should be open to begin with.

Yes, I was planning it.

It's kind of missing showing all the types (read/mixed/write) side by side.

Ok, I'll try that as I need to see if it will fit into screen width. Still I'm not yet sure how to handle period for summary - hardcode 1 week, make it configurable, or just most recent jobs (which will be harder in case of partial rerun e.g. repeat mixed test). Also I wonder if shouldn't support going back to previous summaries.

@roydahan @juliayakovlev, care to take a look ?

fruch commented 1 day ago

summary-widget.html.zip @fruch please take a look basic layout (autogenerated, mostly showing just layout as how it will look in detail will change).

looks like a good start, I would rather see it with less clicks failures should be open to begin with.

Yes, I was planning it.

It's kind of missing showing all the types (read/mixed/write) side by side.

Ok, I'll try that as I need to see if it will fit into screen width. Still I'm not yet sure how to handle period for summary - hardcode 1 week, make it configurable, or just most recent jobs (which will be harder in case of partial rerun e.g. repeat mixed test). Also I wonder if shouldn't support going back to previous summaries.

If we continue with direction of running release ontop the same jobs, we'll need also to filter only dev (or specific releases)

as for going back, it's something we are doing quite a lot with current google doc, one can argue the graph can help with that bit I don't know if we want to take off the table yet...

@roydahan @juliayakovlev, care to take a look ?

soyacz commented 20 hours ago

If we continue with direction of running release ontop the same jobs, we'll need also to filter only dev (or specific releases)

as for going back, it's something we are doing quite a lot with current google doc, one can argue the graph can help with that bit I don't know if we want to take off the table yet...

So, after rethinking, idea is to add summary widget per release (upon adding widget configure which release to monitor - dev, enterprise-dev, 2024.2, 6.2 etc.). Additionally, user will be able to scroll to previous tested versions back and forth (right-left arrows next to version, build date)

WDYT?

scylladb / argus