Ability to group flags and select groups of flags in the UI

[vlad-ko]

What product do you want to improve? Codecov App

Is your feature request related to a problem? Please describe. This is a request from our customer.

The problem this solves is to be able to select a grouping of flags in one click, which would represent a specific setup. For example we have functional and performance tests and we'd never want to mix these numbers together.

Flags passed: functional_editor, functional_native, performance_editor, performance_native, etc Currently you need to select all the individual flags to get the 'functional' overall, and all the 'performance' ones respectively. Group definition:

• functional (functional_editor, functional_native, etc)
• performance (performance_editor, performance_native, etc)

Additional context We use the term functional tests for tests that check the correctness of the code - like a unit test for instance.

Performance tests test how fast the code executes. These tests pass/fail also store a timing number like 0.3ms for instance. This performance data is used to determine if our code is getting slower or faster over time - which is important for our internal product.

It's possible a performance test might cause code to run that we don't have covered by functional tests. However the performance test isn't testing correctness, just how fast we can execute some operations.

When we look at the overall test coverage in CodeCov, it doesn't make sense for us to add the functional and performance test information together. So if we look at our code base and it has 60% test coverage really we'd only want that number to be talking about functional test coverage. Then we'd want some method to switch view to see how much of our code has performance tests.

Summary To add to the above, I see this accessible in the url like:

https://codecov.com/ghe/org/repo?flags_group=functional https://codecov.com/ghe/org/repo?flags_group=performance

And ideally the landing page of https://codecov.com/org/repo/module can be configured to be https://codecov.com/ghe/org/repo?flags_group=functional

[aj-codecov]

@vlad-ko Not sure I'm wrapping my head around this one - it sounds like they could maybe add a flag to each relevant upload that might apply to -performance even if it's uploading to two flags and thus have what they're looking for as far as an aggregate flag. Could even call it something like Functional (aggregate).

[vlad-ko]

@aj-codecov I think it would be best to speak with them a bit more to get additional details about this request.

Generally speaking, we've had a few customers now ask about some sort of flag hierarchy or intersection/matrix of flags, where to two or more flags could report up to one "main" (or group) flag. But, imho, this would require changes to the underlying architecture, UI and overall calculations, so not an easy request by any means. I would try to explore this step by step, for example, roll up multiple flag coverage into one "report"

[nikosatwork]

@aj-codecov

it sounds like they could maybe add a flag to each relevant upload that might apply to -performance even if it's uploading to two flags and thus have what they're looking for as far as an aggregate flag.

I agree that 2 flags could be passed in each upload, one specific to the current upload and one to aggregiate all the similar flags i.e.

./codecov -t <token> 
    -f coverage.xml -F performance_native -F performance

./codecov -t <token> 
    -f coverage.xml -F performance_runtime -F performance

In which case you would expect that selecting Flag: performance in the UI you would get an aggregation of both performance_native and performance_runtime

However, Codecov promotes an one-to-one relationship of Flags and specifying multiple flags for a single report upload can result in erroneous coverage. More informaton here.

[nikosatwork]

Therefore, a trivial solution is to introduce a Flags group, which is essentially a shortcut to select multiple flags in the UI

So in the above example we'd have

./codecov -t <token> 
    -f coverage.xml -F performance_native

./codecov -t <token> 
    -f coverage.xml -F performance_runtime

In codecov.yml the group could be specified like:

flag_management:
  flags_group:
    - name: performance
    - flags:
      - performance_native
      - performance_runtime

And in the UI there would be a new performance flag entry, both in the dropdowns and the flags tab, that would act as if you had selected both performance_native and performance_runtime

[Gguidini]

@nikosatwork would components sufice for this use case of having a "super-group" of flags?

In the component definition you can pass in flag regexes to group them together (seems like what you intend to do).

For example in the YAML you can define a component to aggregate all the flags that start with performance-.*.

component_management:
  individual_components:
    - component_id: performance_grouping
       name: performance
       flag_regexes:
          - "performance-.*"
       statuses:
          - type: "patch" 

ps.: I don't know what are the currently capabilities of visualizing components in the UI. But you can set custom statuses for them at least

However, Codecov promotes an one-to-one relationship of Flags and specifying multiple flags for a single report upload can result in erroneous coverage. More informaton here.

Yes, and components were introduced exactly to have a more flexible grouping structurethat is not directly related to uploads.

[vlad-ko]

While setting up components is a good option, it won't always be a good solution to the grouping of flags as requested above. Some customers define components for other purposes, so while having many components for one goal it would detrimental to create another set just to group flags.

It creates a tight coupling between the two concepts and ideally there should be a separation of concerns between the two.

Note, the goal isn't to assign multiple flags to one report, rather it is to have a similar feature in the flag config as is offered by the components' flag_regexes