Feedback on the new 'Flags' tab

[aj-codecov]

Thanks for dropping by! 👋

We've recently released a whole new area of the app focused on flags. If you're not seeing it yet, keep your eyes peeled as we're slowly rolling it out to our users.

• What do you think about the new Flags tab?
• What would you'd like to see improved?
• Has it posed any challenges or questions?
• How do you use it?
• How could the report better help your workflow?
• Any general thoughts you'd like to share! We greatly appreciate your time and thoughts - looking forward to hearing from you ❤

Codecov team

This issue is intended to share and collect feedback about the tool. If you have support needs or questions, please see our support page.

[austinpray-mixpanel]

There should be an easy way to delete flags from the flags tab.

All of our flags are carry forward:

flag_management:
  default_rules:
    carryforward: true

Which means we have lots of dangling flags left over from when we were configuring codecov for the first time. We experimented with different naming conventions and such. There's no way to clear these flags out, and I'm worried they will contribute to weird coverage numbers.

[guohui-gao]

There should be an easy way to delete flags from the flags tab.

All of our flags are carry forward:

flag_management:
  default_rules:
    carryforward: true

Which means we have lots of dangling flags left over from when we were configuring codecov for the first time. We experimented with different naming conventions and such. There's no way to clear these flags out, and I'm worried they will contribute to weird coverage numbers.

Agree +1

[aj-codecov]

We really appreciate the feedback! We've heard the need for clearing out older flags and are looking at it as work to be done in the next few months. We also have a couple other improvements to the flags tab that will allow you to search and select only a few flags you want to see at a time.

[mattlewis92]

Hi there,

We have a few pieces of feedback that I think would really improve the flags page:

1. Add a branch selector so you can see the flag coverage for a particular branch, right now it's an average of all flags across all branches which isn't really that helpful:
2. Show the date range start and end coverage percentages, instead of this average coverage over the selected time period:

Our use case is that we want the ability to track individual team coverage over time (where each team is a flag) on our monorepo. Right now because of the way that the flags report is averaged over time, across all PRs this severely limits the usefulness of it.

[FelixS90]

After enabling the feature, the status stays at "Pulling historical data" now for roughly half an hour - although we enabled it right after the first scan on the default branch.

What could be the reason? Let me know if you need more info.

[eliatcodecov]

@FelixS90 our historical data pull feature relies on an asynchronous task queue to function. This is essential since, in many cases, users have a a large amount of historical coverage data that we need to pull, aggregate, analyze, and store in a time series friendly format.

With this in mind, there are two reasons a historical data pull can take awhile:

1. The repo in question has lots of historical data (doesn't seem to be your case)
2. The historical data task queue is first in first out. So if there were other repos queued ahead of yours they would need to have been processed first before processing yours.

We generally try to autoscale processing for that queue to keep queue length under control, but it could have just been a particularly busy time in your case. Apologies for the delay, but is the feature working for that particular repository now?

[austinpray-mixpanel]

I get a consistent "There was a problem getting flags data" after a considerable wait time.

JS console output: app.codecov.io-1667315704716.log

[RobertBrunhage]

I don't understand the flags.

In the flag section it's stated 5.5%

But looking a the normal page and the folder that is tagged, that folder is 17.5%

My idea was to have each package have it's own coverage but they don't match in the flag section.

[Wayonb]

I am not seeing data for my configured flags 🤔

[anyweez]

I also see "There was a problem getting flags data" - it appears to work once every 10-15 attempts, but only after a lengthy delay.

From browser console:

Sentry Logger [log]: Event processor "InboundFilters" dropped event
main.ec98bddc.js:2 Sentry Logger [log]: Adding outcome: "event_processor:error"
main.ec98bddc.js:2 Sentry Logger [log]: An event processor returned null, will not send event.
script.js:1 Uncaught Error: It looks like your website URL has been changed. To ensure the proper functioning of your banner, update the registered URL on your CookieYes account (More > Site Settings) and reload this page to retry. If the issue persists, contact us at https://www.cookieyes.com/support.
    at script.js:1:86908
    at script.js:1:97140
    at script.js:1:97143
(anonymous) @ script.js:1
(anonymous) @ script.js:1
(anonymous) @ script.js:1
manifest.json:1 Manifest: Line: 1, column: 1, Syntax error.
main.ec98bddc.js:2 Sentry Logger [log]: Sending outcomes: [{…}]
flags:1 Access to fetch at 'https://api.codecov.io/graphql/gh' from origin 'https://app.codecov.io' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource. If an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled.
main.ec98bddc.js:2          POST https://api.codecov.io/graphql/gh net::ERR_FAILED 502

[aj-codecov]

Hi all, we are actively working on the issues causing the error states, when you encounter those, changing the time scale can oftentimes unblock you.

@RobertBrunhage I'd love to dig in further with you on your particular issue, can you give me a specific repo/flag (if it's public) so I can hopefully get you an answer?

[Wayonb]

Is this a known issue where some of the flags are empty?

[aj-codecov]

@Wayonb flags stick around once they have been created at any point in time, for example 'sdk-ptyhon' was likely a flag at one time, which was corrected to 'sdk-python'. In the case of the empties you see in your flags tab, is it possible each of those flags was uploaded to once and then never again? The lack of data makes me wonder if they were even uploaded to with a blank coverage report? It's hard to say exactly without more info, but those are the possible avenues I'm aware of that might cause this scenario.

[Wayonb]

@aj-codecov Thanks. I guess the only way to remove those is to reset all the data?

[sungam3r]

From https://github.com/codecov/Codecov-user-feedback/issues/27#issuecomment-1314018066 :

My idea was to have each package have it's own coverage

I thought so as well. Is that true for flags feature? My case is Xabaril/AspNetCore.Diagnostics.HealthChecks repo https://app.codecov.io/github/Xabaril/AspNetCore.Diagnostics.HealthChecks/flags with ~40 projects and I would like to track coverage independently for each project (package). Can I archive that by flags feature? I planned to add flags argument to each GitHub CI workflow. Note that in my case I run only those workflows for which some files were changes, i.e. if PR touches only Redis project then only Redis CI file run with only tests for Redis (and coverage data only for Redis), not all ~40 projects. As I understand in that case each time I upload totally different coverage results to codecov. Maybe I do upload totally wrong and it can be achived by other means, not flags?

[ascopes]

I've followed the instructions, but all I get is

Clicking the link to show the reported uploads shows all the coverage data, so no idea what is going on. I've had to disable this entirely now since it no longer reports any coverage at all when I use flags, which is a bit of a pain.

There is nothing to actually indicate what the issue is though, so for now I won't be able to make use of this, which is unfortunate as this feature would be really useful in my project for separating out integration and unit test coverage.

If there is a way to feed back errors more clearly on these pages, it would be greatly appreciated and of great use in the future.

https://app.codecov.io/gh/ascopes/java-compiler-testing/commit/0cb6c43a695b20c446a824c25b8095632b8eeed8

[aj-codecov]

@sungam3r I believe you've got the right idea, only thing I'd say is if you're not uploading all coverage every time you commit, you're going to want to use Carryforward flags as described here: https://docs.codecov.com/docs/carryforward-flags

[aj-codecov]

@ascopes In your first screenshot I can see you've attempted to tie two flags to one upload, Flags must be associated with only one upload to work properly, so you would need to do an upload of coverage for integration tests and an upload of coverage for unit tests where each is flagged with their corresponding label further reading here: https://docs.codecov.com/docs/flags#one-to-one-relationship-of-flags-to-uploads I'm not sure why we're flagging the upload as empty offhand, it could be because Codecov isn't recognizing the format, there was an error processing, or because the upload is actually empty, it's hard to say without a dive into the coverage report itself. Error messaging is a perpetual work in progress and this is certainly an area we aren't quite cutting it at the moment. If we can get to the root of what happened in your case I'm happy to get a relevant error message into our product pipeline.

[sungam3r]

@aj-codecov I have configured carryforward option for each project: https://github.com/Xabaril/AspNetCore.Diagnostics.HealthChecks/blob/master/.github/codecov.yml

Also I set individual flags in each CI pipeline like this one https://github.com/Xabaril/AspNetCore.Diagnostics.HealthChecks/blob/d4c751bdeaa363dec39bb30bd49cead817aae5f5/.github/workflows/healthchecks_applicationstatus_ci.yml#L64

but after the latest upload https://github.com/Xabaril/AspNetCore.Diagnostics.HealthChecks/pull/1616 I still see only coverage from only touched files and overall coverage dropped again after full report upload.

[ascopes]

@aj-codecov Hi, thanks for the response

@ascopes In your first screenshot I can see you've attempted to tie two flags to one upload, Flags must be associated with only one upload to work properly, ...

I was under the impression that the feature flags would handle this automatically since they consume specific paths related to specific flags. Is this not the case?

flag_management:
  default_rules:
    carryforward: true
    statuses:
      - type: project
        target: auto
        threshold: 1%
      - type: patch
        target: 90%

  individual_flags:
    # Unit testing coverage
    - name: unit
      carryforward: true
      paths:
        - '**/target/site/jacoco/unit/jacoco*.xml'
      statuses:
        - type: project
          target: auto
          threshold: 65%
        - type: patch
          target: 90%

    # Integration testing coverage
    - name: integration
      carryforward: true
      paths:
        - '**/target/site/jacoco/int/jacoco*.xml'
      statuses:
        - type: project
          target: auto
          threshold: 0%
        - type: patch
          target: 0%

[vigenere23]

My flags page is not loading the coverages:

Any idea what's happening?

Logs

```txt The connection to https://clientstream.launchdarkly.com/eval/60f9b3cf32beb9244b10f3d1/ was interrupted while the page was loading. main.ad642d24.js:2:550053 [LaunchDarkly] Error on stream connection: {"isTrusted":true}, will continue retrying every 1000 milliseconds. main.ad642d24.js:2:407351 Sentry Logger [error]: Error while triggering instrumentation handler. Type: fetch Name: Error: TypeError: o is undefined Et https://app.codecov.io/static/js/main.ad642d24.js:2 en https://app.codecov.io/static/js/main.ad642d24.js:2 m https://app.codecov.io/static/js/main.ad642d24.js:2 h https://app.codecov.io/static/js/main.ad642d24.js:2 main.ad642d24.js:2:414414 99204/s/ injectedScript:1:126 Sentry Logger [log]: Integration installed: InboundFilters main.ad642d24.js:2:414414 Sentry Logger [log]: Integration installed: FunctionToString main.ad642d24.js:2:414414 Sentry Logger [log]: Integration installed: TryCatch main.ad642d24.js:2:414414 Sentry Logger [log]: Integration installed: Breadcrumbs main.ad642d24.js:2:414414 Sentry Logger [log]: Global Handler attached: onerror main.ad642d24.js:2:414414 Sentry Logger [log]: Global Handler attached: onunhandledrejection main.ad642d24.js:2:414414 Sentry Logger [log]: Integration installed: GlobalHandlers main.ad642d24.js:2:414414 Sentry Logger [log]: Integration installed: LinkedErrors main.ad642d24.js:2:414414 Sentry Logger [log]: Integration installed: Dedupe main.ad642d24.js:2:414414 Sentry Logger [log]: Integration installed: HttpContext main.ad642d24.js:2:414414 Sentry Logger [log]: [Tracing] Starting pageload transaction on scope main.ad642d24.js:2:414414 Sentry Logger [log]: Setting idle transaction on scope. Span ID: 89231477b0e5b4a9 main.ad642d24.js:2:414414 Sentry Logger [log]: [Tracing] Discarding transaction because it's not included in the random sample (sampling rate = 0.2) main.ad642d24.js:2:414414 Sentry Logger [log]: Integration installed: BrowserTracing main.ad642d24.js:2:414414 Sentry Logger [log]: Integration installed: Replay main.ad642d24.js:2:414414 Sentry Logger [log]: [Tracing] Starting 'ui.react.mount' span on transaction '/gh/vigenere23/disma/flags' (89231477b0e5b4a9). main.ad642d24.js:2:414414 Sentry Logger [log]: [Tracing] Finishing 'ui.react.mount' span on transaction '/gh/vigenere23/disma/flags' (89231477b0e5b4a9). main.ad642d24.js:2:414414 Sentry Logger [log]: [Tracing] Starting 'http.client' span on transaction '/gh/vigenere23/disma/flags' (89231477b0e5b4a9). 2 main.ad642d24.js:2:414414 Sentry Logger [log]: [Tracing] Starting 'http.client' span on transaction '/gh/vigenere23/disma/flags' (89231477b0e5b4a9). 2 main.ad642d24.js:2:414414 Loading failed for the Githubissues.

Githubissues is a development platform for aggregating issues.