Open aj-codecov opened 2 years ago
There should be an easy way to delete flags from the flags tab.
All of our flags are carry forward:
flag_management:
default_rules:
carryforward: true
Which means we have lots of dangling flags left over from when we were configuring codecov for the first time. We experimented with different naming conventions and such. There's no way to clear these flags out, and I'm worried they will contribute to weird coverage numbers.
There should be an easy way to delete flags from the flags tab.
All of our flags are carry forward:
flag_management: default_rules: carryforward: true
Which means we have lots of dangling flags left over from when we were configuring codecov for the first time. We experimented with different naming conventions and such. There's no way to clear these flags out, and I'm worried they will contribute to weird coverage numbers.
Agree +1
We really appreciate the feedback! We've heard the need for clearing out older flags and are looking at it as work to be done in the next few months. We also have a couple other improvements to the flags tab that will allow you to search and select only a few flags you want to see at a time.
Hi there,
We have a few pieces of feedback that I think would really improve the flags page:
Our use case is that we want the ability to track individual team coverage over time (where each team is a flag) on our monorepo. Right now because of the way that the flags report is averaged over time, across all PRs this severely limits the usefulness of it.
After enabling the feature, the status stays at "Pulling historical data" now for roughly half an hour - although we enabled it right after the first scan on the default branch.
What could be the reason? Let me know if you need more info.
@FelixS90 our historical data pull feature relies on an asynchronous task queue to function. This is essential since, in many cases, users have a a large amount of historical coverage data that we need to pull, aggregate, analyze, and store in a time series friendly format.
With this in mind, there are two reasons a historical data pull can take awhile:
We generally try to autoscale processing for that queue to keep queue length under control, but it could have just been a particularly busy time in your case. Apologies for the delay, but is the feature working for that particular repository now?
I get a consistent "There was a problem getting flags data" after a considerable wait time.
JS console output: app.codecov.io-1667315704716.log
I don't understand the flags.
In the flag section it's stated 5.5%
But looking a the normal page and the folder that is tagged, that folder is 17.5%
My idea was to have each package have it's own coverage but they don't match in the flag section.
I am not seeing data for my configured flags 🤔
I also see "There was a problem getting flags data" - it appears to work once every 10-15 attempts, but only after a lengthy delay.
From browser console:
Sentry Logger [log]: Event processor "InboundFilters" dropped event
main.ec98bddc.js:2 Sentry Logger [log]: Adding outcome: "event_processor:error"
main.ec98bddc.js:2 Sentry Logger [log]: An event processor returned null, will not send event.
script.js:1 Uncaught Error: It looks like your website URL has been changed. To ensure the proper functioning of your banner, update the registered URL on your CookieYes account (More > Site Settings) and reload this page to retry. If the issue persists, contact us at https://www.cookieyes.com/support.
at script.js:1:86908
at script.js:1:97140
at script.js:1:97143
(anonymous) @ script.js:1
(anonymous) @ script.js:1
(anonymous) @ script.js:1
manifest.json:1 Manifest: Line: 1, column: 1, Syntax error.
main.ec98bddc.js:2 Sentry Logger [log]: Sending outcomes: [{…}]
flags:1 Access to fetch at 'https://api.codecov.io/graphql/gh' from origin 'https://app.codecov.io' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource. If an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled.
main.ec98bddc.js:2 POST https://api.codecov.io/graphql/gh net::ERR_FAILED 502
Hi all, we are actively working on the issues causing the error states, when you encounter those, changing the time scale can oftentimes unblock you.
@RobertBrunhage I'd love to dig in further with you on your particular issue, can you give me a specific repo/flag (if it's public) so I can hopefully get you an answer?
Is this a known issue where some of the flags are empty?
@Wayonb flags stick around once they have been created at any point in time, for example 'sdk-ptyhon' was likely a flag at one time, which was corrected to 'sdk-python'. In the case of the empties you see in your flags tab, is it possible each of those flags was uploaded to once and then never again? The lack of data makes me wonder if they were even uploaded to with a blank coverage report? It's hard to say exactly without more info, but those are the possible avenues I'm aware of that might cause this scenario.
@aj-codecov Thanks. I guess the only way to remove those is to reset all the data?
From https://github.com/codecov/Codecov-user-feedback/issues/27#issuecomment-1314018066 :
My idea was to have each package have it's own coverage
I thought so as well. Is that true for flags feature? My case is Xabaril/AspNetCore.Diagnostics.HealthChecks repo https://app.codecov.io/github/Xabaril/AspNetCore.Diagnostics.HealthChecks/flags with ~40 projects and I would like to track coverage independently for each project (package). Can I archive that by flags feature? I planned to add flags
argument to each GitHub CI workflow. Note that in my case I run only those workflows for which some files were changes, i.e. if PR touches only Redis project then only Redis CI file run with only tests for Redis (and coverage data only for Redis), not all ~40 projects. As I understand in that case each time I upload totally different coverage results to codecov. Maybe I do upload totally wrong and it can be achived by other means, not flags?
I've followed the instructions, but all I get is
Clicking the link to show the reported uploads shows all the coverage data, so no idea what is going on. I've had to disable this entirely now since it no longer reports any coverage at all when I use flags, which is a bit of a pain.
There is nothing to actually indicate what the issue is though, so for now I won't be able to make use of this, which is unfortunate as this feature would be really useful in my project for separating out integration and unit test coverage.
If there is a way to feed back errors more clearly on these pages, it would be greatly appreciated and of great use in the future.
@sungam3r I believe you've got the right idea, only thing I'd say is if you're not uploading all coverage every time you commit, you're going to want to use Carryforward flags as described here: https://docs.codecov.com/docs/carryforward-flags
@ascopes In your first screenshot I can see you've attempted to tie two flags to one upload, Flags must be associated with only one upload to work properly, so you would need to do an upload of coverage for integration tests and an upload of coverage for unit tests where each is flagged with their corresponding label further reading here: https://docs.codecov.com/docs/flags#one-to-one-relationship-of-flags-to-uploads I'm not sure why we're flagging the upload as empty offhand, it could be because Codecov isn't recognizing the format, there was an error processing, or because the upload is actually empty, it's hard to say without a dive into the coverage report itself. Error messaging is a perpetual work in progress and this is certainly an area we aren't quite cutting it at the moment. If we can get to the root of what happened in your case I'm happy to get a relevant error message into our product pipeline.
@aj-codecov I have configured carryforward
option for each project: https://github.com/Xabaril/AspNetCore.Diagnostics.HealthChecks/blob/master/.github/codecov.yml
Also I set individual flags
in each CI pipeline like this one
https://github.com/Xabaril/AspNetCore.Diagnostics.HealthChecks/blob/d4c751bdeaa363dec39bb30bd49cead817aae5f5/.github/workflows/healthchecks_applicationstatus_ci.yml#L64
but after the latest upload https://github.com/Xabaril/AspNetCore.Diagnostics.HealthChecks/pull/1616 I still see only coverage from only touched files and overall coverage dropped again after full report upload.
@aj-codecov Hi, thanks for the response
@ascopes In your first screenshot I can see you've attempted to tie two flags to one upload, Flags must be associated with only one upload to work properly, ...
I was under the impression that the feature flags would handle this automatically since they consume specific paths related to specific flags. Is this not the case?
flag_management:
default_rules:
carryforward: true
statuses:
- type: project
target: auto
threshold: 1%
- type: patch
target: 90%
individual_flags:
# Unit testing coverage
- name: unit
carryforward: true
paths:
- '**/target/site/jacoco/unit/jacoco*.xml'
statuses:
- type: project
target: auto
threshold: 65%
- type: patch
target: 90%
# Integration testing coverage
- name: integration
carryforward: true
paths:
- '**/target/site/jacoco/int/jacoco*.xml'
statuses:
- type: project
target: auto
threshold: 0%
- type: patch
target: 0%
My flags page is not loading the coverages:
Any idea what's happening?
Thanks for dropping by! đź‘‹
We've recently released a whole new area of the app focused on flags. If you're not seeing it yet, keep your eyes peeled as we're slowly rolling it out to our users.
Codecov team
This issue is intended to share and collect feedback about the tool. If you have support needs or questions, please see our support page.