search

status-im / status-go

The Status module that consumes go-ethereum
https://status.im
Mozilla Public License 2.0
728 stars 249 forks source link

Finalize Codecov integration #5963

Open igor-sirotin opened 4 weeks ago

igor-sirotin commented 4 weeks ago

  1. Ensure all PRs have codecov status checks Looks like when 1 test failed, codecov doesn't report the status checks to the PR. Despite that we rerun the test. It's not always like that though. Maybe because of the flaky test detection Codecov feature.

    One of the ways to fix it could be remove test results report to Codecov, keep only the coverage.

  2. Configure Codecov to satisfy our needs.

    • Set threshold for project coverage
    • Set minimum for the patch coverage

  3. Remove codeclimate integration.

igor-sirotin commented 4 weeks ago

I did some investigation of the opened PRs. Need to filter by date, might be some old PRs when status checks were not set up yet.

PR Created at Last commit Date OK Commit checks Comment
#5964 18.10.2024 18.10.2024 ✅ ✅ ✅
#5962 17.10.2024 18.10.2024 ✅ ❌ ✅
#5961 17.10.2024 17.10.2024 ✅ ✅ ✅
#5960 17.10.2024 17.10.2024 ✅ ✅ ⚠️ (4) (flaky: false)
#5959 17.10.2024 07.10.2024 ✅ ✅ ✅
#5958 17.10.2024 11.10.2024 ✅ ✅ ✅
#5957 17.10.2024 17.10.2024 ✅ ❌ ⚠️ (1) (flaky: true)
#5956 16.10.2024 16.10.2024 ✅ ✅ ⚠️ (1) (flaky: true)
#5955 16.10.2024 16.10.2024 ✅ ❌ ❌
#5953 16.10.2024 17.10.2024 ✅ ❌ ✅
#5952 16.10.2024 16.10.2024 ✅ ❌ ✅
#5950 16.10.2024 16.10.2024 ✅ ✅ ✅
#5948 15.10.2024 15.10.2024 ✅ ✅ ✅
#5947 15.10.2024 15.10.2024 ✅ ❌ ⚠️ (1) (flaky: true)
#5946 15.10.2024 17.10.2024 ✅ ❌ ❌
#5945 15.10.2024 15.10.2024 ✅ ❌ ⚠️ (1) (flaky: false)
#5943 14.10.2024 17.10.2024 ✅ ✅ ✅
#5941 14.10.2024 14.10.2024 ✅ ❌ ✅
#5939 11.10.2024 11.10.2024 ✅ ✅ ✅
#5930 09.10.2024 09.09.2024 ✅ ✅ ❌
#5929 08.10.2024 08.10.2024 ✅ ✅ ✅
#5925 07.10.2024 07.10.2024 ✅ ✅ ✅
#5907 03.10.2024 09.10.2024 ✅ ❌ ❌
#5899 01.10.2024 19.09.2024 ❌ ✅ ❌
#5885 27.09.2024 27.09.2024 ❌ ❌ ❌
#5877 25.09.2024 03.10.2024 ✅ ❌ ❌
#5864 23.09.2024 17.10.2024 ✅ ✅ ✅
#5859 20.09.2024 19.09.2024 ❌ ❌ ❌
#5857 19.09.2024 19.09.2024 ❌ ❌ ✅
#5840 16.09.2024 18.10.2024 ✅ ❌ ⚠️ (1) (flaky: false)
Code ```go package main import ( "context" "fmt" "log" "github.com/google/go-github/v48/github" "golang.org/x/oauth2" "strings" "time" "os" ) func main() { ctx := context.Background() ghToken, ok := os.LookupEnv("GITHUB_TOKEN") if !ok { log.Fatalf("GITHUB_TOKEN not set") } ts := oauth2.StaticTokenSource( &oauth2.Token{AccessToken: ghToken}, ) tc := oauth2.NewClient(ctx, ts) client := github.NewClient(tc) owner := "status-im" // Replace with your repository owner repo := "status-go" // Replace with your repository name thresholdDate, _ := time.Parse("02.01.2006", "03.10.2024") // Get open pull requests prs, _, err := client.PullRequests.List(ctx, owner, repo, &github.PullRequestListOptions{State: "open"}) if err != nil { log.Fatalf("Error fetching pull requests: %v", err) } fmt.Printf("| PR | Created at | Last commit | Date OK | Commit checks | Comment |\n") fmt.Printf("|-|-|-|-|-|-|\n") //skip := true for _, pr := range prs { //if skip { // if *pr.Number == 5943 { // skip = false // } // continue //} prDate := pr.GetCreatedAt() // Check the date of the latest commit in the PR latestCommit, _, err := client.Repositories.GetCommit(ctx, owner, repo, pr.Head.GetSHA(), nil) if err != nil { log.Fatalf("Error fetching the latest commit: %v", err) } commitDate := latestCommit.Commit.Author.GetDate() tooOld := commitDate.Before(thresholdDate) && prDate.Before(thresholdDate) prComments, _, err := client.Issues.ListComments(ctx, owner, repo, *pr.Number, nil) if err != nil { log.Fatalf("Error fetching PR comments: %v", err) } var codecovComment *github.IssueComment var codecovChecks []string // Check for codecov comments for _, comment := range prComments { if comment.User != nil && comment.User.GetLogin() == "codecov[bot]" { codecovComment = comment break } } checkRuns, _, err := client.Checks.ListCheckRunsForRef(ctx, owner, repo, *pr.Head.SHA, nil) if err != nil { log.Fatalf("Error fetching combined check runs: %v", err) } for _, run := range checkRuns.CheckRuns { if run.Name == nil { continue } if !strings.HasPrefix(*run.Name, "codecov") { continue } codecovChecks = append(codecovChecks, strings.Replace(*run.Name, "codecov/", "", 1)) } //if codecovStatusChecks < 6 { // fmt.Printf("PR #%d: %s has %d codecov checks\n", *pr.Number, *pr.Title, codecovStatusChecks) //} fmt.Printf("| #%d | %s | %s | %v | %s | %s |\n", *pr.Number, prDate.Format("02.01.2006"), commitDate.Format("02.01.2006"), boolEmoji(!tooOld), boolEmoji(len(codecovChecks) > 0), firstLine(codecovComment), ) } } func boolEmoji(b bool) string { if b { return "✅" } return "❌" } func checksString(checks []string) string { if len(checks) == 0 { return "❌" } return "✅" } func commentURL(comment *github.IssueComment) string { if comment == nil { return "---" } if comment.URL == nil { return "???" } return fmt.Sprintf("[url](%s)", *comment.URL) } func firstLine(comment *github.IssueComment) string { if comment == nil { return "❌" } if comment.Body == nil { return "🙈" } body := *comment.Body if strings.HasPrefix(body, "## [Codecov](https://app.codecov.io/gh/status-im/status-go/pull") { return "✅" } if strings.HasPrefix(body, "### :x:") { numberOfTestsFailed := strings.Split(body, " ")[2] flaky := strings.Contains(body, "flaky") //return fmt.Sprintf("%s failed", numberOfTestsFailed) return fmt.Sprintf("⚠️ (%s) (flaky: %v)", numberOfTestsFailed, flaky) } return strings.Split(*comment.Body, "\n")[0] } ```
igor-sirotin commented 4 weeks ago

Analyzing issues from above:

PR Codecov Checks Codecov Tests Comment EL PROBLEMA
#5962 ❌ ✅ CI job failed, comment old
#5957 ❌ ⚠️ (1) ❄️ - different number of coverage report uploads
- has changed files		 👈
#5955 ❌ ❌ - different number of coverage report uploads
- has changed files
#5953 ❌ ✅ - CI job failed, comment old (already fixed)
#5952 ❌ ✅ - CI job failed, comment old
#5947 ❌ ⚠️ (1) ❄️ - No Files covered by tests were changed
- Commits have different number of coverage report uploads		 👈
#5946 ❌ ❌ - CI job failed
#5945 ❌ ⚠️ (1) 👈
#5941 ❌ ✅ - different number of coverage report uploads
- has changed files
#5930 ✅ ❌ Base branch is not develop
#5907 ❌ ❌ 👈
#5877 ❌ ❌ - Commits have different number of coverage report uploads
igor-sirotin commented 4 weeks ago

Checking #5955.

tests log:

[2024-10-16T14:54:27.579Z] Uploading coverage report to Codecov
[2024-10-16T14:54:27.579Z] info - 2024-10-16 14:54:27,117 -- ci service found: jenkins
[2024-10-16T14:54:27.579Z] info - 2024-10-16 14:54:27,305 -- Found 2 test_results files to report
[2024-10-16T14:54:27.579Z] info - 2024-10-16 14:54:27,306 -- > /home/jenkins/workspace/status-go_prs_tests_PR-5955/report_0.xml
[2024-10-16T14:54:27.579Z] info - 2024-10-16 14:54:27,306 -- > /home/jenkins/workspace/status-go_prs_tests_PR-5955/report_1.xml
[2024-10-16T14:54:28.844Z] info - 2024-10-16 14:54:28,608 -- Process Upload complete
[2024-10-16T14:54:29.627Z] info - 2024-10-16 14:54:29,241 -- ci service found: jenkins
[2024-10-16T14:54:29.627Z] info - 2024-10-16 14:54:29,570 -- Process Commit creating complete
[2024-10-16T14:54:29.995Z] info - 2024-10-16 14:54:29,907 -- Process Report creating complete
[2024-10-16T14:54:29.995Z] info - 2024-10-16 14:54:29,908 -- Finished creating report successfully --- {"response": "{\"status\":\"queued\"}\n"}
[2024-10-16T14:54:29.995Z] warning - 2024-10-16 14:54:29,908 -- xcrun is not installed or can't be found.
[2024-10-16T14:54:30.363Z] warning - 2024-10-16 14:54:30,057 -- No gcov data found.
[2024-10-16T14:54:30.363Z] warning - 2024-10-16 14:54:30,057 -- coverage.py is not installed or can't be found.
[2024-10-16T14:54:30.363Z] info - 2024-10-16 14:54:30,214 -- Found 4 coverage files to report
[2024-10-16T14:54:30.363Z] info - 2024-10-16 14:54:30,214 -- > /home/jenkins/workspace/status-go_prs_tests_PR-5955/c.out
[2024-10-16T14:54:30.363Z] info - 2024-10-16 14:54:30,214 -- > /home/jenkins/workspace/status-go_prs_tests_PR-5955/test_1.coverage.out
[2024-10-16T14:54:30.363Z] info - 2024-10-16 14:54:30,214 -- > /home/jenkins/workspace/status-go_prs_tests_PR-5955/test_0.coverage.out
[2024-10-16T14:54:30.363Z] info - 2024-10-16 14:54:30,215 -- > /home/jenkins/workspace/status-go_prs_tests_PR-5955/coverage_merged.out
[2024-10-16T14:54:39.692Z] info - 2024-10-16 14:54:38,890 -- Your upload is now processing. When finished, results will be available at: https://app.codecov.io/github/status-im/status-go/commit/4b1d93b98e5476e14993b95f9cfb5391049c8941
[2024-10-16T14:54:40.958Z] info - 2024-10-16 14:54:40,694 -- Process Upload complete

tests-rpc log:

[2024-10-16T14:26:26.278Z] Uploading coverage report to Codecov
[2024-10-16T14:26:26.645Z] info - 2024-10-16 14:26:26,458 -- ci service found: jenkins
[2024-10-16T14:26:26.645Z] warning - 2024-10-16 14:26:26,487 -- Some files being explicitly added are found in the list of excluded files for upload. We are still going to search for the explicitly added files. --- {"files": ["/home/jenkins/workspace/status-go_prs_tests-rpc_PR-5955/tests-functional/reports/report.xml"]}
[2024-10-16T14:26:27.025Z] info - 2024-10-16 14:26:26,633 -- Found 1 test_results files to report
[2024-10-16T14:26:27.026Z] info - 2024-10-16 14:26:26,634 -- > /home/jenkins/workspace/status-go_prs_tests-rpc_PR-5955/tests-functional/reports/report.xml
[2024-10-16T14:26:27.394Z] info - 2024-10-16 14:26:27,187 -- Process Upload complete
[2024-10-16T14:26:28.179Z] info - 2024-10-16 14:26:27,806 -- ci service found: jenkins
[2024-10-16T14:26:28.575Z] info - 2024-10-16 14:26:28,344 -- Process Commit creating complete
[2024-10-16T14:26:28.943Z] info - 2024-10-16 14:26:28,758 -- Process Report creating complete
[2024-10-16T14:26:28.943Z] info - 2024-10-16 14:26:28,758 -- Finished creating report successfully --- {"response": "{\"status\":\"queued\"}\n"}
[2024-10-16T14:26:28.943Z] warning - 2024-10-16 14:26:28,759 -- xcrun is not installed or can't be found.
[2024-10-16T14:26:28.943Z] warning - 2024-10-16 14:26:28,901 -- No gcov data found.
[2024-10-16T14:26:28.943Z] warning - 2024-10-16 14:26:28,902 -- coverage.py is not installed or can't be found.
[2024-10-16T14:26:29.311Z] info - 2024-10-16 14:26:29,060 -- Found 2 coverage files to report
[2024-10-16T14:26:29.311Z] info - 2024-10-16 14:26:29,060 -- > /home/jenkins/workspace/status-go_prs_tests-rpc_PR-5955/tests-functional/reports/report.xml
[2024-10-16T14:26:29.311Z] info - 2024-10-16 14:26:29,060 -- > /home/jenkins/workspace/status-go_prs_tests-rpc_PR-5955/tests-functional/coverage/coverage.out
[2024-10-16T14:26:38.650Z] info - 2024-10-16 14:26:37,861 -- Your upload is now processing. When finished, results will be available at: https://app.codecov.io/github/status-im/status-go/commit/4b1d93b98e5476e14993b95f9cfb5391049c8941
[2024-10-16T14:26:39.915Z] info - 2024-10-16 14:26:39,532 -- Process Upload complete

Both reports are detected by the Codecov:

Image

Maybe it's because No files covered by tests were changed 🤔 But it doesn't make any sense. Especially if I'd have the commit check as required.

igor-sirotin commented 4 weeks ago

These are the PRs to consider as problems:

PR Codecov Checks Codecov Tests Comment EL PROBLEMA
#5957 ❌ ⚠️ (1) ❄️ - different number of coverage report uploads
(has changed files)		 👈
#5955 ❌ ❌ - different number of coverage report uploads
(has changed files)
#5947 ❌ ⚠️ (1) ❄️ - different number of coverage report uploads
(no files covered by tests were changed)		 👈
#5945 ❌ ⚠️ (1) 👈
#5941 ❌ ✅ - different number of coverage report uploads
(has changed files)
#5907 ❌ ❌ 👈
#5877 ❌ ❌ - Commits have different number of coverage report uploads

Problems description

  1. has changed files / No Files covered by tests were changed Doesn't seem to affect status checks.

  2. different number of coverage report uploads Will be fixed with https://github.com/status-im/status-go/issues/5809#issuecomment-2422365199 It's now implemented, but required PRs to be rebased. Will watch PRs starting based on e611b1e5131fff706151661d15cc2904c20a7f71 and later.

  3. When a test is failing, no status check is reported.

    Not sure how to tackle this. In general, it makes sense: when one test failed, no point of measuring the coverage. But we automatically re-run failed tests to fight flakiness. So in the same report there's definitely a test failing and same test passing. But Codecov seems to have precedence for the fails.

    I guess we have 2 options:

    • Don't report test results to Codecov Though it's fancy to have, I don't think someone is using it anyway, we have test results in Jenkins.
    • Try to modify the test report before uploading to Codecov and remove any failing tests if there's one passing Not sure how difficult it is. Need to investigate. Will quickly push for the first option first and then let's see if it works.
igor-sirotin commented 3 weeks ago

Another thing that bothers me:

For some reason the last commit for develop branch context is ba37c32. Though it was not on develop. And it was quite long time ago 🤔

Image

https://app.codecov.io/github/status-im/status-go/tree/develop

Image

UPD:

Just noted that a new commit appeared for the develop branch context: 807397f And it's again a commit from @jakubgs 👀 What kind of super-power is that?! 🤔

Image

Image

Interesting:

https://app.codecov.io/github/status-im/status-go/commits/develop

Image

igor-sirotin commented 3 weeks ago

Another issue I found in https://github.com/status-im/status-go/pull/5985:

Image

UPD: probably not an issue. Codecov is actually showing better results, because it counts the functional-tests 🎉

Image

igor-sirotin commented 2 weeks ago

So it seems that only directly pushed commits were visible in develop branch context.

This could be because of the wrong default branch:

Image

I changed it to develop, will monitor further.

Image

UPD:

Doesn't seem to help 🙁 Image

UPD 2:

ok maybe it is actually working now Image

igor-sirotin commented 1 week ago

Alright, the issue with the develop branch seem to be fixed now.

Image