Need help to understand the ids/numbers when upload for parallel jobs/builds.

qzhuyan commented 1 year ago

I have a scenario that in one GitHub run I run cover for two languages thus I need to run two 'uploader' in sequential.

So the steps looks like this:

run make cover and it generates cover data for language A and language B.
run coveralls github action (V2) to upload coverdata for language A.
run another uploader command to upload the coverdata for language B.
Mark the parallel run as finished.

I think the service_number should be the same in steps 2, 3, and 4, correct? Then I used different 'service_job_id' and different flag_name in steps 2 and step 3. The result is I always get the cover report in step 3 only.

example: https://coveralls.io/builds/60485505 (with debug=true)

I also see you have 'master' and 'main' branches, which branch of this action is recommended?

mrexox commented 1 year ago

Hey!

Yes, you are right. service_number should be the same for all parallel jobs, and for the one with parallel-finished: true option. If you have all your coverage reports described in one workflow file - then you shouldn't care about providing service_number - it is automatically fetched from ENVs.

I'd recommend using @v2 tag for your workflow because master and main behave a bit differently. master branch and v1 tag uses node-coveralls which is not being developed for a long time. main branch and v2 tag use coverage-reporter which is under active development and has a lot of bugs fixed. Probably no special configuration is required when you change master to v2.

qzhuyan commented 1 year ago

Now I switched to V2 and from debugging info: Two upload calls for two coverage report: first one:

{
  "service_branch": "test/william/prop-test",
  "service_build_url": "https://github.com/qzhuyan/quic/actions/runs/5197308098",
  "service_job_id": "build",
  "service_job_url": "https://github.com/qzhuyan/quic/actions/runs/5197308098",
  "service_name": "github",
  "service_number": "5197308098",
  "commit_sha": "e95a347600657a95f38f880badf969b2ddc0cc40",
  "repo_name": "qzhuyan/quic",
  "repo_token": "***",
  "flag_name": "run-c-lcov",
  "source_files": [
    {
    ...}
   "parallel": true,

second one:

Content-Type: application/json\r\n\r\n{\"commit_sha\":\"e95a347600657a95f38f880badf969b2ddc0cc40\",\"flag_name\":\"erlang\",\"parallel\":true,\"repo_token\":\"***\",\"service_job_id\":\"build\",\"service_job_number\":\"2\",\"service_name\":\"github\",\"service_number\":\"5197308098\",\"source_files\":[{\"coverage\":[null,null,nu ...

The service_number are both 5197308098 The service_job_id are both build (feels wrong), maybe that is the reason one overrides the other? The service_job_number are different, one omit another is "2" Both have parallel: true I will tweak to use different service_job_id and see what happens.

qzhuyan commented 1 year ago

tested with different service_job_id, one uploader use build another uploader for use build2, no luck.

any idea?

mrexox commented 1 year ago

When using parallel: true you have to close the build after all parallel jobs are reported.

- uses: coverallsapp/github-action@v2
  with:
    parallel-finished: true

Otherwise the build will remain open and won't calculate the overall coverage.

If this doesn't help, could you share your configuration, please? I will try to suggest a solution based on it.

qzhuyan commented 1 year ago

the workflow is here: https://github.com/qzhuyan/quic/blob/test/william/prop-test/.github/workflows/cover.yaml#L36-L61

We have two languages: C and Erlang

C: We use coverallsapp/github-action@v2 to upload cover report
Erlang: we use this rebar3 plugin https://github.com/qzhuyan/coveralls-erl to upload the report and for the API param they are defined here: https://github.com/qzhuyan/quic/blob/test/william/prop-test/rebar.config.script#L6-L14

build log(unsure if you have the access): https://github.com/qzhuyan/quic/actions/runs/5197440164

One build example: https://coveralls.io/builds/60575827

mrexox commented 1 year ago

Ok, it looks like Erlang plugin is a bit outdated and hasn't been updated for a long time. Unfortunately, I am not familiar with Erlang, so I can't say what's exactly wrong here.

Can you say what you are expecting? Looks like builds don't receive Erlang coverage, right?

qzhuyan commented 1 year ago

I just found it works before in March, see: https://coveralls.io/builds/57942310 but I don't have debug enabled and cannot see how API gets called.

What I expect is: since I have two parallel (actually seq ) uploads (one for C and one for Erlang), I expect coveralls could merge them into one after parallel-finish is called.

The problem for me is the erlang upload seems to overwrite what C uploaded. The report in https://coveralls.io/builds/60575827 only shows the stuff from erlang upload. AND if I skip erlang upload I get report from C only.

No worry about erlang plugin, I have forked erlang plugin I could control how we call the API also with debugging info.

Since your backend is not opensource, I just want to get inputs from you on which params shall I use for doing this scenario.

some inputs will be very helpful like the following:

# first call API for C coverage 
with:
service_number: 5197308098
service_job_id: "cover_C"
service_job_number: ""
parallel: true

## Second call for Erlang
with:
service_number: 5197308098
service_job_id: cover_erlang
service_job_number: "2"
parallel: true

third call to finish:
with:
service_number: 5197308098
 parallel-finished: true

thanks!

mrexox commented 1 year ago

If you want to merge them you need to provide a similar flag_name parameter. Also service_number and commit_sha must be the same for both requests.

I can see that only one coverage is being uploaded (for C and LCOV), maybe there's an error when Erlang coverage is being uploaded. Unfortunately the response is not visible from logs, so I am not sure. Can you show the server response from Erlang uploader?

mrexox commented 1 year ago

By the way, in case we get stuck, there is a converter of .coveradata to Cobertura XML format which is supported by coverage-reporter: https://github.com/covertool/covertool.

qzhuyan commented 1 year ago

triggered new build: https://github.com/qzhuyan/quic/actions/runs/5197440164/jobs/9423205035

API call for uploading C part with following (using: coverallsapp/github-action@v2)

"service_branch": "test/william/prop-test",
2023-06-09T09:22:03.5717514Z   "service_build_url": "https://github.com/qzhuyan/quic/actions/runs/5197440164",
2023-06-09T09:22:03.5718107Z   "service_job_id": "build",
2023-06-09T09:22:03.5719170Z   "service_job_url": "https://github.com/qzhuyan/quic/actions/runs/5197440164",
2023-06-09T09:22:03.5719698Z   "service_name": "github",
2023-06-09T09:22:03.5720522Z   "service_number": "5197440164",
2023-06-09T09:22:03.5721060Z   "commit_sha": "bd403c8390cfa85f73bd810647d8c1a5dbf54a1c",
2023-06-09T09:22:03.5721806Z   "repo_name": "qzhuyan/quic",
2023-06-09T09:22:03.5722466Z   "repo_token": "***",
2023-06-09T09:22:03.5723259Z   "flag_name": "run-c-lcov",
2023-06-09T09:22:03.6873324Z.  "parallel": true,
2023-06-09T09:22:03.6873404Z   "git": {
2023-06-09T09:22:03.6873836Z     "branch": "test/william/prop-test",
2023-06-09T09:22:03.6873916Z     "head": {
2023-06-09T09:22:03.6874060Z       "id": "bd403c8390cfa85f73bd810647d8c1a5dbf54a1c",
2023-06-09T09:22:03.6874198Z       "committer_email": "mscame@gmail.com",
2023-06-09T09:22:03.6874314Z       "committer_name": "William Yang",
2023-06-09T09:22:03.6874438Z       "author_email": "mscame@gmail.com",
2023-06-09T09:22:03.6874552Z       "author_name": "William Yang",
2023-06-09T09:22:03.6874708Z       "message": "ci(cover): service job id build2 for erlang"
2023-06-09T09:22:03.6874791Z     }
2023-06-09T09:22:03.6874853Z   },
2023-06-09T09:22:03.6875049Z   "run_at": "2023-06-09T09:22:03+00:00"

2023-06-09T09:22:04.1424095Z ✅ API Response: {"message":"Coverage for parallel build uploaded","url":"https://coveralls.io/builds/60630675"}

now I get error code: 422 while I upload the erlang part.

with

"commit_sha":"bd403c8390cfa85f73bd810647d8c1a5dbf54a1c",
"flag_name":"erlang",
"parallel":true,
"repo_token":"***",
"service_job_id":"build2",
"service_job_number":"2",
"service_name":"github",
"service_number":"5197440164",
"source_files":[{"coverage":....

error I get...

422: "{\"message\":\"Can't add a job to a build that is already closed. Build 5197440164 is closed. See docs.coveralls.io/parallel-builds\",\"error\":true}"

qzhuyan commented 1 year ago

Above is a rerun, the service_number is reused. Looks like it is not a good idea to use github run id as the service_number.

now I will trigger one new run and check the reply from coveralls while uploading Erlang.

qzhuyan commented 1 year ago

ok with new build: https://github.com/qzhuyan/quic/actions/runs/5220552560/jobs/9423759253

C upload with: (using: coverallsapp/github-action@v2)

2023-06-09T09:51:04.1988221Z "service_branch": "test/william/prop-test",
2023-06-09T09:51:04.1988598Z   "service_build_url": "https://github.com/qzhuyan/quic/actions/runs/5220552560",
2023-06-09T09:51:04.1988907Z   "service_job_id": "build",
2023-06-09T09:51:04.1989204Z   "service_job_url": "https://github.com/qzhuyan/quic/actions/runs/5220552560",
2023-06-09T09:51:04.1989505Z   "service_name": "github",
2023-06-09T09:51:04.1989743Z   "service_number": "5220552560",
2023-06-09T09:51:04.1990021Z   "commit_sha": "a907bfb6b5b9be503c5ca2e3e40cfa032d534bf5",
2023-06-09T09:51:04.1990314Z   "repo_name": "qzhuyan/quic",
2023-06-09T09:51:04.1990968Z   "repo_token": "***",
2023-06-09T09:51:04.1991254Z   "flag_name": "run-c-lcov",
2023-06-09T09:51:04.1991465Z   "source_files": [....]
2023-06-09T09:51:04.2820100Z   "parallel": true,
2023-06-09T09:51:04.2820178Z   "git": {
2023-06-09T09:51:04.2820492Z     "branch": "test/william/prop-test",
2023-06-09T09:51:04.2820563Z     "head": {
2023-06-09T09:51:04.2820700Z       "id": "a907bfb6b5b9be503c5ca2e3e40cfa032d534bf5",
2023-06-09T09:51:04.2820822Z       "committer_email": "mscame@gmail.com",
2023-06-09T09:51:04.2820930Z       "committer_name": "William Yang",
2023-06-09T09:51:04.2821045Z       "author_email": "mscame@gmail.com",
2023-06-09T09:51:04.2821145Z       "author_name": "William Yang",
2023-06-09T09:51:04.2821271Z       "message": "chore: just trigger new build"
2023-06-09T09:51:04.2821337Z     }
2023-06-09T09:51:04.2821393Z   },
2023-06-09T09:51:04.2821558Z   "run_at": "2023-06-09T09:51:04+00:00"

get resp:

2023-06-09T09:51:04.9292791Z ✅ API Response: {"message":"Coverage for parallel build uploaded","url":"https://coveralls.io/builds/60631251"}

Erlang upload with:

{"commit_sha":"a907bfb6b5b9be503c5ca2e3e40cfa032d534bf5",
 "flag_name":"erlang",
 "parallel":true,
 "repo_token":"***",
 "service_job_id":"build2",
 "service_job_number":"2",
 "service_name":"github",
 "service_number":"5220552560",
 "source_files":[{"coverage":
 ...

And get resp:

{ok,{{"HTTP/1.1",200,"OK"},
2023-06-09T09:51:06.5695929Z                         [{"cache-control",
2023-06-09T09:51:06.5696408Z                           "max-age=0, private, must-revalidate"},
2023-06-09T09:51:06.5697045Z                          {"connection","keep-alive"},
2023-06-09T09:51:06.5697353Z                          {"date","Fri, 09 Jun 2023 09:51:06 GMT"},
2023-06-09T09:51:06.5697743Z                          {"etag","W/\"bfcb5e7503cecae0ea6dacc588038bb3\""},
2023-06-09T09:51:06.5698067Z                          {"server","cloudflare"},
2023-06-09T09:51:06.5699013Z                          {"content-length","95"},
2023-06-09T09:51:06.5699606Z                          {"content-type","application/json; charset=utf-8"},
2023-06-09T09:51:06.5700026Z                          {"status","200 OK"},
2023-06-09T09:51:06.5700525Z                          {"strict-transport-security",
2023-06-09T09:51:06.5701221Z                           "max-age=31536000; includeSubDomains"},
2023-06-09T09:51:06.5704393Z                          {"referrer-policy","strict-origin-when-cross-origin"},
2023-06-09T09:51:06.5704961Z                          {"x-permitted-cross-domain-policies","none"},
2023-06-09T09:51:06.5705555Z                          {"x-xss-protection","1; mode=block"},
2023-06-09T09:51:06.5708118Z                          {"x-request-id",
2023-06-09T09:51:06.5708590Z                           "0a167d78-d5f4-402e-af7d-5e08124b9657"},
2023-06-09T09:51:06.5709069Z                          {"x-download-options","noopen"},
2023-06-09T09:51:06.5709675Z                          {"x-frame-options","SAMEORIGIN"},
2023-06-09T09:51:06.5710443Z                          {"x-runtime","0.262724"},
2023-06-09T09:51:06.5710855Z                          {"x-content-type-options","nosniff"},
2023-06-09T09:51:06.5711206Z                          {"x-powered-by",
2023-06-09T09:51:06.5711514Z                           "Phusion Passenger(R) Enterprise 6.0.17"},
2023-06-09T09:51:06.5711877Z                          {"x-powered-by","cloud66"},
2023-06-09T09:51:06.5712243Z                          {"cf-cache-status","DYNAMIC"},
2023-06-09T09:51:06.5712563Z                          {"report-to",
2023-06-09T09:51:06.5713603Z                           "{\"endpoints\":[{\"url\":\"https:\\/\\/a.nel.cloudflare.com\\/report\\/v3?s=PPUKb4QP6wMODF%2Fibffq7dPvWMjkiegFFlC0K%2BKejOBo0mU7FfMMMMb8BGealyPYk61qjDUyL5aWwUKXRHiMwVJrV5qFITOO82pRrmQglWX3e5L0zzw%2F2IoN%2Fc%2F3wA%3D%3D\"}],\"group\":\"cf-nel\",\"max_age\":604800}"},
2023-06-09T09:51:06.5714711Z                          {"nel",
2023-06-09T09:51:06.5715144Z                           "{\"success_fraction\":0,\"report_to\":\"cf-nel\",\"max_age\":604800}"},
2023-06-09T09:51:06.5715560Z                          {"cf-ray","7d48795f3b02e781-DFW"},
2023-06-09T09:51:06.5715924Z                          {"alt-svc","h3=\":443\"; ma=86400"}],
2023-06-09T09:51:06.5716337Z                         "{\"message\":\"Coverage for parallel build uploaded\",\"url\":\"https://coveralls.io/builds/60631252\"}"}}

parallel finish call

2023-06-09T09:51:17.6762152Z ⭐️ Calling parallel done webhook: https://coveralls.io/webhook
2023-06-09T09:51:18.1549666Z ---
2023-06-09T09:51:18.1550589Z ✅ API Response: {"done":true,"url":"https://coveralls.io/builds/60631252","jobs":1}
2023-06-09T09:51:18.1551373Z - 💛, Coveralls

I only get erlang report

from: https://coveralls.io/builds/60631252

mrexox commented 1 year ago

Oh, I see, for some reason those reports go to the different builds (you can see that URLs differ in the responses). I need to debug this issue to find out if there's a problem on the backend side or not.

mrexox commented 1 year ago

I think I found the issue and fixed it. The problem was that we weren't checking commit_sha option, and Erlang integration doesn't provide git options like coverage-reporter does.

You can start another build. I can suggest you the following change in your Erlang coverage plugin configuration:

Remove service_job_number - it is not required
Don't edit service_job_id - now you append 2 to the end, but this is not required

qzhuyan commented 1 year ago

it works now! thanks!

https://coveralls.io/builds/60633707 https://github.com/qzhuyan/quic/actions/runs/5221556893/jobs/9425963829

but I got another issue: https://coveralls.io/jobs/123018755/source_files/9867975771 "the commit SHA "0f536fed6d318fbc28ce978f39d2276cce789b7a" was not found in your repository, so the file cannot be loaded. This may be because you posted from a local development environment, or your CI created an ephemeral commit."

mrexox commented 1 year ago

It must be a temporary issue related to Github API limits. Could you try one more time?

qzhuyan commented 1 year ago

solved ! apprecated! 👍

coverallsapp / github-action