search

JuliaCI / julia-buildbot

Buildbot configuration for build.julialang.org
MIT License
18 stars 14 forks source link

Code coverage: checkout the source code for the relevant commit? #210

Closed DilumAluthge closed 3 years ago

DilumAluthge commented 3 years ago

So, if I understand correctly, the code coverage job (https://github.com/JuliaCI/julia-buildbot/blob/master/master/coverage.py) doesn't actually have a copy of the source code. Instead, it seems to just download the tarball of the latest nightly binaries for 64-bit Linux, and then use those nightly binaries when running the tests.

When processing the code coverage information, we need to exclude external stdlibs. Currently, we just hard-code the list of external stdlibs (#208). However, I'd like to auto-generate the list of external stdlibs, so that we don't need to update the hard-coded list every time we add a new external stdlib.

My current approach for auto-generating the list of external stdlibs requires having access to the source code. Specifically, I look for files of the form stdlib/FOO.version to identify external stdlibs. Unfortunately, this won't work on the coverage buildbots because they don't have the source code.

So, is there a way that we can checkout the source code for the appropriate commit on the coverage buildbots?

DilumAluthge commented 3 years ago

cc: @vtjnash @staticfloat

staticfloat commented 3 years ago

What about using the UNREGISTERED_STDLIBS list that Pkg keeps? E.g. something like this:

julia> [(k, v) for (k, v) in Pkg.Types.stdlibs() if k ∉ keys(Pkg.Types.UNREGISTERED_STDLIBS)]
31-element Vector{Tuple{Base.UUID, String}}:
 (UUID("efcefdf7-47ab-520b-bdef-62a2eaa19f15"), "PCRE2_jll")
 (UUID("745a5e78-f969-53e9-954f-d19f2f74f4e3"), "LibUnwind_jll")
 (UUID("05823500-19ac-5b8b-9628-191a04bc5112"), "OpenLibm_jll")
 (UUID("56f22d72-fd6d-98f1-02f0-08ddc0907c33"), "Artifacts")
 (UUID("05ff407c-b0c1-5878-9df8-858cc2e60c36"), "dSFMT_jll")
 (UUID("e37daf67-58a4-590a-8e99-b0245dd2ffc5"), "LibGit2_jll")
 (UUID("bea87d4a-7f5b-5778-9afe-8cc45184846c"), "SuiteSparse_jll")
 (UUID("e66e0078-7015-5450-92f7-15fbd957f2ae"), "CompilerSupportLibraries_jll")
 (UUID("8e850b90-86db-534c-a0d3-1478176c7d93"), "libblastrampoline_jll")
 (UUID("781609d7-10c4-51f6-84f2-b8444358ff6d"), "GMP_jll")
 (UUID("0dad84c5-d112-42e6-8d28-ef12dabb789f"), "ArgTools")
 (UUID("4536629a-c528-5b80-bd46-f80d51c5b363"), "OpenBLAS_jll")
 (UUID("ca575930-c2e3-43a9-ace4-1e988b2c1908"), "NetworkOptions")
 (UUID("3a97d323-0669-5f0c-9066-3539efd106a3"), "MPFR_jll")
 (UUID("83775a58-1f1d-513f-b197-d71354ab007a"), "Zlib_jll")
 (UUID("29816b5a-b9ab-546f-933c-edad1886dfa8"), "LibSSH2_jll")
 (UUID("3f19e933-33d8-53b3-aaab-bd5110c3b7a0"), "p7zip_jll")
 (UUID("14a3606d-f60d-562e-9121-12d972cd8159"), "MozillaCACerts_jll")
 (UUID("f43a241f-c20a-4ad4-852c-f6b1247861c6"), "Downloads")
 (UUID("deac9b47-8bc7-5906-a0fe-35ac56dc84c0"), "LibCURL_jll")
 (UUID("47c5dbc3-30ba-59ef-96a6-123e260183d9"), "LLVMLibUnwind_jll")
 (UUID("4af54fe1-eca0-43a8-85a7-787d91b784e3"), "LazyArtifacts")
 (UUID("fa267f1f-6049-4f14-aa54-33bafae1ed76"), "TOML")
 (UUID("b27032c2-a3e7-50c8-80cd-2d36dbcbfd21"), "LibCURL")
 (UUID("c8ffd9c3-330d-5841-b78e-0817d7145fa1"), "MbedTLS_jll")
 (UUID("4607b0f0-06f3-5cda-b6b1-a6196a1729e9"), "SuiteSparse")
 (UUID("8e850ede-7688-5339-a07c-302acd2aaf8d"), "nghttp2_jll")
 (UUID("a4e569a6-e804-4fa4-b0f3-eef7a1d5b13e"), "Tar")
 (UUID("8f36deef-c2a5-5394-99ed-8e07531fb29a"), "libLLVM_jll")
 (UUID("ea8e919c-243c-51af-8825-aaa63cd721ce"), "SHA")
 (UUID("183b4373-6708-53ba-ad28-60e28bb38547"), "LibUV_jll")
staticfloat commented 3 years ago

The JLLs are not "true" external stdlibs because they're kinda-sorta vendored in (and altered) but that will stop being the case eventually, and we also don't really care about coverage of them anyway.

staticfloat commented 3 years ago

SHA should actually become an external stdlib

staticfloat commented 3 years ago

Hmmm, Artifacts is a tricky case though....

DilumAluthge commented 3 years ago

Also, technically TOML isn't an external stdlib (it's vendored in).

DilumAluthge commented 3 years ago

The JLLs are not "true" external stdlibs because they're kinda-sorta vendored in (and altered) but that will stop being the case eventually, and we also don't really care about coverage of them anyway.

BTW, I think this answers my question in https://github.com/JuliaLang/julia/issues/41237? That is, it sounds like we should just exclude the stdlibs JLLs when processing code coverage?

DilumAluthge commented 3 years ago

So I think the strategy in https://github.com/JuliaLang/julia/pull/41238 is working well. In https://github.com/JuliaLang/julia/pull/41238, we:

  1. Exclude all external stdlibs. Since we have a copy of the source code, it is easy to identify external stdlibs: just look for the stdlib/*.version files.
  2. Exclude all stdlib JLLs. This is also easy: just look for the folders named stdlib/*_jll/.

I think the best plan forward will be to turn off the coverage buildbots and only use Buildkite for coverage.