Open froody opened 2 years ago
Likewise, there are no binaries for Ubuntu 16.04, which were provided for 13.0.0. I recall that LLVM had previously planned to drop Ubuntu 16.04 support in LLVM 11, but those binaries are also required for Debian 9 (which doesn't reach End-of-Life until end of June 2022) due to sharing a glibc version, and subsequent LLVM releases including 13.0.0 have provided them.
I am confused as to why support for pre-18.04 Ubuntu releases would be dropped in a patch release; was this intentional? If not, could we have those variants generated for 13.0.1?
If it was intentional, it would be helpful to document LLVM's policy for dropping platform support. There is a Support Policy document, but it only covers what parts of the LLVM codebase are supported, not the platforms that supported releases are built for.
A previous example of release platform inconsistency:
x86_64-apple-darwin
, x86_64-linux-gnu-ubuntu-16.04
, and x86_64-linux-gnu-ubuntu-20.04
x86_64-linux-gnu-ubuntu-16.04
, but was missing binaries for x86_64-apple-darwin
and x86_64-linux-gnu-ubuntu-20.04
.At the time, I assumed this meant that the 12.0.1 patches were not relevant to x86_64-apple-darwin
, and I had missed the fact that Ubuntu 20.04 was missing (as I only depend on the lowest-supported Ubuntu target).
The reason there is platform inconsistency is because the binaries are provided by volunteers. We don't have any official builds that we produce consistently. For anyone interested in trying to fix this, I recommend writing a GitHub workflow that we can use to produce official binaries.
The reason there is platform inconsistency is because the binaries are provided by volunteers. We don't have any official builds that we produce consistently.
That is good information to know, thanks.
For anyone interested in trying to fix this, I recommend writing a GitHub workflow that we can use to produce official binaries.
Sure, now that I'm aware of the current process, I'd be happy to contribute an automated one (at least for the platforms corresponding to standard GitHub builders).
Sure, now that I'm aware of the current process, I'd be happy to contribute an automated one (at least for the platforms corresponding to standard GitHub builders).
You can build inside containers on the GitHub builders, so you aren't limited to just the distributions supported by the builders (Currently Ubuntu, Windows, Mac OS).
What distribution process should I follow to reproduce the distributions currently produced? I've found https://llvm.org/docs/BuildingADistribution.html which references clang/cmake/caches/DistributionExample.cmake
, but that currently causes an error locally, and I haven't found anything else listing what a "standard" distribution is configured with.
For building the release binaries, all you need to do is run the test-release.sh script. More details here: https://llvm.org/docs/ReleaseProcess.html#test-release-sh
I've finally had time to finish working on this, and have a branch up that modifies the release tasks workflow to build binaries (currently only building on Ubuntu 18.04 and 20.04 for testing purposes): https://github.com/str4d/llvm-project/blob/release-workflow/.github/workflows/release-tasks.yml
The issue I'm running into is that the builders take longer than six hours to run, which exceeds the maximum job runtime for GitHub-hosted workers.
@llvm/issue-subscribers-backend-x86
@str4d Thanks for working on this. A few suggestions:
I think it would be better to factor this workflow out into it's own file so that it can be run manually, this will make it easier to test, and you can still have the release-task job trigger it (see https://github.com/llvm/llvm-project/blob/main/.github/workflows/llvm-project-tests.yml as an example).
We have access to more powerful runners that you can use. Once you have a way to run it manually ping me and I can help get it setup.
- I think it would be better to factor this workflow out into it's own file
Heh, that's how I originally had it set up when I started working on it in April, but when I picked it back up I saw the new release tasks script that was added in July and figured that was where it should go. I can easily split it back out (though I'll need to duplicate the validation step that I currently share), and for now I'll change it to only run manually (rather than trying to have the release tasks workflow trigger it, though I do know how to do this if desired).
- We have access to more powerful runners that you can use.
Nice! I'll get the branch updated to also have setups for all the workers that GitHub Actions has (on which I can run the workflow up to the "start the multi-hour build phase"). Then we can see about migrating it to the set of runners that you use.
Okay, I now have a separate workflow that runs manually:
Currently it only builds binaries for targets matching GitHub-hosted runners. I did look into containers, but I could not find recent FreeBSD Docker images that I felt were trustworthy, and I don't know enough about the LLVM release process to know how cross-compiling might work.
@str4d I enabled the more powerful runners and also hacked the workflow so it would trigger on a pull request: https://github.com/llvm/llvm-project-release-prs/pull/208
Unfortunately, it looks like the more powerful runners may no longer be free, so we will have to wait to enable this until we get a budget or verify that we aren't going to be charged. I will work on this.
@str4d Sorry for the delay. We have the more powerful builders enabled now, and I ran a test run with your patch: https://github.com/llvm/llvm-project-release-prs/actions/runs/3833935156/jobs/6631222305
Unfortunately some of the tests fail. I think we need to add an option to test-release.sh that will package the binaries even if the tests fail. We also may want to use the --ninja flag since I think that will give us faster builds. If you want to update your branch with these fixes, I can test it again.
I think we need to add an option to test-release.sh that will package the binaries even if the tests fail.
I don't know enough about the test-release.sh
script to do this. If someone else can suggest the necessary changes I can include them.
We also may want to use the --ninja flag since I think that will give us faster builds.
I've added the -use-ninja
flag to test-release.sh
, and CI now installs Ninja on the configured builders. I also rebased onto the llvmorg-15.0.6
tag.
https://github.com/str4d/llvm-project/commit/69b47d855fad658e0e8760a2c07f5464b22d39cf
I think we need to add an option to test-release.sh that will package the binaries even if the tests fail.
I don't know enough about the
test-release.sh
script to do this. If someone else can suggest the necessary changes I can include them.
Ok, I can try to come up with a patch.
For reference, I pulled out the unique targets that have published binary releases for 13.0.0, 14.0.0, and/or 15.0.0:
x86_64-apple-darwin
x86_64-linux-gnu-ubuntu-16.04
(previously, now removed)x86_64-linux-gnu-ubuntu-18.04
(currently deprecated)x86_64-linux-gnu-ubuntu-20.04
x86_64
targets that we could (maybe) build via Docker container:
x86_64-linux-sles12.4
x86_64
targets that we can't build via Docker (because they don't use Linux kernels):
amd64-pc-solaris2.11
amd64-unknown-freebsd12
amd64-unknown-freebsd13
aarch64-linux-gnu
arm64-apple-darwin21.0
armv7a-linux-gnueabihf
i386-unknown-freebsd12
i386-unknown-freebsd13
powerpc64-ibm-aix-7.2
powerpc64le-linux-rhel-7.9
powerpc64le-linux-rhel-8.4
powerpc64le-linux-ubuntu-18.04
sparc64-unknown-linux-gnu
sparcv9-sun-solaris2.11
@str4d FYI I've created a GitHub action [1] that allows to run commands on platforms not supported by GitHub (they run in a VM). It currently supports FreeBSD, OpenBSD and NetBSD.
Linux on ARM and PPC can easily be run using Docker and QEMU. Here's an example from one of my pipelines [2].
[1] https://github.com/marketplace/actions/cross-platform-action [2] https://github.com/jacob-carlborg/lime/blob/f4d9c8c4265b61b2844b93a01f38b2d09433830e/.github/workflows/ci.yml#L163-L179
@str4d Would you be able to submit the patch on Phabricator?
@str4d A few more suggestions:
@str4d Would you be able to submit the patch on Phabricator?
I'll see if I can figure this out.
@str4d I went ahead and submitted the patch here: https://reviews.llvm.org/D143535
I was blocked from upgrading to LLVM 16+ in my use case because the Intel macOS binaries stopped being built by volunteers, and were removed from my automated workflow when it was upstreamed (due to timeouts per above). Now that we've dropped support for Intel macOS in our use case, I'm back looking at this again.
It looks like @tstellar chose Ubuntu 22.04 as the target to build for when upstreaming my workflow, and removed the Ubuntu 18.04 and 20.04 versions. Unfortunately, this results in a glibc
version dependency that is newer than several of the other Debian-based versions that are still in their support windows:
(EDIT: I now see that whoever is building the Linux binaries manually built the LLVM 18 binary on Ubuntu 18.04, so at least on LLVM 18 we have current glibc compatibility.)
Platform | glibc |
End of Support | Notes |
---|---|---|---|
Ubuntu 18.04 | 2.27 | April 2023 | Binaries available for LLVM 15 and below, as well as LLVM 18.1.4 |
Debian Buster | 2.28 | June 2024 | |
Ubuntu 20.04 | 2.31 | April 2025 | |
Debian Bullseye | 2.31 | June 2026 | |
Ubuntu 22.04 | 2.35 | April 2027 | Binaries available for LLVM 16 and 17 |
Debian Bookworm | 2.36 | June 2028 |
Debian Buster reaches EoS in two months, so I think we can ignore that. Hence, if the automated release workflow built with Ubuntu 20.04 instead of Ubuntu 22.04, we would have maximal glibc
coverage.
@tstellar it looks like PRs are now done directly in GitHub? If that's the case, I'll open a PR with this proposed change.
Additionally, now that GitHub is deploying ARM macOS runners, it would be a good idea to look into adding that as an automated target (replacing the current volunteer-uploaded builds, reducing burden on them). The ARM macOS runners are noticeably faster than their Intel macOS counterparts AFAICT, so timeouts should hopefully not be an issue.
@str4d For Linux, how about building fully statically linked binaries? Then you don't need to worry about which distribution or version. Build in a container running Alpine, which uses Musl libc.
BTW, the Linux runners have been updated and are faster now as well.
@str4d For Linux, how about building fully statically linked binaries? Then you don't need to worry about which distribution or version. Build in a container running Alpine, which uses Musl libc.
I personally don't mind that, but I'm not an LLVM developer so a) IDK if there are other considerations necessary here in terms of how LLVM compilers behave when compiled against Musl vs glibc, and b) I have no idea how to actually set up LLVM to be compiled in that way (i.e. if compiling in an Alpine container is actually sufficient, or if there are additional LLVM build system considerations). In my own experience with Musl in end-user binaries (for applications, not compilers), I've generally had to provide two sets of binaries for different users, and I could imagine the same possibly applying here (but again, I'm not an LLVM developer so IDK).
In any case, the automated release workflow I wrote is intentionally pluggable, so it should be relatively straightforward to define another runner that does a Musl build.
I've given this some more thought and my idea would only work for the binaries and not the pre compiled libraries.
@str4d I went ahead and submitted the patch here: https://reviews.llvm.org/D143535
Looks like this diff was merged and the yml file is in the repo. Is there another step that's required here to get this to fire when new releases are cut? I saw comments about needing paid runners in some PRs that touched that file. Is funding the limiting factor here? (Can we contribute somewhere?) The releases also each still include a comment about how the binaries are built by volunteers, so I'm unsure how this all works currently.
@adzenith the release binaries workflow is already firing on releases: https://github.com/llvm/llvm-project/actions/workflows/release-binaries.yml
However, it appears to be failing tests, which I think is leading to the final binary tarball not being assembled, and therefore the subsequent upload fails due to a missing file. I do not know enough about the LLVM test infrastructure to know how to address these, but I've pulled (what I believe to be) the test results from the most recent run (which was manually triggered, and seems to have more failures than the prior automated run):
2024-04-05T05:59:31.9924169Z Failed Tests (25):
2024-04-05T05:59:31.9931865Z BOLT :: perf2bolt/perf_test.test
2024-04-05T05:59:31.9932405Z Flang :: Lower/io-implied-do-fixes.f90
2024-04-05T05:59:31.9932863Z Flang :: Lower/io-statement-2.f90
2024-04-05T05:59:31.9933496Z Flang :: Lower/vector-subscript-io.f90
2024-04-05T05:59:31.9934127Z lldb-api :: commands/expression/context-object/TestContextObject.py
2024-04-05T05:59:31.9935128Z lldb-api :: commands/expression/import-std-module/deque-dbg-info-content/TestDbgInfoContentDequeFromStdModule.py
2024-04-05T05:59:31.9936347Z lldb-api :: commands/expression/import-std-module/list-dbg-info-content/TestDbgInfoContentListFromStdModule.py
2024-04-05T05:59:31.9937579Z lldb-api :: commands/expression/import-std-module/vector-dbg-info-content/TestDbgInfoContentVectorFromStdModule.py
2024-04-05T05:59:31.9938709Z lldb-api :: functionalities/asan/TestMemoryHistory.py
2024-04-05T05:59:31.9939462Z lldb-api :: functionalities/data-formatter/data-formatter-advanced/TestDataFormatterAdv.py
2024-04-05T05:59:31.9940391Z lldb-api :: functionalities/data-formatter/data-formatter-categories/TestDataFormatterCategories.py
2024-04-05T05:59:31.9941277Z lldb-api :: functionalities/data-formatter/data-formatter-cpp/TestDataFormatterCpp.py
2024-04-05T05:59:31.9942200Z lldb-api :: functionalities/data-formatter/data-formatter-python-synth/TestDataFormatterPythonSynth.py
2024-04-05T05:59:31.9943175Z lldb-api :: functionalities/data-formatter/data-formatter-smart-array/TestDataFormatterSmartArray.py
2024-04-05T05:59:31.9944205Z lldb-api :: functionalities/data-formatter/data-formatter-stl/libcxx/iterator/TestDataFormatterLibccIterator.py
2024-04-05T05:59:31.9945217Z lldb-api :: functionalities/data-formatter/data-formatter-stl/libcxx/map/TestDataFormatterLibccMap.py
2024-04-05T05:59:31.9946301Z lldb-api :: functionalities/data-formatter/data-formatter-stl/libcxx/unordered_map/TestDataFormatterLibccUnorderedMap.py
2024-04-05T05:59:31.9947305Z lldb-api :: functionalities/data-formatter/data-formatter-synth/TestDataFormatterSynth.py
2024-04-05T05:59:31.9948048Z lldb-api :: functionalities/gdb_remote_client/TestGDBRemoteClient.py
2024-04-05T05:59:31.9948782Z lldb-api :: functionalities/process_save_core_minidump/TestProcessSaveCoreMinidump.py
2024-04-05T05:59:31.9949460Z lldb-api :: lang/c/global_variables/TestGlobalVariables.py
2024-04-05T05:59:31.9950006Z lldb-api :: lang/cpp/class_static/TestStaticVariables.py
2024-04-05T05:59:31.9950676Z lldb-api :: python_api/formatters/TestFormattersSBAPI.py
2024-04-05T05:59:31.9951152Z lldb-api :: python_api/value/TestValueAPI.py
2024-04-05T05:59:31.9951821Z lldb-shell :: SymbolFile/DWARF/x86/debug-types-dwo-cross-reference.cpp
2024-04-05T05:59:31.9952196Z
2024-04-05T05:59:31.9952200Z
2024-04-05T05:59:31.9952296Z Testing Time: 1720.15s
2024-04-05T05:59:31.9952473Z
2024-04-05T05:59:31.9952581Z Total Discovered Tests: 126112
2024-04-05T05:59:31.9952881Z Skipped : 77 (0.06%)
2024-04-05T05:59:31.9953189Z Unsupported : 4844 (3.84%)
2024-04-05T05:59:31.9953502Z Passed : 120830 (95.81%)
2024-04-05T05:59:31.9953818Z Expectedly Failed: 335 (0.27%)
2024-04-05T05:59:31.9954128Z Unresolved : 1 (0.00%)
2024-04-05T05:59:31.9954428Z Failed : 25 (0.02%)
2024-04-05T05:59:32.6323217Z FAILED: CMakeFiles/check-all /home/runner/work/llvm-project/llvm-project/final/build/tools/clang/stage2-instrumented-bins/tools/clang/stage2-bins/CMakeFiles/check-all
On the releases page for 13.0.0 there is clang+llvm-13.0.0-x86_64-linux-gnu-ubuntu-20.04.tar.xz, but for 13.0.1 there's no corresponding binary. Is it possible to generate the same variants for 13.0.1 as 13.0.0?