Clarify dependencies required

nazar-pc commented 8 months ago

I am trying to get GitHub actions and container image builds to succeed and looks like documentation right now is not quite accurate in terms of dependencies necessary.

About Linux it current says:

In addition to a valid C build environment, this requires autotools on Unices and CMake on Windows

While I noticed that for Ubuntu automake,libtool and pkg-config are all necessary for compilation.

Didn't get to macOS yet, but I expect similar surprises there.

Would be great to make documentation more exhaustive regarding dependencies.

UPD: libtool and pkg-config are also needed on macOS.

nazar-pc commented 8 months ago

Also just found this on https://www.open-mpi.org/projects/hwloc/doc/v2.10.0/a00334.php:

Nightly development snapshots are available on the web site, they can be configured and built without any need for Git or GNU Autotools.

Which is interesting and would be very handy to not depend on extra stuff.

HadrienG2 commented 8 months ago

Regarding the initial issue, this is a result of upstream dependencies being poorly documented and github actions runners coming with lots of deps pre-installed.

If you have the time for it (otherwise I'll get to it at some point), the right thing to do for Linux would be to start from a minimal docker image (we're talking about one of those images weighting tens of megabytes that don't even have bash), then...

Start by building hwloc on its own. Carefully inspect the output of the configure script, to know when optional features are being silently disabled. Add dependencies one by one, taking notes on what you've added, until configure fully succeeds, then cross-check that make goes smoothly and output binaries like lstopo run.
Once hwloc itself builds and runs, hwlocality's vendored feature should work without issues. But check it anyway to be sure.
At the end of this process you will know which dependencies are needed, both for core hwloc features and for optional features like PCI device enumeration.

Concerning macOS, I'm pretty sure you will need more than libtool, at least pkg-config and probably the Xcode developer tools or similar as well. But I'm not sure how to get a minimal macOS VM to check to true requirements.

As for the nightly snapshots, it would be risky to use those because as a result of being nightly snapshots, they will break when hwloc introduces API-breaking changes that hwlocality has not yet been adapted for (which is not a hypothetical possibility with hwloc v3 on the horizon). It would be better to use the per-release source tarballs, but the problem with those is that AFAIK there is no way to request the latest tarball from the 2.x series (unlike with git which has branches for that), so I would need to hardcode a version number in the build.rs file and keep it up to date, which is not ideal.

Ideas are welcome!

nazar-pc commented 8 months ago

I am starting with ubuntu:20.04 and installing packages. I also use self-hosted runners all 3 platforms, so I don't have as much software installed there as GitHub does.

Concerning macOS, I'm pretty sure you will need more than libtool, at least pkg-config and probably the Xcode developer tools or similar as well. But I'm not sure how to get a minimal macOS VM to check to true requirements.

Yes, Xcode, C toolchain and other typical stuff are expected to be there, I'm just taking about less common dependencies.

As for the nightly snapshots, it would be risky to use those because as a result of being nightly snapshots, they will break when hwloc introduces API-breaking changes that hwlocality has not yet been adapted for (which is not a hypothetical possibility with hwloc v3 on the horizon). It would be better to use source tarballs, but the problem with those is that AFAIK there is no way to request the latest tarball from the 2.x series (unlike with git which has branches for that), so I would need to hardcode a version number in the build.rs file and keep it up to date, which is not ideal.

I was just curious how they do it and maybe their release tarballs on the website are also like that, in which case it would be possible to pull them from website rather than GitHub. Still not clear how/why extra tooling is not necessary in that case :shrug:

HadrienG2 commented 8 months ago

Still not clear how/why extra tooling is not necessary in that case 🤷

That's how the autotools work. It's a bit like CMake: you start by writing a recipe for a configure script that sets some autotools configuration parameters, then you run some tool to generate the configure script, then your users will in turn run that configure script on the target platform to configure their build.

The tarballs on the open-mpi website have already gone through the first step of this process, but the git repo hasn't because the configure script is redundant with the autotools configuration files and thus storing both in version control is not good practice.

nazar-pc commented 8 months ago

I am horrified by CMake to be honest. Rarely have to deal with C/C++ and after I tried Meson a couple years ago I don't understand why people continue to suffer with other tools. But that isn't something that is easy to change upstream.

HadrienG2 / hwlocality

Clarify dependencies required #84