MESA r15140 dependencies won't download

[LourensVeen]

Describe the bug I'm trying to install MESA r15140, and it fails, seemingly because some of the dependencies it tries to download have vanished off the internet.

To Reproduce Install an Amuse development build with everything else into a virtualenv, then inside the same venv: $ DOWNLOAD_CODES=1 make mesa_r15140.code

Expected behavior MESA to be downloaded, built and installed (over the next couple hours :smiley:)

Logs src/amuse/community/mesa_r15140/download.py tries and fails to download the following (they all 404):

http://www.astro.wisc.edu/~townsend/resource/download/sdk2/src/lapack95.tgz http://www.astro.wisc.edu/~townsend/resource/download/sdk2/src/fpx3.tar.gz http://www.astro.wisc.edu/~townsend/resource/download/sdk2/src/crlibm-1.0beta4.tar.gz

It then crashes because it cannot patch the downloaded codes, but the root of the problem is that they're gone.

Environment (please complete the following information):

• OS and version: Kubuntu 22.04
• Compiler: gcc 11.4.0

Additional context I've found a mirror of crlibm at https://github.com/taschini/crlibm, from which one can download the code using https://github.com/taschini/crlibm/archive/refs/heads/master.tar.gz. This claims to be version 1.0beta5, updated from 1.0beta3. Not sure what happened to beta4 there, except that it apparently used to be in Wisconsin.

There are many things called fpx3 online, but it appears to be a Fortran preprocessor. There's a link to it at https://fortranwiki.org/fortran/show/Fpx3, but that gives me a 403 Forbidden.

Perhaps Mr. Townsend could be prevailed on to put the files back where they were? Or maybe we could mirror them somewhere, if we can get a copy and that's possible license-wise?

[rieder]

I'm in favour of mirroring the files to somewhere we can be sure they'll exist years from now (this goes for all codes that require something to be downloaded).

[LourensVeen]

Yeah. Zenodo may be a good idea, if the files are small, and I'm not sure whether you can upload someone else's stuff there. I'll check.

I think this is a more general issue, is it? We have all these community codes embedded, and they sometimes embed further dependencies and sometimes download them on installation, and many of these codes are old and require old and sometimes somewhat exotic tools to build. I'm not sure how many of them are still maintained, but they're also essentially forks, and adding new versions is a lot of work, as with any packaging problem. I also wonder how many copies of BLAS and LAPACK we have exactly...

I don't see an obvious solution though. Containers could work pretty well, but would require people to install Docker to use Amuse, and have a cluster that supports Singularity/Apptainer, and there would definitely be some tweaking to get MPI to work efficiently across containers. Conda is fairly easy to install and already used by many people, and it can do binary packages as well as Python, but AFAIK it doesn't do MPI and I don't know whether that is solveable. Homebrew and MacPorts and pkgsrc and Fink are MacOS only, EasyBuild is interesting and can be installed on a laptop, but it takes forever because you need to build the entire toolchain from scratch, and then there's GUIX and Nix and similar, which I'd have to look into but which are also yet another dependency.

There's an issue about using Meson, which I think is an interesting idea that could replace the whole amalgamation of setuptools, Python scripts, shell scripts, Makefiles, CMake, and who knows what else that builds the whole thing now. Maybe combined with putting all the community codes and their dependencies somewhere where they won't disappear unexpectedly could work then, although at some point some of the codes will be old enough that a container with a similarly obsolete OS environment is the only thing that will still work. And getting all this stuff to build from a single Meson configuration would probably be quite a lot of work too.

Meanwhile, there seems to be a copy of fpx3 at https://github.com/lesgo-jhu/dependencies-lesgo/tree/master/fpx3-0.20, although I'm not sure if it's the same version or whether it'll work. I'll keep digging :smile:

[GijsVermarien]

To weigh in, I just managed to build 28/55 codes on arm64 with conda openmpi, so maybe we can do MPI with conda? See this package on conda-forge: https://anaconda.org/conda-forge/openmpi

[rieder]

@GijsVermarien Which codes failed to build? 28/55 is not terribly successful... Also, were you able to run tests successfully?

[LourensVeen]

I think you're right about MPI and Conda being possible, I've recently been working with a package called hoomd which has issues, but it could be just that package and not a general problem. HPC may still be an issue though, if you're on a machine that requires a specific MPI implementation and won't work with plain OpenMPI. This Cray machine for example: https://hpc.ncsu.edu/Software/Apps.php?app=Conda-MPI

SURF also recommends against using Conda on their HPC machines, and instead say you should compile everything locally to get every last bit of performance, but honestly I think convenience may trump that.

I seem to have MESA compiled now, but I haven't found drop-in replacements for the missing tarballs so it's half-manual and a bit of a mess. Maybe we should just make a separate repository amusecode/amuse-dependencies where we mirror all the stuff it downloads?

[GijsVermarien]

I think conda would mostly be a nice use case for macbooks, but if macports works better that would also be fine (but currently I cannot find the documentation on how to make this work). I agree with the MPI problem, but in my personal opinion not every AMUSE user needs more than just one machine, so MPI is a bit simpler. For HPC I agree with Surf' view that you should compile to squeeze out all the performance.

Find below my latest installation of the packages with conda on Apple Silicon. Probably some more elbow grease is needed to compile a few more codes. But for the AMUSE course, the installation covers all packages but Fi (see #982), so it would be ideal as many students are familiar with conda. amuse-apple-silicon-conda.txt

[rieder]

I see that codes like bhtree and hop failed to build- that’s really not good. Those are ‘core’ packages… can you find out why these failed to build?

[rieder]

I asked the folks at MESA about the missing packages.

[rieder]

fixed by #986