search

firedrakeproject / firedrake

Firedrake is an automated system for the portable solution of partial differential equations using the finite element method (FEM)
https://firedrakeproject.org
Other
482 stars 157 forks source link

INSTALL: Script fails at PETSc while downloading HWLOC #3644

Open jonathonmckay opened 1 week ago

jonathonmckay commented 1 week ago

Describe the error The installation script fails at the "Installing petsc" phase, while the PETSc script is trying to download HWLOC (Specifically: https://web.cels.anl.gov/projects/petsc/download/externalpackages/hwloc-2.10.0.tar.gz)

Steps to Reproduce Following steps at https://www.firedrakeproject.org/download.html Specifically:

curl -O https://raw.githubusercontent.com/firedrakeproject/firedrake/master/scripts/firedrake-install
python3 firedrake-install --verbose --no-package-manager --disable-ssh

Expected behavior The install script should run and download all required files

Error message Terminal Messages

Installing petsc/
Depending on your platform, PETSc may take an hour or more to build!
Traceback (most recent call last):
  File "/minerva/source/firedrake/firedrake-install", line 1816, in <module>
    install("petsc/")
  File "/minerva/source/firedrake/firedrake-install", line 1043, in install
    build_and_install_petsc()
  File "/minerva/source/firedrake/firedrake-install", line 1155, in build_and_install_petsc
    check_call([python, "./configure", "PETSC_DIR={}".format(petsc_dir), "PETSC_ARCH={}".format(petsc_arch)] + petsc_options)
  File "/minerva/source/firedrake/firedrake-install", line 672, in check_call
    log.debug(subprocess.check_output(arguments, stderr=subprocess.STDOUT, env=os.environ).decode())
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/minerva/opt/python-3.11.6/lib/python3.11/subprocess.py", line 466, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/minerva/opt/python-3.11.6/lib/python3.11/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/minerva/source/firedrake/firedrake/bin/python', './configure', 'PETSC_DIR=/minerva/sourc'PETSC_ARCH=default', '--with-c2html=0', '--with-fortran-bindings=0', '--download-bison', '--with-zlib', '--download-netcdf', '--dotix', '--download-superlu_dist', '--download-mpich', '--download-suitesparse', '--download-pnetcdf', '--download-ptscotch', '--down-download-hypre', '--with-debugging=0', '--download-hwloc', '--with-shared-libraries=1', '--download-metis', '--download-cmake']' r

firedrake-install.log (Tail end)

=============================================================================================
                 Running make install on BISON; this may take several minutes
=============================================================================================
=============================================================================================
  Trying to download https://download.open-mpi.org/release/hwloc/v2.10/hwloc-2.10.0.tar.gz
  for HWLOC
=============================================================================================
=============================================================================================
  Trying to download
  https://web.cels.anl.gov/projects/petsc/download/externalpackages/hwloc-2.10.0.tar.gz for
  HWLOC
=============================================================================================

*********************************************************************************************
           UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details):
---------------------------------------------------------------------------------------------
  Error during download/extract/detection of HWLOC:
  Unable to download package HWLOC from:
  https://download.open-mpi.org/release/hwloc/v2.10/hwloc-2.10.0.tar.gz
  * If URL specified manually - perhaps there is a typo?
  * If your network is disconnected - please reconnect and rerun ./configure
  * Or perhaps you have a firewall blocking the download
  * You can run with --with-packages-download-dir=/adirectory and ./configure will instruct
  you what packages to download manually
  * or you can download the above URL manually, to
  /yourselectedlocation/hwloc-2.10.0.tar.gz
  and use the configure option:
  --download-hwloc=/yourselectedlocation/hwloc-2.10.0.tar.gz
  Unable to download package HWLOC from:
  https://web.cels.anl.gov/projects/petsc/download/externalpackages/hwloc-2.10.0.tar.gz
  * If URL specified manually - perhaps there is a typo?
  * If your network is disconnected - please reconnect and rerun ./configure
  * Or perhaps you have a firewall blocking the download
  * You can run with --with-packages-download-dir=/adirectory and ./configure will instruct
  you what packages to download manually
  * or you can download the above URL manually, to
  /yourselectedlocation/hwloc-2.10.0.tar.gz
  and use the configure option:
  --download-hwloc=/yourselectedlocation/hwloc-2.10.0.tar.gz
*********************************************************************************************

Environment:

Additional context The link that fails is https://web.cels.anl.gov/projects/petsc/download/externalpackages/hwloc-2.10.0.tar.gz I have tried this URL on several computers. All return a "404".

connorjward commented 1 week ago

There appear to be two links that they search to find the URL. The former (https://download.open-mpi.org/release/hwloc/v2.10/hwloc-2.10.0.tar.gz) appears to work. It's possible that that server or your network experienced a temporary issue. Can you please try again?

jonathonmckay commented 1 week ago

Unfortunately, I have been trying this off an on for the last month or so, hoping that I just caught someone with a missing file. It's never worked.

I have also noticed that it is trying two places. The download.open-mpi.org URL works when I try it manually,but the script seems to be hung up on the web.cels.anl.gov URL not working.

Also the help message from the log says Unable to download package HWLOC from: https://download.open-mpi.org/release/hwloc/v2.10/hwloc-2.10.0.tar.gz, yet this is the URL that works, and was the one tried first (according to the order of the log). While the web.cels.anl.gov URL is the one the doesn't work, its not mentioned in the help message (even though the help message happens immediately after trying this URL (again, according to the order of the log)). The order of the messages is kinda odd.

connorjward commented 4 days ago

Strange. Do you have a non-standard network setup? Or some sort of firewall?

This is really an issue with PETSc instead of Firedrake so I would recommend asking for help there. If you want to reproduce the issue without using Firedrake you should be able to run ./configure --download-hwloc from inside the firedrake/src/petsc directory.

Lastly, we updated our fork of PETSc last Friday. I think that is unlikely to have fixed this issue, but you could try a fresh install and see if things work.

jonathonmckay commented 4 days ago

We do in-fact have non-standard network with a very annoying firewall. But this is a know problem with this sort of automatic install script, so I tried downloading the failing hwloc file with a computer on the guest network, and my personal phone. All got the same problem.

Yeah, I know this problem is ultimately in the script provided by PETSc, but I didn't know how much control your script was exerting over it. I will check their forums. I'll post here if I get any meaningful information there for completeness.

While I've been trying this for over a month, I downloaded a new "firedrake-install" script on 2024-06-21.

connorjward commented 4 days ago

We do in-fact have non-standard network with a very annoying firewall. But this is a know problem with this sort of automatic install script, so I tried downloading the failing hwloc file with a computer on the guest network, and my personal phone. All got the same problem.

Yeah, I know this problem is ultimately in the script provided by PETSc, but I didn't know how much control your script was exerting over it. I will check their forums. I'll post here if I get any meaningful information there for completeness.

While I've been trying this for over a month, I downloaded a new "firedrake-install" script on 2024-06-21.

It is also possible to install PETSc separately from the rest of the installation script. If you run firedrake-install --show-petsc-configure-options you can see what flags we pass to PETSc. It is quite annoying and manual but you could then clone your own PETSc, download all of the packages manually and then run something like

$ ./configure --hwloc-dir=/path/to/downloaded/tar/file ...etc

I don't really recommend this though. Figuring out how to satisfy the firewall is definitely a better option.

Alternatively we do have Docker and Singularity containers that might be suitable.

balay commented 4 days ago

Actually the option here is:

$ ./configure --download-hwloc=/path/to/downloaded/tar/file

Alternative:

  --with-packages-download-dir=<dir>
       Skip network download of package tarballs and locate them in specified dir. If not found in dir, print package URL - so it can be obtained manually.