search

pnnl / ExaGO

High-performance power grid optimization for stochastic, security-constrained, and multi-period ACOPF problems.
Other
64 stars 8 forks source link

issues installing hiop+sparse and ipopt+coinhsl #124

Closed nedpfeiffer closed 2 months ago

nedpfeiffer commented 7 months ago

Issue type

Relates to

Summary

Apologies if this isn't the right place to troubleshoot this issue, as it seems more related to Spack than to the ExaGO project. Regardless, the installing_with_spack.md documentation seems like it's a bit outdated.

When I attempt to install CoinHSL dependencies with spack install hiop+sparse and spack install ipopt+coinhsl I get the following errors:

ubuntu@workstation:~$ ls | grep coinhsl
coinhsl-archive-2023.11.17.tar.gz
ubuntu@workstation:~$ spack install hiop+sparse
[+] /home/ubuntu/spack/opt/spack/linux-ubuntu22.04-skylake/gcc-11.4.0/gcc-runtime-11.4.0-vfbxhvvvqok2icbiykzq3fupmz4oizzz
[+] /home/ubuntu/spack/opt/spack/linux-ubuntu22.04-skylake/gcc-11.4.0/gmake-4.4.1-echlqtvk7cd6yqexjoly3zqwtpogvsqo
[+] /home/ubuntu/spack/opt/spack/linux-ubuntu22.04-skylake/gcc-11.4.0/metis-5.1.0-n6pcbnuc4bwoehxb4dzrj4j42ahaq7zv
[+] /home/ubuntu/spack/opt/spack/linux-ubuntu22.04-skylake/gcc-11.4.0/zstd-1.5.5-fkspl7nljnx73w4mocpgfvkkot73lghq
[+] /home/ubuntu/spack/opt/spack/linux-ubuntu22.04-skylake/gcc-11.4.0/libiconv-1.17-xvvr34ffol76epxa5arrgy3uchubanwa
[+] /home/ubuntu/spack/opt/spack/linux-ubuntu22.04-skylake/gcc-11.4.0/zlib-1.3-5scpb5gvxq7yh5hugfbkfdevsukh7ooe
[+] /home/ubuntu/spack/opt/spack/linux-ubuntu22.04-skylake/gcc-11.4.0/nghttp2-1.57.0-25g474kpy6wfl4hayd4q7quoagxgzm2m
[+] /home/ubuntu/spack/opt/spack/linux-ubuntu22.04-skylake/gcc-11.4.0/xz-5.4.6-e6rkvbgwyeo77qicmu4ddwwwvw3x5lfb
[+] /home/ubuntu/spack/opt/spack/linux-ubuntu22.04-skylake/gcc-11.4.0/libxcrypt-4.4.35-pni3qdbpxmgu6xfjuprgaylqpxniyjdc
[+] /home/ubuntu/spack/opt/spack/linux-ubuntu22.04-skylake/gcc-11.4.0/ncurses-6.4-cmrehw7viwlh36wnu6rkvtiljbti5ems
[+] /home/ubuntu/spack/opt/spack/linux-ubuntu22.04-skylake/gcc-11.4.0/bzip2-1.0.8-zjctsiftp4xmbzz2yd7hc2hbwk4452y3
[+] /home/ubuntu/spack/opt/spack/linux-ubuntu22.04-skylake/gcc-11.4.0/libpciaccess-0.17-ydanjvfdy6ulpsqm4nyycnq57goteb2s
[+] /home/ubuntu/spack/opt/spack/linux-ubuntu22.04-skylake/gcc-11.4.0/numactl-2.0.14-aofoqec5ue637h65zueuloleh6ewwjuv
[+] /home/ubuntu/spack/opt/spack/linux-ubuntu22.04-skylake/gcc-11.4.0/openblas-0.3.26-qanhjryihop6q243llzqkbn7mqk2gbxu
[+] /home/ubuntu/spack/opt/spack/linux-ubuntu22.04-skylake/gcc-11.4.0/openssl-3.2.1-xile2ofo4urkm26xn76u7pltgifbevkl
[+] /home/ubuntu/spack/opt/spack/linux-ubuntu22.04-skylake/gcc-11.4.0/pigz-2.8-fej2sygumzye325n4j2qjopdeipe2jd2
[+] /home/ubuntu/spack/opt/spack/linux-ubuntu22.04-skylake/gcc-11.4.0/libxml2-2.10.3-yfkixayk6mwl3p2gaprupd3y2ckhvwyu
[+] /home/ubuntu/spack/opt/spack/linux-ubuntu22.04-skylake/gcc-11.4.0/libedit-3.1-20210216-k7a6zijg6a73nnmfguocf3uf7yuzlrm6
==> Installing coinhsl-2015.06.23-ig5vov43i7od4t56zfwknldjkgvaelhs [19/30]
==> No binary for coinhsl-2015.06.23-ig5vov43i7od4t56zfwknldjkgvaelhs found: installing from source
==> Error: FetchError: Manual download is required for coinhsl. Refer to https://www.hsl.rl.ac.uk/ipopt/ for download instructions.
==> Warning: Skipping build of hiop-1.0.3-64f3enpgyyvdwyqnwgvfjth4ck3ep7n2 since coinhsl-2015.06.23-ig5vov43i7od4t56zfwknldjkgvaelhs failed
[+] /home/ubuntu/spack/opt/spack/linux-ubuntu22.04-skylake/gcc-11.4.0/curl-8.6.0-54umrtz6jnnugys7eoc2znv7q5bpnxw5
[+] /home/ubuntu/spack/opt/spack/linux-ubuntu22.04-skylake/gcc-11.4.0/libevent-2.1.12-mpzmde2g53hyejvr72o3u5dkhcp6kklb
[+] /home/ubuntu/spack/opt/spack/linux-ubuntu22.04-skylake/gcc-11.4.0/tar-1.34-q33ylueecqcufvlzsbrpjp5og6mkjaes
[+] /home/ubuntu/spack/opt/spack/linux-ubuntu22.04-skylake/gcc-11.4.0/hwloc-2.9.1-gpqlwvar2jymxwzrix5o7hn5xro6lqts
[+] /home/ubuntu/spack/opt/spack/linux-ubuntu22.04-skylake/gcc-11.4.0/cmake-3.27.9-ezbxlzaiz4zagtx7knuqy3dpcrh2uw2x
[+] /home/ubuntu/spack/opt/spack/linux-ubuntu22.04-skylake/gcc-11.4.0/gettext-0.22.4-hj2jj6hrfwjldi27tjxwszu62ui72zpw
[+] /home/ubuntu/spack/opt/spack/linux-ubuntu22.04-skylake/gcc-11.4.0/pmix-5.0.1-72uhenofnj5lcadum2rzb3wma2qwhdy5
[+] /home/ubuntu/spack/opt/spack/linux-ubuntu22.04-skylake/gcc-11.4.0/krb5-1.20.1-hm3svss6ofiuj5iw7yv7bhtn27ctuikp
[+] /home/ubuntu/spack/opt/spack/linux-ubuntu22.04-skylake/gcc-11.4.0/openssh-9.5p1-aei4csmm4csausigzqtw7lvhb5yy7tlp
[+] /home/ubuntu/spack/opt/spack/linux-ubuntu22.04-skylake/gcc-11.4.0/openmpi-5.0.2-vjugycwppwkoidwnd55rqnaln2sbdclg
==> Error: hiop-1.0.3-64f3enpgyyvdwyqnwgvfjth4ck3ep7n2: Package was not installed
==> Error: Installation request failed.  Refer to reported errors for failing package(s).

Attempting a hacky workaround, the checksum fails.

ubuntu@workstation:~$ mv coinhsl-archive-2023.11.17.tar.gz coinhsl-archive-2015.06.23.tar.gz
ubuntu@workstation:~$ spack install hiop+sparse
[+] /home/ubuntu/spack/opt/spack/linux-ubuntu22.04-skylake/gcc-11.4.0/gcc-runtime-11.4.0-vfbxhvvvqok2icbiykzq3fupmz4oizzz
[+] /home/ubuntu/spack/opt/spack/linux-ubuntu22.04-skylake/gcc-11.4.0/libiconv-1.17-xvvr34ffol76epxa5arrgy3uchubanwa
[+] /home/ubuntu/spack/opt/spack/linux-ubuntu22.04-skylake/gcc-11.4.0/libpciaccess-0.17-ydanjvfdy6ulpsqm4nyycnq57goteb2s
[+] /home/ubuntu/spack/opt/spack/linux-ubuntu22.04-skylake/gcc-11.4.0/metis-5.1.0-n6pcbnuc4bwoehxb4dzrj4j42ahaq7zv
[+] /home/ubuntu/spack/opt/spack/linux-ubuntu22.04-skylake/gcc-11.4.0/xz-5.4.6-e6rkvbgwyeo77qicmu4ddwwwvw3x5lfb
[+] /home/ubuntu/spack/opt/spack/linux-ubuntu22.04-skylake/gcc-11.4.0/nghttp2-1.57.0-25g474kpy6wfl4hayd4q7quoagxgzm2m
[+] /home/ubuntu/spack/opt/spack/linux-ubuntu22.04-skylake/gcc-11.4.0/numactl-2.0.14-aofoqec5ue637h65zueuloleh6ewwjuv
[+] /home/ubuntu/spack/opt/spack/linux-ubuntu22.04-skylake/gcc-11.4.0/openblas-0.3.26-qanhjryihop6q243llzqkbn7mqk2gbxu
[+] /home/ubuntu/spack/opt/spack/linux-ubuntu22.04-skylake/gcc-11.4.0/zlib-1.3-5scpb5gvxq7yh5hugfbkfdevsukh7ooe
[+] /home/ubuntu/spack/opt/spack/linux-ubuntu22.04-skylake/gcc-11.4.0/ncurses-6.4-cmrehw7viwlh36wnu6rkvtiljbti5ems
[+] /home/ubuntu/spack/opt/spack/linux-ubuntu22.04-skylake/gcc-11.4.0/libxcrypt-4.4.35-pni3qdbpxmgu6xfjuprgaylqpxniyjdc
[+] /home/ubuntu/spack/opt/spack/linux-ubuntu22.04-skylake/gcc-11.4.0/bzip2-1.0.8-zjctsiftp4xmbzz2yd7hc2hbwk4452y3
[+] /home/ubuntu/spack/opt/spack/linux-ubuntu22.04-skylake/gcc-11.4.0/zstd-1.5.5-fkspl7nljnx73w4mocpgfvkkot73lghq
[+] /home/ubuntu/spack/opt/spack/linux-ubuntu22.04-skylake/gcc-11.4.0/gmake-4.4.1-echlqtvk7cd6yqexjoly3zqwtpogvsqo
[+] /home/ubuntu/spack/opt/spack/linux-ubuntu22.04-skylake/gcc-11.4.0/libxml2-2.10.3-yfkixayk6mwl3p2gaprupd3y2ckhvwyu
[+] /home/ubuntu/spack/opt/spack/linux-ubuntu22.04-skylake/gcc-11.4.0/pigz-2.8-fej2sygumzye325n4j2qjopdeipe2jd2
[+] /home/ubuntu/spack/opt/spack/linux-ubuntu22.04-skylake/gcc-11.4.0/openssl-3.2.1-xile2ofo4urkm26xn76u7pltgifbevkl
==> Installing coinhsl-2015.06.23-ig5vov43i7od4t56zfwknldjkgvaelhs [18/30]
==> No binary for coinhsl-2015.06.23-ig5vov43i7od4t56zfwknldjkgvaelhs found: installing from source
==> Fetching file:///home/ubuntu/coinhsl-archive-2015.06.23.tar.gz
==> Error: ChecksumError: sha256 checksum failed for /tmp/ubuntu/spack-stage/spack-stage-coinhsl-2015.06.23-ig5vov43i7od4t56zfwknldjkgvaelhs/coinhsl-archive-2015.06.23.tar.gz
    Expected 3e955a2072f669b8f357ae746531b37aea921552e415dc219a5dd13577575fb3 but got 43438fb9317dd4648625a6f5dd46ffedf1d33bd47d05885805b651fe93729065. File size = 1029574 bytes. Contents = b'\x1f\x8b\x08\x08\xbd\xacWe\x02\xffcoinhs...\xa8\xa0\x82\n?&\xfc\x07\x1eR\xdcO\x00\x08R\x00'
==> Warning: Skipping build of hiop-1.0.3-64f3enpgyyvdwyqnwgvfjth4ck3ep7n2 since coinhsl-2015.06.23-ig5vov43i7od4t56zfwknldjkgvaelhs failed
[+] /home/ubuntu/spack/opt/spack/linux-ubuntu22.04-skylake/gcc-11.4.0/libedit-3.1-20210216-k7a6zijg6a73nnmfguocf3uf7yuzlrm6
[+] /home/ubuntu/spack/opt/spack/linux-ubuntu22.04-skylake/gcc-11.4.0/hwloc-2.9.1-gpqlwvar2jymxwzrix5o7hn5xro6lqts
[+] /home/ubuntu/spack/opt/spack/linux-ubuntu22.04-skylake/gcc-11.4.0/tar-1.34-q33ylueecqcufvlzsbrpjp5og6mkjaes
[+] /home/ubuntu/spack/opt/spack/linux-ubuntu22.04-skylake/gcc-11.4.0/libevent-2.1.12-mpzmde2g53hyejvr72o3u5dkhcp6kklb
[+] /home/ubuntu/spack/opt/spack/linux-ubuntu22.04-skylake/gcc-11.4.0/curl-8.6.0-54umrtz6jnnugys7eoc2znv7q5bpnxw5
[+] /home/ubuntu/spack/opt/spack/linux-ubuntu22.04-skylake/gcc-11.4.0/gettext-0.22.4-hj2jj6hrfwjldi27tjxwszu62ui72zpw
[+] /home/ubuntu/spack/opt/spack/linux-ubuntu22.04-skylake/gcc-11.4.0/pmix-5.0.1-72uhenofnj5lcadum2rzb3wma2qwhdy5
[+] /home/ubuntu/spack/opt/spack/linux-ubuntu22.04-skylake/gcc-11.4.0/cmake-3.27.9-ezbxlzaiz4zagtx7knuqy3dpcrh2uw2x
[+] /home/ubuntu/spack/opt/spack/linux-ubuntu22.04-skylake/gcc-11.4.0/krb5-1.20.1-hm3svss6ofiuj5iw7yv7bhtn27ctuikp
[+] /home/ubuntu/spack/opt/spack/linux-ubuntu22.04-skylake/gcc-11.4.0/openssh-9.5p1-aei4csmm4csausigzqtw7lvhb5yy7tlp
[+] /home/ubuntu/spack/opt/spack/linux-ubuntu22.04-skylake/gcc-11.4.0/openmpi-5.0.2-vjugycwppwkoidwnd55rqnaln2sbdclg
==> Error: hiop-1.0.3-64f3enpgyyvdwyqnwgvfjth4ck3ep7n2: Package was not installed
==> Error: Installation request failed.  Refer to reported errors for failing package(s).

I encounter the same issues with spack install ipopt+coinhsl

Thank you for your help, I really appreciate it!

jaelynlitz commented 6 months ago

Hi @nedpfeiffer - have you gotten your licensed download from CoinHSL? Even when installing via spack, it requires the manual download of the CoinHSL tarball from here. The academic license is free 😄 You may have to go through the CoinHSL Archive - as the newest version we have tested with is 2019.05.21.

==> Installing coinhsl-2015.06.23-ig5vov43i7od4t56zfwknldjkgvaelhs [19/30]
==> No binary for coinhsl-2015.06.23-ig5vov43i7od4t56zfwknldjkgvaelhs found: installing from source
==> Error: FetchError: Manual download is required for coinhsl. Refer to https://www.hsl.rl.ac.uk/ipopt/ for download instructions.
jaelynlitz commented 6 months ago

Hi @nedpfeiffer - have you gotten your licensed download from CoinHSL? Even when installing via spack, it requires the manual download of the CoinHSL tarball from here. The academic license is free 😄 You may have to go through the CoinHSL Archive - as the newest version we have tested with is 2019.05.21.

==> Installing coinhsl-2015.06.23-ig5vov43i7od4t56zfwknldjkgvaelhs [19/30]
==> No binary for coinhsl-2015.06.23-ig5vov43i7od4t56zfwknldjkgvaelhs found: installing from source
==> Error: FetchError: Manual download is required for coinhsl. Refer to https://www.hsl.rl.ac.uk/ipopt/ for download instructions.

Whoops missed the grep at the top of your snippet, try the 2019 version 😄

nedpfeiffer commented 6 months ago

I could certainly be missing something, but I looked everywhere and couldn't find CoinHSL 2019.05.21 available for download.

Screenshot from 2024-03-20 09-28-09

Regardless, I was able to use the oldest available version (2022.11.09) to build a minimal install with IPOPT.

ubuntu@workstation:~$ ls | grep coinhsl
coinhsl-archive-2022.11.09.tar.gz
ubuntu@workstation:~$ source /home/ubuntu/spack/share/spack/setup-env.sh
ubuntu@workstation:~$ spack install exago@develop%gcc \
  ^openmpi ^ipopt@3.12.10+coinhsl~mumps ^coinhsl+blas \
  ^petsc+mpi~hypre~superlu-dist~mumps+shared
[+] /home/ubuntu/spack/opt/spack/linux-ubuntu22.04-skylake/gcc-11.4.0/gcc-runtime-11.4.0-vfbxhvvvqok2icbiykzq3fupmz4oizzz
.
.
.
==> Installing exago-develop-md6imm7robavq5qxhsfd7r6roqqo6eai [50/50]
==> No binary for exago-develop-md6imm7robavq5qxhsfd7r6roqqo6eai found: installing from source
==> No patches needed for exago
==> exago: Executing phase: 'cmake'
==> exago: Executing phase: 'build'
==> exago: Executing phase: 'install'
==> exago: Successfully installed exago-develop-md6imm7robavq5qxhsfd7r6roqqo6eai
  Stage: 1m 32.60s.  Cmake: 2.36s.  Build: 22.46s.  Install: 0.37s.  Post-install: 0.06s.  Total: 1m 58.29s
[+] /home/ubuntu/spack/opt/spack/linux-ubuntu22.04-skylake/gcc-11.4.0/exago-develop-md6imm7robavq5qxhsfd7r6roqqo6eai
ubuntu@workstation:~$ spack load exago
ubuntu@workstation:~$ opflow -help
ExaGO 1.6.0 built on Mar 20 2024
.
.
.

I'm worried about future incompatibility issues, but for now this issue can probably be closed. Thank you for your help!

nedpfeiffer commented 6 months ago

Apologies, re-opening this issue.

While I was able to build ExaGO with Spack and CoinHSL 2022.11.09, I am now receiving the following error:

dev@ubuntu-vm:~/ExaGO/datafiles$ opflow -netfile case_ACTIVSg2000.m
[ExaGO] Creating OPFlow

Exception of type: OPTION_INVALID in file "/tmp/dev/spack-stage/spack-stage-ipopt-3.12.10-jaiiuircitf5t6zqlfjgvhowknv6exkq/spack-src/Ipopt/src/Algorithm/IpAlgBuilder.cpp" at line 271:
 Exception message: Selected linear solver MA27 not available.
Tried to obtain MA27 from shared library "libhsl.so", but the following error occured:
libhsl.so: cannot open shared object file: No such file or directory

EXIT: Invalid option encountered.
[ExaGO] Finalizing opflow application.

Thank you for your help and patience in helping me troubleshoot this issue.

cameronrutherford commented 6 months ago

Okay I have re-requested my own CoinHSL license just so I can go onto their page and poke around... If a download of that version is no longer supported, we might need to seriously re-visit our testing to make sure this works. I am pretty certain that there are links in other areas of the website to archived versions of CoinHSL that allow you to download older versions.

We use this pinned version in our testing because it is stable, but in theory the newer versions should work. The only caveat is that a newer Ipopt with a newer CoinHSL might cause different solver performance...

This is pushing the limits of my understanding of Ipopt, but I wonder if that older version of Ipopt is just breaking with the newer CoinHSL. Tagging @pelesh and @maksud to see if they have any ideas as they are more proficient in Ipopt than I.

cameronrutherford commented 6 months ago

Apologies, re-opening this issue.

While I was able to build ExaGO with Spack and CoinHSL 2022.11.09, I am now receiving the following error:

dev@ubuntu-vm:~/ExaGO/datafiles$ opflow -netfile case_ACTIVSg2000.m
[ExaGO] Creating OPFlow

Exception of type: OPTION_INVALID in file "/tmp/dev/spack-stage/spack-stage-ipopt-3.12.10-jaiiuircitf5t6zqlfjgvhowknv6exkq/spack-src/Ipopt/src/Algorithm/IpAlgBuilder.cpp" at line 271:
 Exception message: Selected linear solver MA27 not available.
Tried to obtain MA27 from shared library "libhsl.so", but the following error occured:
libhsl.so: cannot open shared object file: No such file or directory

EXIT: Invalid option encountered.
[ExaGO] Finalizing opflow application.

Thank you for your help and patience in helping me troubleshoot this issue.

Some other thoughts:

nychiang commented 6 months ago

Seems to me coinhsl and ipopt is not installed correctly, and hence ipopt cannot find coinhsl/hsl. Check if libcoinhsl.so is installed. If yes, create a symbolic link and name it as libhsl.so. Save it in the search path. This usually works, at least I use this setting for julia. @cameronrutherford I guess you mean building ipopt without hsl and using Re::Solve.

cameronrutherford commented 6 months ago

@nychiang yes... would be good to avoid hsl altogether

cameronrutherford commented 6 months ago

I just got my CoinHSL and CoinHSL Archive download links and can confirm that none of them have the older tarballs... Guess our hand is being forced here to upgrade Ipopt + CoinHSL... cc @abhyshr

nedpfeiffer commented 6 months ago

Seems to me coinhsl and ipopt is not installed correctly, and hence ipopt cannot find coinhsl/hsl. Check if libcoinhsl.so is installed. If yes, create a symbolic link and name it as libhsl.so. Save it in the search path. This usually works, at least I use this setting for julia. @cameronrutherford I guess you mean building ipopt without hsl and using Re::Solve.

Thank you so much, symlinking got things working. For any people struggling with the same issue in the future:

# Symlink libcoinhsl.so to the expected file name libhsl.so
sudo ln -s /home/ubuntu/spack/opt/spack/linux-ubuntu22.04-skylake/gcc-11.4.0/coinhsl-2022.11.09-l6f7jg3hmgpbjcg7p3jqwyu7aydkskzg/lib/libcoinhsl.so /home/ubuntu/spack/opt/spack/linux-ubuntu22.04-skylake/gcc-11.4.0/coinhsl-2022.11.09-l6f7jg3hmgpbjcg7p3jqwyu7aydkskzg/lib/libhsl.so

# Run ExaGO
opflow datafiles/case_ACTIVSg2000.m
nedpfeiffer commented 6 months ago

Sharing this email correspondence with the folks at STFC UKRI in case it is helpful:

Thank you for helping me work through this problem. The checksum is used in the Spack package manager. This GitHub issue explains the issue in more detail: https://github.com/pnnl/ExaGO/issues/124

I've found a temporary workaround, detailed on my blog: https://ned.vc/posts/exago-for-dummies/

If you are able to find coinhsl-2019-05-21.tar.gz​ please let me know, but in the meantime, I'll rely on my workaround.

Thank you again for your help.

Best, Ned

Their response:

Hi Ned,

Sorry I must have missed that nuance! Thanks for feeding this back, it will be useful if anyone else asks about it. I'm not well acquainted with Spack, but looking the config, I imagine it will be tricky to update to use our newer build system while keeping backward compatibility. We stopped using the autotools system that it is built for as it was getting quite hard to maintain and use Meson now. I'll have a look at if there's something we can do from our end to help out the Spack users.

Your temporary workaround with adding the checksum to the file is probably the best way to do it. We offer a checksum file on our licencing page if you'd rather use that than calculate the sum.

As for the symlink issue you encountered, it's a bug that was introduced by our build script when we edited it to allow for prebuilt binaries. The file libhsl.so should exist but the link is an absolute path so will be broken on all machines but the system I tested it on... I'm planning to make a new release soon to fix this, but we haven't received any bug reports, I suppose people have found and shared the workaround without reaching out.

Hopefully we'll be able to look at submitting a PR to update the Spack offering but I wont promise anything just yet (as I've only had a quick look at what's going on there!).

Kind regards,

AndrewLister-STFC commented 4 months ago

Hi folks, I work in the HSL team and have taken on the maintainer role for the CoinHSL spack recipe. I've updated the recipe to work with the 2023+ versions (though it highlighted a bug so it's really the 2024.05.15+ versions...).

Since I know it's used by people here, I thought I'd ask if anyone is happy to test it ~and leave a comment on the PR (https://github.com/spack/spack/pull/43610)?~ to see if it fixed this issue.

~If it gets approved this issue might be fixed?~

Edit: It's now merged