search

LLNL / H5Z-ZFP

A registered ZFP compression plugin for HDF5
Other
50 stars 22 forks source link

Update spack recipe to pin versions correctly #123

Closed AlexLong152 closed 1 year ago

AlexLong152 commented 1 year ago

Running spack install h5z-zfp gives the following log

==> h5z-zfp: Executing phase: 'edit'
==> h5z-zfp: Executing phase: 'build'
==> [2023-08-10-13:32:58.659792] 'make' '-j8' 'PREFIX=/home/alex/spack/opt/spack/linux-manjaro23-skylake/gcc-13.1.1/h5z-zfp-1.0.1-hceptxfyniryctra5xl6qvyt4bdly7a2' 'CC=/home/alex/spack/opt/spack/linux-manjaro23-skylake/gcc-13.1.1/openmpi-4.1.5-e74unvu5stl6a7xofrep47lsyytjud4f/bin/mpicc' 'HDF5_HOME=/home/alex/spack/opt/spack/linux-manjaro23-skylake/gcc-13.1.1/hdf5-1.14.1-2-olutqyj477ivhudzxiyaaex57h3d6ax7' 'ZFP_HOME=/home/alex/spack/opt/spack/linux-manjaro23-skylake/gcc-13.1.1/zfp-1.0.0-quqjakne3vm6hfusiharnwxtopxvflr2' 'FC=/home/alex/spack/opt/spack/linux-manjaro23-skylake/gcc-13.1.1/openmpi-4.1.5-e74unvu5stl6a7xofrep47lsyytjud4f/bin/mpif90' 'all'
grep: warning: stray \ before #
grep: /H5Zzfp_plugin.h: No such file or directory
cd src; make ZFP_HOME=/home/alex/spack/opt/spack/linux-manjaro23-skylake/gcc-13.1.1/zfp-1.0.0-quqjakne3vm6hfusiharnwxtopxvflr2  PREFIX=/home/alex/spack/opt/spack/linux-manjaro23-skylake/gcc-13.1.1/h5z-zfp-1.0.1-hceptxfyniryctra5xl6qvyt4bdly7a2 all
make[1]: Entering directory '/tmp/alex/spack-stage/spack-stage-h5z-zfp-1.0.1-hceptxfyniryctra5xl6qvyt4bdly7a2/spack-src/src'
grep: warning: stray \ before #
/home/alex/spack/opt/spack/linux-manjaro23-skylake/gcc-13.1.1/openmpi-4.1.5-e74unvu5stl6a7xofrep47lsyytjud4f/bin/mpicc -c H5Zzfp.c -o H5Zzfp_lib.o -DH5Z_ZFP_AS_LIB -fPIC -I. -I/home/alex/spack/opt/spack/linux-manjaro23-skylake/gcc-13.1.1/zfp-1.0.0-quqjakne3vm6hfusiharnwxtopxvflr2/include -I/home/alex/spack/opt/spack/linux-manjaro23-skylake/gcc-13.1.1/hdf5-1.14.1-2-olutqyj477ivhudzxiyaaex57h3d6ax7
/home/alex/spack/opt/spack/linux-manjaro23-skylake/gcc-13.1.1/openmpi-4.1.5-e74unvu5stl6a7xofrep47lsyytjud4f/bin/mpicc -c H5Zzfp_props.c -o H5Zzfp_props.o -fPIC -I. -I/home/alex/spack/opt/spack/linux-manjaro23-skylake/gcc-13.1.1/zfp-1.0.0-quqjakne3vm6hfusiharnwxtopxvflr2/include -I/home/alex/spack/opt/spack/linux-manjaro23-skylake/gcc-13.1.1/hdf5-1.14.1-2-olutqyj477ivhudzxiyaaex57h3d6ax7
/home/alex/spack/opt/spack/linux-manjaro23-skylake/gcc-13.1.1/openmpi-4.1.5-e74unvu5stl6a7xofrep47lsyytjud4f/bin/mpif90 -c H5Zzfp_props_f.F90 -o H5Zzfp_props_f.o  -I. -I/home/alex/spack/opt/spack/linux-manjaro23-skylake/gcc-13.1.1/zfp-1.0.0-quqjakne3vm6hfusiharnwxtopxvflr2/include -I/home/alex/spack/opt/spack/linux-manjaro23-skylake/gcc-13.1.1/hdf5-1.14.1-2-olutqyj477ivhudzxiyaaex57h3d6ax7
/home/alex/spack/opt/spack/linux-manjaro23-skylake/gcc-13.1.1/openmpi-4.1.5-e74unvu5stl6a7xofrep47lsyytjud4f/bin/mpicc -c H5Zzfp.c -o H5Zzfp_plugin.o -fPIC -I. -I/home/alex/spack/opt/spack/linux-manjaro23-skylake/gcc-13.1.1/zfp-1.0.0-quqjakne3vm6hfusiharnwxtopxvflr2/include -I/home/alex/spack/opt/spack/linux-manjaro23-skylake/gcc-13.1.1/hdf5-1.14.1-2-olutqyj477ivhudzxiyaaex57h3d6ax7
H5Zzfp.c:42:10: fatal error: bitstream.h: No such file or directory
   42 | #include "bitstream.h"
      |          ^~~~~~~~~~~~~
compilation terminated.
make[1]: *** [Makefile:11: H5Zzfp_lib.o] Error 1
make[1]: *** Waiting for unfinished jobs....
H5Zzfp.c:42:10: fatal error: bitstream.h: No such file or directory
   42 | #include "bitstream.h"
      |          ^~~~~~~~~~~~~
compilation terminated.
make[1]: *** [Makefile:7: H5Zzfp_plugin.o] Error 1
make[1]: Leaving directory '/tmp/alex/spack-stage/spack-stage-h5z-zfp-1.0.1-hceptxfyniryctra5xl6qvyt4bdly7a2/spack-src/src'
make: *** [Makefile:30: all] Error 2

Attempting to install without the fortran compiler with spack install h5z-zfp~fortran also fails.

Any advice on how to fix my install is appreciated.

markcmiller86 commented 1 year ago

I think I might need to know which version of spack you are running and which versioin of H5Z-ZFP you are trying to build with spack. If I go look at package.py file for H5Z-ZFP in spack, I see 

    version("develop", branch="master")
    version("1.0.1", sha256="b9ed91dab8e2ef82dc6706b4242c807fb352875e3b21c217dd00782dd1a22b24")
    version("0.8.0", sha256="a5eb089191369a5e929c51ec9e5da107afaee39c6ab3b7ad693c454319ab9217")
    version("0.7.0", sha256="f728b0bcb9e9cf8bafe05909ab02fec39415635d275e98b661176f69d34f87b3")

Can you maybe add @1.0.1 as in spack install h5z-zfp@1.0.1~fortran.

I will look into what is causing the grep commands to fail. Maybe grep needs to be included as a configure dependency.

markcmiller86 commented 1 year ago

Sorry, thats @1.0.1 not @1.01

markcmiller86 commented 1 year ago

Well, a closer look at your output indicates you are getting 1.0.1 version of H5Z-ZFP as in...

PREFIX=/home/alex/spack/opt/spack/linux-manjaro23-skylake/gcc-13.1.1/h5z-zfp-1.0.1-hceptxfyniryctra5xl6qvyt4bdly7a2
AlexLong152 commented 1 year ago

Thank you for your response.

$ spack --version
0.21.0.dev0 (37634f8b08067a1b6912e04bd23597135c83d4de)

grep was already installed on my system but I explicitly installed it with spack install grep and attempted the install again. I got the same errors as before.

As you suggest, I attempted running 

spack install h5z-zfp@1.0.1~fortran

and 

spack install h5z-zfp@1.0.1

but this gave the same error message as in my first post.

I am not an expert but it appears to me that grep is working, but I am actually missing some files. Is it possible I am missing a dependency? The missing files are H5Zzfp_plugin.h and bitstream.h.

markcmiller86 commented 1 year ago

My eye first caught the grep: warning: stray \ before # messages. I believe those are coming from lines in H5Z-ZFP's config.make file https://github.com/LLNL/H5Z-ZFP/blob/4ae30429c6c3913ad1e0688f92e374c8258412b3/config.make#L45-L55 and these lines appear to work fine on many other machines. What is your OS type and version? What about shell type and version?

Regarding missing bitstream.h, that almost seems like ZFP library itself is not installed correctly. I don't see anything in ZFP Library's Config file that might conditionally control installation of bitstream.h. Can you manually confirm there is a bitstream.h in /home/alex/spack/opt/spack/linux-manjaro23-skylake/gcc-13.1.1/zfp-1.0.0-quqjakne3vm6hfusiharnwxtopxvflr2/include

AlexLong152 commented 1 year ago

I am using Manjaro 23.0.0 with zsh as my shell.

bitstream.h is in include's sub-directory zfp 

~/spack/opt/spack/linux-manjaro23-skylake/gcc-13.1.1/zfp-1.0.0-quqjakne3vm6hfusiharnwxtopxvflr2/include/zfp >>>ls
codec  internal  array1.hpp  array2.hpp  array3.hpp  array4.hpp  array.h  array.hpp  bitstream.h  bitstream.inl  constarray1.hpp  constarray2.hpp  constarray3.hpp  constarray4.hpp  factory.hpp  index.hpp  version.h

Additionally, I found "H5Zzfp_plugin.h" but it is in the wrong place. It can be found in: /tmp/alex/spack-stage/spack-stage-h5z-zfp-1.0.1-hceptxfyniryctra5xl6qvyt4bdly7a2/spack-src/src/H5Zzfp_plugin.h

AlexLong152 commented 1 year ago

The grep not working in the make file is very strange because I can run it in my terminal with no problem when I substitute out $(ZFP_HOME).

$ grep '^\#define ZFP_VERSION_[MPT]' ~/spack/opt/spack/linux-manjaro23-skylake/gcc-13.1.1/zfp-1.0.0-quqjakne3vm6hfusiharnwxtopxvflr2/include/zfp/version.h 2>/dev/null | tr ' ' '\n' | grep '[0-9]' | tr -d '\n'

1:5:#define3:112:6:#define14:023:7:#define25:034:8:#define36:0%
markcmiller86 commented 1 year ago

The grep not working in the make file is very strange because I can run it in my terminal with no problem when I substitute out $(ZFP_HOME).

Well, is the shell being run from inside make the same shell you are running? And, is it finding the same grep there too?

To find out, you could construct a faux makefile that would print out what it think its shell is what it things its path to grep is.

markcmiller86 commented 1 year ago

Ok, I am having multiple challenges diagnosing...

  1. I have no idea what the command @some-path@ >>>ls is actually doing. No system I have access to interprets that correctly. I get Missing name for redirect. So, I am not trusting the list of files it produces. Are those all in include or are some in children of include. For example, my install of ZFP library version 1.0.0 shows only zfp zfp.h zfp.hpp and the zfp entry is a directory containing bitstream.h
  2. You appear to be using a development version of spack...not a released version. I am not entirely confident all dependencies involved here are getting handled correctly.
  3. Next, you are using H5Z-ZFP version 1.0.1 (would prefer you use 1.1.0) and that version pre-dates the release of ZFP library version 1.0.0 which is what you are trying to use here. So, you can't even use those two versions together.

Maybe try...

spack install h5z-zfp~fortran ^zfp@0.5.5
markcmiller86 commented 1 year ago

Or, maybe try a released version of spack

hgrie commented 1 year ago

I confirm AlexLong152's issue, but under a different OS:

LSB Version: core-11.1.0ubuntu2-noarch:printing-11.1.0ubuntu2-noarch:security-11.1.0ubuntu2-noarch Distributor ID: Ubuntu Description: Ubuntu 20.04.6 LTS Release: 20.04 Codename: focal

I used the installation instructions at https://h5z-zfp.readthedocs.io/en/latest/installation.html :

git clone https://github.com/llnl/spack.git . spack/share/spack/setup-env.sh <this should be "sh spack/share/spack/setup-env.sh"> spack install h5z-zfp

Resulting log file:

==> h5z-zfp: Executing phase: 'edit' ==> h5z-zfp: Executing phase: 'build' ==> [2023-08-12-15:13:24.975361] 'make' '-j8' 'PREFIX=/home/hgrie/spack/opt/spack/linux-ubuntu20.04-skylake/gcc-9.4.0/h5z-zfp-1.0.1-lz2b4txtaz3cbceyddjth4rptqgk5del' 'CC=/home/hgrie/spack/opt/spack/linux-ubuntu20.04-skylake/gcc-9.4.0/openmpi-4.1.5-ricuqzfgxm4qg3r6vr4brm73mq7eog2v/bin/mpicc' 'HDF5_HOME=/home/hgrie/spack/opt/spack/linux-ubuntu20.04-skylake/gcc-9.4.0/hdf5-1.14.1-2-asioe4rzjs7izv5togqrapqj6chotiyb' 'ZFP_HOME=/home/hgrie/spack/opt/spack/linux-ubuntu20.04-skylake/gcc-9.4.0/zfp-1.0.0-szxgwnppmwh6nccgr33sttw2youfvzjd' 'FC=/home/hgrie/spack/opt/spack/linux-ubuntu20.04-skylake/gcc-9.4.0/openmpi-4.1.5-ricuqzfgxm4qg3r6vr4brm73mq7eog2v/bin/mpif90' 'all' grep: /H5Zzfp_plugin.h: No such file or directory cd src; make ZFP_HOME=/home/hgrie/spack/opt/spack/linux-ubuntu20.04-skylake/gcc-9.4.0/zfp-1.0.0-szxgwnppmwh6nccgr33sttw2youfvzjd PREFIX=/home/hgrie/spack/opt/spack/linux-ubuntu20.04-skylake/gcc-9.4.0/h5z-zfp-1.0.1-lz2b4txtaz3cbceyddjth4rptqgk5del all make[1]: Entering directory '/tmp/hgrie/spack-stage/spack-stage-h5z-zfp-1.0.1-lz2b4txtaz3cbceyddjth4rptqgk5del/spack-src/src' /home/hgrie/spack/opt/spack/linux-ubuntu20.04-skylake/gcc-9.4.0/openmpi-4.1.5-ricuqzfgxm4qg3r6vr4brm73mq7eog2v/bin/mpicc -c H5Zzfp.c -o H5Zzfp_lib.o -DH5Z_ZFP_AS_LIB -fPIC -I. -I/home/hgrie/spack/opt/spack/linux-ubuntu20.04-skylake/gcc-9.4.0/zfp-1.0.0-szxgwnppmwh6nccgr33sttw2youfvzjd/include -I/home/hgrie/spack/opt/spack/linux-ubuntu20.04-skylake/gcc-9.4.0/hdf5-1.14.1-2-asioe4rzjs7izv5togqrapqj6chotiyb /home/hgrie/spack/opt/spack/linux-ubuntu20.04-skylake/gcc-9.4.0/openmpi-4.1.5-ricuqzfgxm4qg3r6vr4brm73mq7eog2v/bin/mpicc -c H5Zzfp_props.c -o H5Zzfp_props.o -fPIC -I. -I/home/hgrie/spack/opt/spack/linux-ubuntu20.04-skylake/gcc-9.4.0/zfp-1.0.0-szxgwnppmwh6nccgr33sttw2youfvzjd/include -I/home/hgrie/spack/opt/spack/linux-ubuntu20.04-skylake/gcc-9.4.0/hdf5-1.14.1-2-asioe4rzjs7izv5togqrapqj6chotiyb /home/hgrie/spack/opt/spack/linux-ubuntu20.04-skylake/gcc-9.4.0/openmpi-4.1.5-ricuqzfgxm4qg3r6vr4brm73mq7eog2v/bin/mpif90 -c H5Zzfp_props_f.F90 -o H5Zzfp_props_f.o -I. -I/home/hgrie/spack/opt/spack/linux-ubuntu20.04-skylake/gcc-9.4.0/zfp-1.0.0-szxgwnppmwh6nccgr33sttw2youfvzjd/include -I/home/hgrie/spack/opt/spack/linux-ubuntu20.04-skylake/gcc-9.4.0/hdf5-1.14.1-2-asioe4rzjs7izv5togqrapqj6chotiyb /home/hgrie/spack/opt/spack/linux-ubuntu20.04-skylake/gcc-9.4.0/openmpi-4.1.5-ricuqzfgxm4qg3r6vr4brm73mq7eog2v/bin/mpicc -c H5Zzfp.c -o H5Zzfp_plugin.o -fPIC -I. -I/home/hgrie/spack/opt/spack/linux-ubuntu20.04-skylake/gcc-9.4.0/zfp-1.0.0-szxgwnppmwh6nccgr33sttw2youfvzjd/include -I/home/hgrie/spack/opt/spack/linux-ubuntu20.04-skylake/gcc-9.4.0/hdf5-1.14.1-2-asioe4rzjs7izv5togqrapqj6chotiyb H5Zzfp.c:42:10: fatal error: bitstream.h: No such file or directory 42 | #include "bitstream.h" | ^~~~~ compilation terminated. make[1]: [Makefile:11: H5Zzfp_lib.o] Error 1 make[1]: Waiting for unfinished jobs.... H5Zzfp.c:42:10: fatal error: bitstream.h: No such file or directory 42 | #include "bitstream.h" | ^~~~~ compilation terminated. make[1]: [Makefile:7: H5Zzfp_plugin.o] Error 1 make[1]: Leaving directory '/tmp/hgrie/spack-stage/spack-stage-h5z-zfp-1.0.1-lz2b4txtaz3cbceyddjth4rptqgk5del/spack-src/src' make: [Makefile:30: all] Error 2

If this does not use the released version of spack, then maybe the instalation instructions need rewriting...

markcmiller86 commented 1 year ago

If this does not use the released version of spack, then maybe the instalation instructions need rewriting...

@hgrie what do you recommend?

I mean, do you think users need to be explicitly told that if they try a development version of spack and it doesn't work, then maybe they should try a released version?

IMHO, development versions are called development versions for a reason...they are still in development.

That said, as I look at the spack recipe for h5z-zfp, I think the crux of the problem is really there. We don't have sufficient logic pinning versions of zfp and h5z-zfp there causing a would-be user to have to know which versions to pin on a command line to spack install. We should update that logic in spack.

I've updated the title of this issue to reflect that.

hgrie commented 1 year ago

I think we all want the same thing: that the tool can be used widely, by the relatively average (scientific) user, without too much guesswork -- I am talking about using, not developing.

I fathom myself to be a relatively experienced user, but not developer, having grown up at a time when installing linux regularly needed quite a bit of tinkering on the command level (xf86config,...).

I followed the instructions at https://h5z-zfp.readthedocs.io/en/latest/installation.html to the letter. These do not address any difference between spack /development/ and /released/ version. That may be spack's problem, not h5z-zfp's, but having recognised the issue, I do think that users should be alerted to the issue and provided with a solution, command-line by command-line. And indeed, one cannot assume that the "average user" will know that the github clone is to the /development/ version, not to the /release/ version. Again, I think of myself as relatively literate, but I do not know how to get the /release/ version....

brtnfld commented 1 year ago

Hopefully, this issue will disappear when we switch to CMake builds for Spack.

markcmiller86 commented 1 year ago

Hopefully, this issue will disappear when we switch to CMake builds for Spack.

It will if people use CMake rather than spack to build it. If they use spack, we'll need to get Spack's recipe updated to pin versions properly.

hgrie commented 1 year ago

So no resolution, no interim advice how to install? We just wait this one out?

markcmiller86 commented 1 year ago

See last remark in this comment

markcmiller86 commented 1 year ago

And, the section on Manual installation

markcmiller86 commented 1 year ago

Or, you can try to use the new CMake approach available on the master branch....

Assuming you have CMake and HDF5 somewhere on your system...

  1. git clone https://github.com/LLNL/H5Z-ZFP.git
  2. cd H5Z-ZFP
  3. Follow CMake instructions
AlexLong152 commented 1 year ago

On my system, the develop branch now installs with spack without any errors.

spack install h5z-zfp@develop
markcmiller86 commented 1 year ago

I've updated docs. Preview available here