Closed cameronrutherford closed 5 months ago
reidpr
commented
6 months ago Thanks Cameron for the bug report and sorry you are having trouble.
Yes, config.log would be worth our looking at.
The one thing I noticed is that it looks like you are installing libfuse3 via Spack rather than externalizing it, which tingles my spidey sense.
If you haven't seen it, does spack/spack#42590 provide any insight?
j-ogas
commented
6 months ago @cameronrutherford FWIW, this is my go-to $HOME/.spack/packages.yaml file I use when building charliecloud. This will reduce the number of dependencies and make installing considerably faster. As @reidpr points out, you will want to use your system provided libfuse.
One thing to note: Spack has trouble finding external packages in /usr/lib64, don't ask me why, but to avoid that I simply make a directory and symlink in my home directory, e.g., /home/spack_kludge/lib -> /usr/lib64.
Another note: the following assumes a system with python 3.6.8+, git 2.28+, and python requests installed. Aside from libfuse, removing any of others in the example below is fine.
packages:
# Libfuse is installed in /usr/lib64, which spack has trouble with; make a symlink, `lib -> /usr/lib64`
# in `/home/spack_kludge` to avoid issues.
libfuse:
buildable: false
externals:
- spec: "libfuse@your-system-version"
prefix: /home/spack_kludge
git:
buildable: false
externals:
- spec: "git@your-system-version"
prefix: /usr
python:
buildable: false
externals:
- spec: "python@your-system-version"
prefix: /usr
py-requests:
buildable: false
externals:
- spec: "py-requests@your-system-version"
prefix: /usr
cameronrutherford
commented
6 months ago Really appreciate the help, but now my errors are being propagated up the stack to squashfuse... I took some liberties in editing logs and spack.yaml to remove unnecessary output, but hopefully this is progress.
config.log is attached as a file since it is so long...
EDIT: To clarify I ran mkdir -p ~/spack-kludge && ln -s /usr/lib64 ~/spack-kludge/lib to generate this error, and the same error happens with the external located in /usr/lib64
reidpr
commented
6 months ago I’m not a Spack expert, but it looks like SquashFUSE is trying to use pkg-config to find libfuse3 and failing (starts on line 1539 of the config.log in the previous comment). What do to about that, I don’t know.
It does seem that for the build in the first comment, SquashFUSE built OK, i.e. it found at least some libfuse (whether 2 or 3 I can’t tell; Charliecloud requires a SquashFUSE built against libfuse3).
I’d suggest next trying one or the other of:
Return to the Spack configuration of the first comment, try the build again, and attach both the complete Spack build log and Charliecloud’s config.log.
Build against SquashFUSE 0.4.0. (0.5.0 changed how the FUSE headers are found.)
j-ogas
commented
6 months ago libfuse: buildable: false externals: - spec: "libfuse@3.6.1" prefix: ~/spack-kludge
Just to make sure, in the actual packages.yaml file, do we have the ~ in prefix: ~/spack-kludge? If so, try replacing that with the absolute path.
cameronrutherford
commented
6 months ago I’m not a Spack expert, but it looks like SquashFUSE is trying to use
pkg-configto find libfuse3 and failing (starts on line 1539 of theconfig.login the previous comment). What do to about that, I don’t know.It does seem that for the build in the first comment, SquashFUSE built OK, i.e. it found at least some libfuse (whether 2 or 3 I can’t tell; Charliecloud requires a SquashFUSE built against libfuse3).
I’d suggest next trying one or the other of:
- Return to the Spack configuration of the first comment, try the build again, and attach both the complete Spack build log and Charliecloud’s
config.log.- Build against SquashFUSE 0.4.0. (0.5.0 changed how the FUSE headers are found.)
Attempting to use 0.5.0 with the full pathing and that results in identical errors:
reidpr
commented
6 months ago
- Thankfully I can just grab the old CI log that Spack keeps around, so here that is config.log
The problem here is that Charliecloud can’t find fuse_lowlevel.h. This is #1784 (Charliecloud can't build against SquashFUSE 0.5.0), which was fixed in 0.36.
Try this way again but with charliecloud@0.35+squashfuse^squashfuse@0.4.0 (spec may not be right, the point is use SquashFUSE 0.4.0)?
@j-ogas is working on spack/spack#42590 which I believe (among other things) encodes this dependency into Charliecloud's Spack package.
2. Downgrading and adding @j-ogas's suggestion resulted in a similar build error:
This one looks like SquashFUSE can’t find libfuse (which is what I think Jordan's suggestion was trying to address). Maybe a bug in SquashFUSE's Spack packaging?
cameronrutherford
commented
6 months ago Try this way again but with
charliecloud@0.35+squashfuse^squashfuse@0.4.0(spec may not be right, the point is use SquashFUSE 0.4.0)?
This was your original suggestion for debugging. The issue form my perspective is:
charliecloud@0.35.0+squashfuse fails with
squashfuse@0.4.0 - this fails to build with external libfuse (logs attached to 2)squashfuse@0.5.0libfuse, charliecloud fails to build (initial issue logs)libfuse, squashfuse fails to build (final set of logs in comment from 4 hours ago)Things I can try:
squashfuse@0.4.0 with spack built libfusecharliecloud@0.36.0 using custom package.py / from PR forkj-ogas
commented
6 months ago @cameronrutherford is pkg-config installed on your host machine? I hit this error on one of our production machines. I was able to move forward by adding pkg-config to the .spack/packages.yaml I shared above, e.g.:
[...]
pkg-config:
buildable: false
externals:
- spec: "pkg-config@FOO"
prefix: /usrIn my case, even with the https://github.com/spack/spack/pull/42590 branch, spack was trying to build squashfuse before pkg-config was installed. By making it an external package, it appears to behave as expected.
cameronrutherford
commented
6 months ago Using external pkg-config on my system as the pkgconf provider also didn't result in any luck for both squashfuse@0.4.0 and squashfuse@0.5.0 with identical looking errors based on spack install output. I can re-add logs if it would be worthwhile...
FWIW I have built charliecloud@0.30.0 with an older version of spack with external python@3.8 on this system before, and I didn't need to specify any version of these externals to really get anything to work. I can do some forensics to figure out what spack version + concretization worked previously if need be, but I was hoping this build would work.
Most appealing option to me at the moment is to just wait for the 0.36+ PR in Spack to be merged, as this isn't blocking anything for the moment....
reidpr
commented
6 months ago Sorry for this mess.
Another future thing that may help is #1844, which would bypass pkg-config, which smells like a key culprit for this bad interaction. I'm hoping this could go in 0.38, but Spack does have a master version of Charliecloud so that can be tried as soon as it's merged.
cameronrutherford
commented
6 months ago Thank you for the help! I still have my charliecloud sticker on my water bottle that I snagged from you at SC '23!
In trying to understand this better, but for now I think I will wait until there is a stable PR merged into spack@develop that I can use. Or are you suggesting that I just try spack@git.develop=0.35 in latest spack@develop?
I might try some older versions of charliecloud to see what work, but I also could try older libfuse2 installed on my system?
reidpr
commented
6 months ago Or are you suggesting that I just try
spack@git.develop=0.35in latestspack@develop?
I'm not sure what this means, so I'm not but maybe Jordan is.
older libfuse2 installed on my system
This definitely won't work; Charliecloud requires libfuse3. Do you also have libfuse3 installed on your system? If not, perhaps that's the culprit.
j-ogas
commented
5 months ago @cameronrutherford See two references above. It looks like the PR I approved to be merged into develop will address your issue, provided you provide:
packages:
libfuse:
buildable: false
externals:
- spec: libfuse@VERSION
prefix: /usrNote: use the real prefix /usr.
Not really sure what's going on here, as I have had success installing charliecloud in the past, but this bug seems be new and persistent/reproducible:
Spack concretization
``` ==> Concretized charliecloud+squashfuse - 7y3yp4q charliecloud@0.35%gcc@11.2.0~docs+squashfuse build_system=autotools arch=linux-centos7-zen2 [+] 2al52uo ^autoconf@2.72%gcc@11.2.0 build_system=autotools arch=linux-centos7-zen2 [e] 3enz5ve ^perl@5.26.0%gcc@11.2.0+cpanm+opcode+open+shared+threads build_system=generic patches=0eac10e,8cf4302 arch=linux-centos7-zen2 [+] iq3awo2 ^automake@1.16.5%gcc@11.2.0 build_system=autotools arch=linux-centos7-zen2 [+] 3wkhnll ^gcc-runtime@11.2.0%gcc@11.2.0 build_system=generic arch=linux-centos7-zen2 [e] hymxaj5 ^git@2.37.3.dirty%gcc@11.2.0+man+nls+perl+subtree~svn+tcltk build_system=autotools arch=linux-centos7-zen2 [+] ynvrnsk ^gmake@4.4.1%gcc@11.2.0~guile build_system=generic arch=linux-centos7-zen2 [+] y2ubeek ^libtool@2.4.7%gcc@11.2.0 build_system=autotools arch=linux-centos7-zen2 [+] jsju533 ^findutils@4.9.0%gcc@11.2.0 build_system=autotools patches=440b954 arch=linux-centos7-zen2 [+] tyus3ia ^m4@1.4.19%gcc@11.2.0+sigsegv build_system=autotools patches=9dc5fbd,bfdffa7 arch=linux-centos7-zen2 [+] 3ei4ld3 ^diffutils@3.9%gcc@11.2.0 build_system=autotools arch=linux-centos7-zen2 [+] tp2gtwt ^libiconv@1.17%gcc@11.2.0 build_system=autotools libs=shared,static arch=linux-centos7-zen2 [+] s6uah56 ^libsigsegv@2.14%gcc@11.2.0 build_system=autotools arch=linux-centos7-zen2 [+] iwxqwde ^py-requests@2.31.0%gcc@11.2.0~socks build_system=python_pip arch=linux-centos7-zen2 [+] dfkvarp ^py-certifi@2023.7.22%gcc@11.2.0 build_system=python_pip arch=linux-centos7-zen2 [+] i2gbmfp ^py-charset-normalizer@3.3.0%gcc@11.2.0 build_system=python_pip arch=linux-centos7-zen2 [+] yady4lx ^py-idna@3.4%gcc@11.2.0 build_system=python_pip arch=linux-centos7-zen2 [+] wgaoh33 ^py-flit-core@3.9.0%gcc@11.2.0 build_system=python_pip arch=linux-centos7-zen2 [+] a7e3nwj ^py-pip@23.1.2%gcc@11.2.0 build_system=generic arch=linux-centos7-zen2 [+] y66xhz6 ^py-setuptools@68.0.0%gcc@11.2.0 build_system=generic arch=linux-centos7-zen2 [+] m2xzxui ^py-urllib3@2.1.0%gcc@11.2.0~brotli~socks build_system=python_pip arch=linux-centos7-zen2 [+] 4gbpcmi ^py-hatchling@1.21.0%gcc@11.2.0 build_system=python_pip arch=linux-centos7-zen2 [+] f4x5obh ^py-editables@0.3%gcc@11.2.0 build_system=python_pip arch=linux-centos7-zen2 [+] aqrk464 ^py-packaging@23.1%gcc@11.2.0 build_system=python_pip arch=linux-centos7-zen2 [+] bd6hdly ^py-pathspec@0.11.1%gcc@11.2.0 build_system=python_pip arch=linux-centos7-zen2 [+] c7stp6m ^py-pluggy@1.4.0%gcc@11.2.0 build_system=python_pip arch=linux-centos7-zen2 [+] t726pwf ^py-setuptools-scm@7.1.0%gcc@11.2.0+toml build_system=python_pip arch=linux-centos7-zen2 [+] uv6zltr ^py-tomli@2.0.1%gcc@11.2.0 build_system=python_pip arch=linux-centos7-zen2 [+] xxpix3n ^py-typing-extensions@4.8.0%gcc@11.2.0 build_system=python_pip arch=linux-centos7-zen2 [+] ebda7js ^py-trove-classifiers@2023.8.7%gcc@11.2.0 build_system=python_pip arch=linux-centos7-zen2 [+] 5knu5dh ^py-calver@2022.6.26%gcc@11.2.0 build_system=python_pip arch=linux-centos7-zen2 [+] qx5qvui ^py-wheel@0.41.2%gcc@11.2.0 build_system=generic arch=linux-centos7-zen2 [+] tto3fkv ^python@3.11.7%gcc@11.2.0+bz2+crypt+ctypes+dbm~debug+libxml2+lzma~nis~optimizations+pic+pyexpat+pythoncmd+readline+shared+sqlite3+ssl~tkinter+uuid+zlib build_system=generic patches=13fa8bf,b0615b2,ebdca64,f2fd060 arch=linux-centos7-zen2 [+] gyga46v ^bzip2@1.0.8%gcc@11.2.0~debug~pic+shared build_system=generic arch=linux-centos7-zen2 [+] d757m2y ^expat@2.5.0%gcc@11.2.0+libbsd build_system=autotools arch=linux-centos7-zen2 [+] l5p4guk ^libbsd@0.11.7%gcc@11.2.0 build_system=autotools arch=linux-centos7-zen2 [+] jxdfgoy ^libmd@1.0.4%gcc@11.2.0 build_system=autotools arch=linux-centos7-zen2 [+] tnyroyl ^gdbm@1.23%gcc@11.2.0 build_system=autotools arch=linux-centos7-zen2 [+] ikohprx ^gettext@0.22.4%gcc@11.2.0+bzip2+curses+git~libunistring+libxml2+pic+shared+tar+xz build_system=autotools arch=linux-centos7-zen2 [+] ucsdksq ^libxml2@2.10.3%gcc@11.2.0+pic~python+shared build_system=autotools arch=linux-centos7-zen2 [+] mtkc5m2 ^tar@1.34%gcc@11.2.0 build_system=autotools zip=pigz arch=linux-centos7-zen2 [+] jb6xetj ^pigz@2.8%gcc@11.2.0 build_system=makefile arch=linux-centos7-zen2 [+] epmgpg5 ^libffi@3.4.4%gcc@11.2.0 build_system=autotools patches=070b1f3 arch=linux-centos7-zen2 [+] 53shqex ^libxcrypt@4.4.35%gcc@11.2.0~obsolete_api build_system=autotools patches=4885da3 arch=linux-centos7-zen2 [+] tw5jsud ^ncurses@6.4%gcc@11.2.0~symlinks+termlib abi=none build_system=autotools arch=linux-centos7-zen2 [+] 6vcq2dc ^openssl@3.2.1%gcc@11.2.0~docs+shared build_system=generic certs=mozilla arch=linux-centos7-zen2 [+] xuylapa ^ca-certificates-mozilla@2023-05-30%gcc@11.2.0 build_system=generic arch=linux-centos7-zen2 [+] smraxwd ^pkgconf@1.9.5%gcc@11.2.0 build_system=autotools arch=linux-centos7-zen2 [+] vuyom3r ^readline@8.2%gcc@11.2.0 build_system=autotools patches=bbf97f1 arch=linux-centos7-zen2 [+] sfklrrr ^sqlite@3.43.2%gcc@11.2.0+column_metadata+dynamic_extensions+fts~functions+rtree build_system=autotools arch=linux-centos7-zen2 [+] bld2qix ^util-linux-uuid@2.38.1%gcc@11.2.0 build_system=autotools arch=linux-centos7-zen2 [+] la3lqgr ^xz@5.4.6%gcc@11.2.0~pic build_system=autotools libs=shared,static arch=linux-centos7-zen2 [+] u54ev23 ^zlib@1.3%gcc@11.2.0+optimize+pic+shared build_system=makefile arch=linux-centos7-zen2 [+] hwnny62 ^squashfs@4.6.1%gcc@11.2.0+gzip~lz4~lzo~static~xz~zstd build_system=makefile default_compression=gzip arch=linux-centos7-zen2 - lnqtrni ^squashfuse@0.5.0%gcc@11.2.0+lz4+lzo~min_size+shared+static+xz+zlib+zstd build_system=autotools arch=linux-centos7-zen2 - yyg7z36 ^libfuse@3.16.2%gcc@11.2.0~strip~system_install~useroot+utils build_system=meson buildtype=release default_library=shared patches=3ad6719 arch=linux-centos7-zen2 [+] gouelpf ^meson@1.2.2%gcc@11.2.0 build_system=python_pip patches=0f0b1bd,ae59765 arch=linux-centos7-zen2 [+] ybbbaw2 ^ninja@1.11.1%gcc@11.2.0+re2c build_system=generic arch=linux-centos7-zen2 [+] 3qhumfv ^re2c@2.2%gcc@11.2.0 build_system=generic arch=linux-centos7-zen2 [+] je6fgie ^lz4@1.9.4%gcc@11.2.0+pic build_system=makefile libs=shared,static arch=linux-centos7-zen2 [+] 2fsnzdy ^lzo@2.10%gcc@11.2.0 build_system=autotools libs=shared,static arch=linux-centos7-zen2 [+] svz4d5l ^zstd@1.5.5%gcc@11.2.0+programs build_system=makefile compression=none libs=shared,static arch=linux-centos7-zen2 ```Spack build log
``` ==> Installing charliecloud-0.35-7y3yp4qv3rj4hsqvjukt6osqf7bc3pzr [107/107] ==> No binary for charliecloud-0.35-7y3yp4qv3rj4hsqvjukt6osqf7bc3pzr found: installing from source ==> Fetching https://mirror.spack.io/_source-cache/archive/04/042f5be5ed8eda95f45230b4647510780142a50adb4e748be57e8dd8926b310e.tar.gz ==> No patches needed for charliecloud ==> charliecloud: Executing phase: 'autoreconf' ==> charliecloud: Executing phase: 'configure' ==> Error: ProcessError: Command exited with status 1: '/.../spack-stage/spack-stage-charliecloud-0.35-7y3yp4qv3rj4hsqvjukt6osqf7bc3pzr/spack-src/configure' '--prefix=/.../spack/opt/spack/linux-centos7-zen2/gcc-11.2.0/charliecloud-0.35-7y3yp4qv3rj4hsqvjukt6osqf7bc3pzr' '--with-python=/.../spack/opt/spack/linux-centos7-zen2/gcc-11.2.0/python-3.11.7-tto3fkv2ijstzai4lf33v26v6hmg4gtm/bin/python3.11' '--disable-html' '--with-libsquashfuse=/.../spack/opt/spack/linux-centos7-zen2/gcc-11.2.0/squashfuse-0.5.0-lnqtrnin67p6z33nmtlj5nb5twrtqgoc' 6 errors found in build log: 12 13 + [[ -f Makefile ]] 14 + rm -Rf Makefile.in ./bin/Makefile.in ./doc/Makefile.in ./examples/ Makefile.in ./lib/Makefile.in ./misc/Makefile.in ./packaging/Makefil e.in ./test/Makefile.in aclocal.m4 bin/config.h.in build-aux configu re 15 + [[ -n '' ]] 16 + [[ -z '' ]] 17 + autoreconf --force --install -Wall -Werror >> 18 configure.ac:234: installing 'build-aux/compile' >> 19 configure.ac:21: installing 'build-aux/config.guess' >> 20 configure.ac:21: installing 'build-aux/config.sub' >> 21 configure.ac:64: installing 'build-aux/install-sh' >> 22 configure.ac:64: installing 'build-aux/missing' 23 bin/Makefile.am: installing 'build-aux/depcomp' 24 + [[ ! -e lib/lark ]] 25 + [[ -e lib/lark ]] 26 + [[ ! -e lib/lark-0.11.3.dist-info/INSTALLER ]] 27 + set +x 28 ... 77 checking if mksquashfs version >= 4.2... ok (4.2-git-stable) 78 checking for squashfuse... squashfuse 79 checking if squashfuse version >= 0.1.100... ok (0.5.0) 80 checking for fuse_set_signal_handlers in -lfuse3... yes 81 checking for sqfs_ll_mount in -lsquashfuse_ll... yes 82 checking for squashfuse/ll.h... no >> 83 configure: error: libsquashfuse requested but not found ```I have verified that
ll.hexists in the squashfuse install tree underinclude/squashfuse/ll.h, so not sure how this is erroring.config.logseemed unhelpful, but I can attach as needed.