search

E4S-Project / e4s

E4S for Spack
https://e4s.readthedocs.io
MIT License
28 stars 13 forks source link

[support]: E4S/21.05 issue with buildcache #63

Closed shahzebsiddiqui closed 2 years ago

shahzebsiddiqui commented 2 years ago

Name of Software

E4S

URL for Software

https://e4s.io/

Contact Details

shahzebsiddiqui@lbl.gov

HPC System

Cori

Request Description

There is an issue with E4S/21.05 modulefile on Cori which was provided by E4S team see https://docs.nersc.gov/applications/e4s/cori/21.05/.

The issue is that we can't get the GPG key or see the buildcache content. I am not sure why this happened but all of a sudden it stopped working and it was reported by user ticket INC0184927

Here is the code in debug mode which is hitting a python bug. Note this working in past and i don't see anything change except the Operating System was upgraded

 ~/ spack mirror list
cori-e4s-21.05    https://cache.e4s.io/21.05
spack-public      https://spack-llnl-mirror.s3-us-west-2.amazonaws.com/
 ~/ spack -d buildcache keys -it
==> [2022-05-12-10:29:15.658062] Imported buildcache from built-in commands
==> [2022-05-12-10:29:15.660913] Imported buildcache from built-in commands
==> [2022-05-12-10:29:15.662584] Reading config file /global/common/software/spackecp/e4s-21.05/spack/etc/spack/defaults/mirrors.yaml
==> [2022-05-12-10:29:15.664839] Reading config file /global/common/software/spackecp/e4s-21.05/spack/etc/spack/mirrors.yaml
==> [2022-05-12-10:29:15.666567] Reading config file /global/common/software/spackecp/e4s-21.05/spack/var/spack/environments/e4s/spack.yaml
Traceback (most recent call last):
  File "/global/common/software/spackecp/e4s-21.05/spack/bin/spack", line 76, in <module>
    sys.exit(spack.main.main())
  File "/global/common/software/spackecp/e4s-21.05/spack/lib/spack/spack/main.py", line 774, in main
    return _invoke_command(command, parser, args, unknown)
  File "/global/common/software/spackecp/e4s-21.05/spack/lib/spack/spack/main.py", line 498, in _invoke_command
    return_val = command(parser, args)
  File "/global/common/software/spackecp/e4s-21.05/spack/lib/spack/spack/cmd/buildcache.py", line 798, in buildcache
    args.func(args)
  File "/global/common/software/spackecp/e4s-21.05/spack/lib/spack/spack/cmd/buildcache.py", line 531, in getkeys
    bindist.get_keys(args.install, args.trust, args.force)
  File "/global/common/software/spackecp/e4s-21.05/spack/lib/spack/spack/binary_distribution.py", line 1474, in get_keys
    _build_cache_keys_relative_path)
  File "/global/common/software/spackecp/e4s-21.05/spack/lib/spack/spack/util/url.py", line 157, in join
    paths[i], scheme=None, allow_fragments=False)
  File "/usr/lib64/python3.6/urllib/parse.py", line 371, in urlparse
    splitresult = urlsplit(url, scheme, allow_fragments)
  File "/usr/lib64/python3.6/urllib/parse.py", line 428, in urlsplit
    scheme = _remove_unsafe_bytes_from_url(scheme)
  File "/usr/lib64/python3.6/urllib/parse.py", line 417, in _remove_unsafe_bytes_from_url
    url = url.replace(b, "")
AttributeError: 'NoneType' object has no attribute 'replace'

the HEAD commit for spack is 

 /global/common/software/spackecp/e4s-21.05/spack/ [e4s-21.05*] git log --oneline -1
abed824555 (HEAD -> e4s-21.05, origin/e4s-21.05) hdf5: filter compiler wrapper: h5pcc, h5pfc (#24092)

Version of spack

 /global/common/software/spackecp/e4s-21.05/spack/ [e4s-21.05*] spack --version
0.16.1-2805-abed824555

When installing GPG key we should get it set in our home directory, this is already taken care of in modulefile

echo $SPACK_GNUPGHOME
/global/homes/s/siddiq90/.gnupg

Content of our modulefile

 /global/common/software/spackecp/e4s-21.05/spack/ [e4s-21.05*] cat /global/common/software/nersc/cle7up03/extra_modulefiles/e4s/21.05
#%Module1.0

set root /global/common/software/spackecp/e4s-21.05/spack
set         fullname    "E4S"
set             version         "21.05"
set         externalurl "https://e4s-project.github.io/"
set             description     "Extreme-scale Scientific Software Stack (E4S)"

set shell [module-info shell]

# Initialize spack's shell support
if { [ module-info mode load ] } {
    if { $shell == "bash" || $shell == "sh" || $shell == "zsh" } {
          puts stdout ". $root/share/spack/setup-env.sh ;"
    } elseif { $shell == "csh" || $shell == "tcsh" } {
          puts stdout "source $root/share/spack/setup-env.csh ;"
    }

    #activate spack e4s environment, this is where user will find the spack packages for e4s.
    puts stdout "spack env activate e4s ;"

} elseif { [module-info mode remove] } {

    unset-alias "spacktivate"
    unset-alias "despacktivate"
    unsetenv "SPACK_ROOT"
    unsetenv "SPACK_ENV"
    remove-path PATH $root/bin
}

proc ModulesHelp { } {
    global description externalurl
    puts stderr "Description - $description"
    puts stderr "External URL - $externalurl"
    puts stderr "The Extreme-scale Scientific Software Stack (E4S) is a collection of open source software packages for running scientific applications on high-performance computing (HPC) platforms."
    puts stderr ""
}

module-whatis "$description"

# Location where e4s 21.05 modules are located
prepend-path     MODULEPATH /global/common/software/spackecp/e4s-21.05/modules/cray-cnl7-haswell/
setenv          SPACK_GNUPGHOME $env(HOME)/.gnupg

## Log user info about module loads
if {[ module-info mode load ]} {
    set usgsbin /global/common/shared/usg/sbin
    if {[ expr [ file exists $usgsbin/logmod.e4s ] ]} {
        exec $usgsbin/logmod.e4s ${fullname} ${version}
    }
}

I already have the GPG installed since i trusted the key before we had this problem. The gpg key is the last one with ID 25645FA2B218FE55B4EF649E4345F04B40005581

 /global/common/software/spackecp/e4s-21.05/spack/ [e4s-21.05*] gpg -k
/global/homes/s/siddiq90/.gnupg/pubring.kbx
-------------------------------------------
pub   rsa2048 2020-09-22 [SCEA]
      EA172EB6343D30750A56522F0140A256659E0CBD
uid           [ultimate] Spack GPG Key (Spack E4S GPG Key) <shahzebsiddiqui@lbl.gov>
sub   rsa2048 2020-09-22 [SEA]

pub   rsa2048 2020-08-06 [SC] [expires: 2022-08-06]
      86D2AC5C44244034E20F9FF9F28D4556E19D435F
uid           [ unknown] Aditya Kavalur <akavalur@lbl.gov>
sub   rsa2048 2020-08-06 [E] [expires: 2022-08-06]

pub   rsa4096 2021-07-02 [SC]
      25645FA2B218FE55B4EF649E4345F04B40005581
uid           [ unknown] University of Oregon - E4S
sub   rsa4096 2021-07-02 [E]

We cant list content of buildcache which is hitting same error.

 /global/common/software/spackecp/e4s-21.05/spack/ [e4s-21.05*] spack -d buildcache list -Lva
==> [2022-05-12-10:34:34.423898] Imported buildcache from built-in commands
==> [2022-05-12-10:34:34.426750] Imported buildcache from built-in commands
==> [2022-05-12-10:34:34.430798] Reading config file /global/common/software/spackecp/e4s-21.05/spack/var/spack/environments/e4s/spack.yaml
==> [2022-05-12-10:34:34.552893] Reading config file /global/common/software/spackecp/e4s-21.05/spack/etc/spack/defaults/mirrors.yaml
==> [2022-05-12-10:34:34.555012] Reading config file /global/common/software/spackecp/e4s-21.05/spack/etc/spack/mirrors.yaml
Traceback (most recent call last):
  File "/global/common/software/spackecp/e4s-21.05/spack/bin/spack", line 76, in <module>
    sys.exit(spack.main.main())
  File "/global/common/software/spackecp/e4s-21.05/spack/lib/spack/spack/main.py", line 774, in main
    return _invoke_command(command, parser, args, unknown)
  File "/global/common/software/spackecp/e4s-21.05/spack/lib/spack/spack/main.py", line 498, in _invoke_command
    return_val = command(parser, args)
  File "/global/common/software/spackecp/e4s-21.05/spack/lib/spack/spack/cmd/buildcache.py", line 798, in buildcache
    args.func(args)
  File "/global/common/software/spackecp/e4s-21.05/spack/lib/spack/spack/cmd/buildcache.py", line 512, in listspecs
    specs = bindist.update_cache_and_get_specs()
  File "/global/common/software/spackecp/e4s-21.05/spack/lib/spack/spack/binary_distribution.py", line 1453, in update_cache_and_get_specs
    binary_index.update()
  File "/global/common/software/spackecp/e4s-21.05/spack/lib/spack/spack/binary_distribution.py", line 329, in update
    needs_regen = self._fetch_and_cache_index(mirror_url)
  File "/global/common/software/spackecp/e4s-21.05/spack/lib/spack/spack/binary_distribution.py", line 358, in _fetch_and_cache_index
    mirror_url, _build_cache_relative_path, 'index.json')
  File "/global/common/software/spackecp/e4s-21.05/spack/lib/spack/spack/util/url.py", line 157, in join
    paths[i], scheme=None, allow_fragments=False)
  File "/usr/lib64/python3.6/urllib/parse.py", line 371, in urlparse
    splitresult = urlsplit(url, scheme, allow_fragments)
  File "/usr/lib64/python3.6/urllib/parse.py", line 428, in urlsplit
    scheme = _remove_unsafe_bytes_from_url(scheme)
  File "/usr/lib64/python3.6/urllib/parse.py", line 417, in _remove_unsafe_bytes_from_url
    url = url.replace(b, "")
AttributeError: 'NoneType' object has no attribute 'replace'

Relevant log output

No response

Reproduce Bug

1. Load e4s/21.05 modulefile on Cori `module load e4s/21.05`
2. Run `spack buildcache keys -it`
3. Run `spack buildcache list`
...
eugeneswalker commented 2 years ago

Hi Shahzeb, I wanted to acknowledge that we are now aware of the issue and are able to reproduce the same error both on Cori and, separately, on an entirely different system. We are in the process of narrowing down the issue and will post an update here ASAP. Thank you for bringing this to our attention.

eugeneswalker commented 2 years ago

Hi Shahzeb, I tracked the problem down to a recent upgrade on Cori. As part of the upgrade Python's urllib module was upgraded. The new urllib implementation broke assumptions Spack made at the time E4S 21.05 was released in May 2021. I have cherry-picked a patch onto the E4S 21.05 deployment at Cori. Can you try again and verify that the problem is corrected?

shahzebsiddiqui commented 2 years ago

Hi Luke,

Yes i was able to confirm the changes for time being i was able to get some of this working

 ~/ module load e4s/21.05
 ~/ spack buildcache list -Lv | wc -l
501
 ~/ spack find --format "{name}" | wc -l
157

I was able to reinstall the GPG key so this looks good so far

 ~/ spack -d buildcache keys -it
==> [2022-05-13-08:10:47.940845] Imported buildcache from built-in commands
==> [2022-05-13-08:10:47.943579] Imported buildcache from built-in commands
==> [2022-05-13-08:10:47.945106] Reading config file /global/common/software/spackecp/e4s-21.05/spack/etc/spack/defaults/mirrors.yaml
==> [2022-05-13-08:10:47.947183] Reading config file /global/common/software/spackecp/e4s-21.05/spack/etc/spack/mirrors.yaml
==> [2022-05-13-08:10:47.948979] Reading config file /global/common/software/spackecp/e4s-21.05/spack/var/spack/environments/e4s/spack.yaml
==> [2022-05-13-08:10:48.056097] Finding public keys in https://cache.e4s.io/21.05
==> [2022-05-13-08:10:48.382165] Warning: Expected /tmp/user to have group 0, but it is 92503
==> [2022-05-13-08:10:48.382764] Creating stage lock build_cache
==> [2022-05-13-08:10:48.383553] Checking existence of https://cache.e4s.io/21.05/build_cache/_pgp/25645FA2B218FE55B4EF649E4345F04B40005581.pub
==> [2022-05-13-08:10:48.387041] '/usr/bin/curl' '--stderr' '-' '-s' '-f' '-r' '0-0' 'https://cache.e4s.io/21.05/build_cache/_pgp/25645FA2B218FE55B4EF649E4345F04B40005581.pub'
==> [2022-05-13-08:10:48.735441] Fetching https://cache.e4s.io/21.05/build_cache/_pgp/25645FA2B218FE55B4EF649E4345F04B40005581.pub
==> [2022-05-13-08:10:48.737010] '/usr/bin/curl' '-C' '-' '-o' '/tmp/user/spack-stage/siddiq90/build_cache/25645FA2B218FE55B4EF649E4345F04B40005581.pub.part' '-f' '-D' '-' '-L' 'https://cache.e4s.io/21.05/build_cache/_pgp/25645FA2B218FE55B4EF649E4345F04B40005581.pub' '-#' '--connect-timeout' '10'
###################################################################################################################################################################################### 100.0%
==> [2022-05-13-08:10:49.124310] Found key 25645FA2B218FE55B4EF649E4345F04B40005581
==> [2022-05-13-08:10:49.126575] '/usr/bin/gpgconf' '--version'
==> [2022-05-13-08:10:49.134829] '/usr/bin/gpgconf' '--dry-run' '--create-socketdir'
gpgconf: socketdir is '/run/user/92503/gnupg'
==> [2022-05-13-08:10:49.141661] '/usr/bin/gpg2' '--version'
==> [2022-05-13-08:10:49.165876] '/usr/bin/gpgconf' '--create-socketdir'
==> [2022-05-13-08:10:49.172147] '/usr/bin/gpg2' '--import' '/tmp/user/spack-stage/siddiq90/build_cache/25645FA2B218FE55B4EF649E4345F04B40005581.pub'
gpg: key 4345F04B40005581: "University of Oregon - E4S" not changed
gpg: Total number processed: 1
gpg:              unchanged: 1
==> [2022-05-13-08:10:49.181653] Added this key to trusted keys.
==> [2022-05-13-08:10:49.182398] Finding public keys in https://spack-llnl-mirror.s3-us-west-2.amazonaws.com/

Appreciate your help.