Open ocaisa opened 3 years ago
We will also probably need to place $EPREFIX/bin
into PATH
That's a brilliant idea!
At JSC we use
BASH_ENV=/path/to/lmod/lmod/init/bash
so that the module command works in non-interactive shells. If this variable is set in the users session it can break software installation on top of EESSSI with errors that show we are using Lmod from the host rather than EESSI
BASH_ENV
is an interesting environment variable to leverage so we can automate that jobs submitted to select the right microarchitecture for the node on which the code will run on
@bedroge After having a meeting about using the EESSI stack as a baseline installation, I thought a lot about what this means and how you would implement it. In the list of environment variables that are required to build the EESSI stack there are 3 key ones:
export EASYBUILD_SYSROOT=${EPREFIX}
export EASYBUILD_RPATH=1
export EASYBUILD_FILTER_DEPS=$DEPS_TO_FILTER (calculated)
To build on top of EESSI in a consistent way, you need to use the 3 of these. The main problem is that $EPREFIX
is (at least currently) a moving target. The way I can see of pinning it down is to stick to one $EPREFIX
per toolchain generation, then one can generate a EB hook to automatically chose the right one for you depending on what you are building (if you build something that uses the foss/2020a
hierarchy you use the $EPREFIX
corresponding to the 2021.03
pilot).
I also thought about being able to retire pilots in EESSI. One way to do this is to make a tarball of what you will remove (e.g., /cvmfs/pilot.eessi-hpc.org/2021.03/
), and make that accessible somewhere, you then clean out the directory contents and turn the directory into a variable symlink. If someone want to continue using a retired pilot, they can unpack the tarball locally and set the symlink to point to it.
I'm reworking the install script now to make rebuilding your own module tree easy.
In https://github.com/EESSI/software-layer/pull/100 there are the code changes that allow you to rebuild the EESSI module tree according to your preferred naming scheme (has been tested with flat and hierarchical schemes).
Nice! I also like that idea with the variant symlink for people that want to use an old/removed version. We could dump these in for instance some AWS S3 bucket (and maybe even move them to S3 Glacier at some point).
Speaking of life-cycle handling of S3 buckets, my testing environment for tarball storage for ingestion to S0 looks like this right now:
resource "aws_s3_bucket" "bucket" {
provider = eessiaws
bucket = "eessi-staging"
policy = data.aws_iam_policy_document.public_read_s3.json
acl = "private"
tags = {
Name = "EESSI staging bucket",
}
lifecycle_rule {
id = "layers"
enabled = true
# prefix = "/"
tags = {
rule = "layers"
autoclean = "true"
}
transition {
days = 30
storage_class = "STANDARD_IA" # or "ONEZONE_IA"
}
transition {
days = 60
storage_class = "GLACIER"
}
expiration {
days = 90
}
}
}
S3 is fun. Except for access policies.
I tested building on top of the EESSI setup this week and I think there is scope to add an easyblock to EasyBuild that will set up a module for you to allow you to easily do this. If you look at the current build script we use, there are only a few environment variables that are required:
I also think there is scope for another module for people who want to build their own software (not with EB). We can update the buildenv easyblock to (conditionally) include the rpath wrappers.