Support for using environment variables in EasyBuild's configuration file

[dominikborkowski]

It would be very useful to have the ability to leverage existing environment variables in easybuild's config file, especially in shared environments, perhaps via something like %(env:FOO)s.

For example: buildpath=/dev/shm/%(env:USER)s/build or: repositorypath=/easybuild/easybuild_repo/%(env:MACHTYPE) subdir-software=/easybuild/software/%(env:HOSTTYPE) or: prefix=%{env:HOME}/.local/easybuild/%{env:HOSTNAME}

Some of this can be accomplished now by use of EASYBUILD environment variables, but one quickly ends up having to maintain two sets of files: one setting EASYBUILD variables, and the config file. At the same time, not everything can be accomplished just with the environment variables, such as prepending and appending to the robot paths.

What do you think? Cheers!

[boegel]

I like the idea, we could use it on our systems too, since we rely on environment variables for some configuration settings too...

@stdweird: thoughts?

[stdweird]

EB can export whatever templates to the configfiles as it sees it useful (eg USER etc etc).

but something free as what you ask is some disaster waiting to happen (you make it really really easy to screw up). it will be very hard to garantee anything, and it also renders the concept of configfiles useless. if you want to set everything via environment, well, we'll fix the robot path issue so you can do that (or any other discrepancy between config, environment and commandline options for that matter).

but i'm afraid this simply hints to some underlying granularity that is not reachable via usual configoptions. the current proposal looks like a hack to solve something deeper. it's be good to understand what you want to achieve with the added flexibility, and if there is a better, safer way to achieve this then to rely on environment variables (either via environment directly using EASYBUILD_ or some sideway via these templated configfiles).

e.g., similar to the custom naming schemes, we could allow some custom module to overload the default options.py or safer only the defaults. the main advantage is that you can then do basic checks in the module (eg does the variable exist? does the path already exist) or any more advanced logic (i think this is wat we do know and what @boegel hints at).

[dominikborkowski]

The ultimate goal behind this request is to easily maintain multiple easybuild instances via centralized configurations or environment modules, while allowing to re-use code, and support multiple users. My original approach was to use environment modules and EASYBUILD_ variables; however, after finding out that I couldn't prepend paths to EASYBUILD_ROBOTPATHS, I tried the config file approach. It's easy to set up a custom, per user, build path via modules, there is no equivalent via the configs. Hence this feature request.

I'm not sure I agree with the statement that extending functionality of configuration files makes them somewhat useless. Quite the contrary, they become more useful. For example:

• regular users can maintain a single easybuild configuration file, and the software they build ends up in a location specific to architecture or/and machine they're on
• administrators can maintain a single easybuild configuration file, and do the same

By using a single method for configuring easybuild, be it configuration files or modules, the task of maintaining said configuration is much easier. The lack of guarantee is the same, whether the configuration options leverage static values or ones pulled from another source: they still result in the same failures or successes.

Another point to consider is that easybuild has a very useful function built in: --confighelp. This generates a vanilla configuration file, with nice sections and comments. Working with said configuration is certainly easier than trying to manually populate a module file with appropriate EASYBUILD_ environment variables. They are not as readily available nor documented as the config file.

Ultimately, the goal is to leverage easybuild in a shared environment, where a group of administrators will be maintaining software for their users, using shared storage spanning multiple heterogenous clusters. Documentation on these setups is sparse, and the efforts done on my end hope to improve that. I can share our desired file/dir structure for this, to illustrate exactly what we're trying to achieve.

Thanks!

[boegel]

@dominikborkowski: Please don't hesitate to share more details on your particular setup. Maybe do that via a gist your link to in here, so you can reuse the effort of documenting it in other contexts (e.g. discussing it on IRC).

I had a chat with @stdweird about your feature request (we're in the same office, which helps (sometimes ;-)).

Although I don't fully agree with his view, I do understand his concern that making it easy to use (any) environment variables in configuration files basically makes things too flexible. It becomes really easy to shoot yourself in the foot unexpectedly, and suddenly end up with a partial software stack in a different location due to a typo or an environment variable being set incorrectly for whatever reason (e.g. a shady install procedure that involves sourcing a buggy script).

Although this is already (kind of) the case with the $EASYBUILD_-prefixed environment variables and command line options (where you can already use any random environment variable to configure eb), it doesn't mean we should make things 'worse'.

I briefly discussed @stdweird's alternative suggestion of adding support for a 'configuration plugin' (a Python module), to (re)define the default for the various configuration options. This would be a configuration level 'below' configuration files (i.e. values in a config file would override the default set via a config plugin). Not only will this solve your problem (being able to configure eb in a single location/file, with access to environment variables), it has several other advantages:

• allowing to check whether your configuration is 'sane', i.e. whether the specified paths exist, whether module files are already there in that location, whether the settings match with the active $MODULEPATH, etc.
• more power to define the configuration, e.g., checking system characteristics (OS, processor architecture, hostname, ...) programmatically, rather than only using environment variables that get defined in whatever way, etc.

The default values for the configuration options would then also be defined via such a separate Python module, which can be used as a starting-point for your own 'config plugin' (i.e., you can derive from the one EB uses by default, and only tweak certain settings).

Thoughts on this? It's probably not terribly hard to add support for this (but not trivial either).

Next to that, we probably need a command line option like --show-current-config that gives you an overview of the currently active configuration, possibly together with how each setting was determined ((plugin,) cfgfile, env var, cmdline option). It should print the value for the 'important' configuration options (software/module install prefix, etc.), plus any others that differ from the default, to provide a clear and concise yet complete view.

[dominikborkowski]

Apologies about the delay, I finally had a chance to sit down and put together a few details of what I'd like to accomplish. Per suggestion, saving it as a gist: https://gist.github.com/dominikborkowski/1e53bd64af9d937d2c3f

[fgeorgatos]

Hi,

I would like to highlight that the last item mentioned in the gist is a very often recurring need, and it should indeed have an easier solution. We all find workarounds but that doesn't make it OK :)

thanks for raising the points, F.

On 25 March 2015 at 15:26, Dominik L. Borkowski notifications@github.com wrote:

Apologies about the delay, I finally had a chance to sit down and put together a few details of what I'd like to accomplish. Per suggestion, saving it as a gist: https://gist.github.com/dominikborkowski/1e53bd64af9d937d2c3f

— Reply to this email directly or view it on GitHub https://github.com/hpcugent/easybuild/issues/106#issuecomment-86081782.

echo "sysadmin know better bash than english"|sed s/min/mins/ \ | sed 's/better bash/bash better/' # signal detected in a CERN forum

[stdweird]

the last point is basically expresses the need for a --user option for eb, to append the username to some of the paths. @dominikborkowski, can you open a PR for this (shouldn't be that hard)

the 6th point has PRs underreview

[dominikborkowski]

Revised the VBI plan document https://gist.github.com/dominikborkowski/1e53bd64af9d937d2c3f . Added a little bit more of introduction, information about EasyBuild's git integration and links to the test modules and config files we've been trying out to get this environment working. I'll send this to our cluster manager's vendor along with few more links to get them more interested in providing official support for EasyBuild in their product.

I'm also hoping that other folks will find it useful.

Cheers!

[shahzebsiddiqui]

I ran into similar issue when I wanted to set my prefix to $WORK at stampede. I have the following configuration

login4(359)$ cat ~/.config/easybuild/config.cfg
[config]
prefix: $WORK/easybuild

When i ran this i got a $WORK directory instead of doing a directory translation

login4(368)$ ls -l
total 12
drwx------ 11 sms1990 G-817668 4096 Dec 16 10:34 buildtest-framework
drwx------  3 sms1990 G-817668 4096 Dec 17 10:52 easybuild
drwx------  3 sms1990 G-817668 4096 Dec 17 10:52 $WORK

I found reference to three places where DEFAULT_PREFIX is used.

https://github.com/easybuilders/easybuild-framework/blob/307abbee754b2f4f5a883cd129d165d85ffb499c/test/framework/options.py#L3459 https://github.com/easybuilders/easybuild-framework/blob/307abbee754b2f4f5a883cd129d165d85ffb499c/easybuild/tools/options.py#L497 https://github.com/easybuilders/easybuild-framework/blob/develop/easybuild/tools/config.py#L101

I think the issue is os.path.expandvars() needs to be run on all directories to ensure directory translation is set when setting variable. https://docs.python.org/3/library/os.path.html#os.path.expandvars

[Falkor]

@boegel in line with this issue, I wanted to use reference to the current directory of the easybuild config file $EASYBUILD_CONFIGFILES (something like os.path.dirname(os.path.realpath(__file__))) as a template variable that can be used to better resolve the path relative to other directories (typically the hooks/ directory for instance as %(configfile_dir)s/../hooks), the objective being to avoid hardcoding absolute paths in the [config/]easyconfig.cfg. Complementary/alternatively, can we use Easybuild variables as template inside the config file ? (but then I fear a chicken and egg problem)