Easybuild as a library

[espenfl]

On the lookout for strategies to iterate over many combinations of some base software perturbations using Easybuild, an external wrapper seems the way to go. In that context, is there a way to call Easybuild directly from Python? Say that we generate the easyconfig parameters as a dictionary and then execute one Easybuild step with a call to something similar to what is done in main.py etc., basically running eb without using run_cmd etc.

Thanks in advance.

[boegel]

You can use EasyBuild directly from Python in theory, but I'm not aware of people doing this, and I'm not sure how well this would work in practice (I haven't tried this myself).

One important thing you'll definitely need to do is set up the configuration before doing anything:

from easybuild.tools.options import set_up_configuration
set_up_configuration(silent=True)

(see also https://github.com/easybuilders/easybuild-framework/pull/2638)

From that point forward, you can import stuff from the easybuild namespace and play around.

Generating easyconfigs on the fly as dicitionaries and feeding them in is an interesting idea, I haven't tried that myself, and I'm not sure how easy that would be currently. Maybe just give it a shot, and see what you end up with?

One way could be to:

1. generate easyconfig files (on disk)
2. specify the list of locations via args to the main function in easybuild/main.py

(then you don't even need to run set_up_configuration first, since main will take care of that for you)

Or maybe what you're looking for is hooks, see https://easybuild.readthedocs.io/en/latest/Hooks.html.

The start hook is called early enough to inject easyconfigs from there, but the way it is implemented now it doesn't give you the option to fiddle with the options or arguments that get processed, so some minor changes would have to be made to the start hook if you'd like to follow that approach...

[espenfl]

Thanks for the very quick reply. Greatly appreciated. Okey, I will try this. This question also goes in hand with #2865.

[espenfl]

Okey, nearly there following 1. and 2. as an initial approach. I can now pass a list to main containing the generated easyconfig files. However, how do I pass along the rest of the arguments? Seems args only contains the easyconfig files and not the other arguments. I typically also need include-easyblocks, detect-loaded-modules, info, and robot.

[boegel]

The args you can pass to the main function can both include easyconfig file names and options, so this should work as expected:

main(args['/tmp/example.eb', '--include-easyblocks', '/tmp/*.py'])

The args list will be passed to the option parser, which will return a list of parsed options and remaining arguments (which typically are easyconfig file names).