fizwit / easy_update

Update EasyBuild package configuration files for R and Python bundles
19 stars 9 forks source link

Subtoolchains not considered when searching for ECs #31

Closed Flamefire closed 4 years ago

Flamefire commented 4 years ago

Related to #30

For TensorFlow-2.4.0-fosscuda-2019b-Python-3.7.4.eb it searches for a Python-3.7.4-fosscuda-2019b.eb but only a GCCcore version exists.

Maybe instead of rolling your own parser and such the easybuild packages should be used? IIRC with those you can easily find ECs using the same algorithm as EB itself avoiding inconsistencies

fizwit commented 4 years ago

Two issues need to be addressed

Find the full search path for easyconfigs. This will require the EasyBuild module to be loaded for easy_update to work.

Need to support minimal toolchains for Python. Dependency names are constructed from the current toolchain. ('Python', '3.7.4') becomes Python-3.7.4-foss-2019b.eb which is not found. For case of Python the base luange easyconfig should bePython-3.7.4-GCCcore-8.3.0.eb```

fizwit commented 4 years ago

thanks for using easy_update lots of changes made to support this. Tested with Tensorflow and fhPython. Please download 2.1.0 release.

Flamefire commented 4 years ago

Thanks for the update! :+1:

As you mentioned in #30 "easy_update now requires that EasyBuild be loaded.": May I suggest a potentially easier and more reliable approach? See this code:

from easybuild.framework.easyconfig.tools import get_paths_for
get_paths_for() # -> '/home/s3248973/.local/EasyBuildDev/easybuild-easyconfigs/easybuild/easyconfigs'
from easybuild.framework.easyconfig.tools import parse_easyconfigs
ecs=parse_easyconfigs([('/home/s3248973/git/easybuild-easyconfigs/easybuild/easyconfigs/t/TensorFlow/TensorFlow-2.4.0-fosscuda-2019b-Python-3.7.4.eb', False)])
[dep for dep in ecs[0][0]['dependencies'] if dep['name']=='Python'] # -> [{'external_module_metadata': {}, 'short_mod_name': 'Python/3.7.4-GCCcore-8.3.0', 'toolchain': {'version': '8.3.0', 'name': 'GCCcore'}, 'name': 'Python', 'full_mod_name': 'Python/3.7.4-GCCcore-8.3.0', 'system': False, 'build_only': False, 'versionsuffix': '', 'version': '3.7.4', 'toolchain_inherited': False, 'hidden': False, 'external_module': False}]

Note how this already resolves the toolchain for Python. Maybe this is helpful for future changes.