Closed Flamefire closed 4 years ago
Two issues need to be addressed
Find the full search path for easyconfigs. This will require the EasyBuild module to be loaded for easy_update to work.
Need to support minimal toolchains for Python. Dependency names are constructed from the current toolchain. ('Python', '3.7.4')
becomes Python-3.7.4-foss-2019b.eb which is not found. For case of Python the base luange easyconfig should be
Python-3.7.4-GCCcore-8.3.0.eb```
thanks for using easy_update lots of changes made to support this. Tested with Tensorflow and fhPython. Please download 2.1.0 release.
Thanks for the update! :+1:
As you mentioned in #30 "easy_update now requires that EasyBuild be loaded.": May I suggest a potentially easier and more reliable approach? See this code:
from easybuild.framework.easyconfig.tools import get_paths_for
get_paths_for() # -> '/home/s3248973/.local/EasyBuildDev/easybuild-easyconfigs/easybuild/easyconfigs'
from easybuild.framework.easyconfig.tools import parse_easyconfigs
ecs=parse_easyconfigs([('/home/s3248973/git/easybuild-easyconfigs/easybuild/easyconfigs/t/TensorFlow/TensorFlow-2.4.0-fosscuda-2019b-Python-3.7.4.eb', False)])
[dep for dep in ecs[0][0]['dependencies'] if dep['name']=='Python'] # -> [{'external_module_metadata': {}, 'short_mod_name': 'Python/3.7.4-GCCcore-8.3.0', 'toolchain': {'version': '8.3.0', 'name': 'GCCcore'}, 'name': 'Python', 'full_mod_name': 'Python/3.7.4-GCCcore-8.3.0', 'system': False, 'build_only': False, 'versionsuffix': '', 'version': '3.7.4', 'toolchain_inherited': False, 'hidden': False, 'external_module': False}]
Note how this already resolves the toolchain for Python. Maybe this is helpful for future changes.
Related to #30
For TensorFlow-2.4.0-fosscuda-2019b-Python-3.7.4.eb it searches for a Python-3.7.4-fosscuda-2019b.eb but only a GCCcore version exists.
Maybe instead of rolling your own parser and such the easybuild packages should be used? IIRC with those you can easily find ECs using the same algorithm as EB itself avoiding inconsistencies