reframe-hpc / reframe

A powerful Python framework for writing and running portable regression tests and benchmarks for HPC systems.
https://reframe-hpc.readthedocs.org
BSD 3-Clause "New" or "Revised" License
214 stars 101 forks source link

Repetition of code when multiple modules are using the same path. #2604

Open ekouts opened 1 year ago

ekouts commented 1 year ago

I am opening this after a conversation with @jgphpc : When we define module with the full path there can be unnecessary duplication in the code: config.py

...
            'modules': [
                {
                    'name': 'cray-mpich-binary/8.1.18.4-gcc-twtq6ct',
                    'path': '/scratch/e1000/gcc/11.3.0',
                },
                {
                    'name': 'cuda/11.7.1-yg5gvzu.lua',
                    'path': '/scratch/e1000/gcc/11.3.0',
                },
             ]
...

ReFrame will generate this in the script:

module use /scratch/e1000/gcc/11.3.0
module load cray-mpich-binary/...
module use /scratch/e1000/gcc/11.3.0
module load cuda/...

If we remove the path from the second one ReFrame will complain

...
            'modules': [
                {
                    'name': 'cray-mpich-binary/8.1.18.4-gcc-twtq6ct',
                    'path': '/scratch/e1000/gcc/11.3.0',
                },
                {
                    'name': 'cuda/11.7.1-yg5gvzu.lua',
                    # 'path': '/scratch/e1000/gcc/11.3.0',
                },
             ]
...

with this

Lmod Warning: Failed to find the following module(s): "cuda/11.7.1-yg5gvzu" in your MODULEPATH

Maybe we could do something better in reframe so that it doesn't try to add the module path unnecessarily.

jgphpc commented 1 year ago

Likewise, it would be nice to be able to mix definitions:

...
 'modules': [
                {
                    'name': 'cray-mpich-binary/8.1.18.4-gcc-twtq6ct',
                    'path': '/scratch/e1000/gcc/11.3.0',
                },
                'cuda/11.7.1-yg5gvzu.lua',
             ]
...

Reframe will refuse:

  * Reason: environ error: could not execute module operation: 
  * command '/usr/share/lmod/lmod/libexec/lmod python show cuda/11.7.1-yg5gvzu' 
    failed with exit code 1:
...
Lmod Warning: Failed to find the following module(s): "cuda/11.7.1-yg5gvzu" in
your MODULEPATH
Try:
    $ module spider cuda/11.7.1-yg5gvzu

or stg like:

               {
                   'path': '/scratch/e1000/gcc/11.3.0', 
                   'name': ['cray-mpich-binary/8.1.18.4-gcc-twtq6ct',    
                                   'cuda/11.7.1-yg5gvzu']
                },
jgphpc commented 1 year ago

@ekouts suggests:

'path': os.getenv('MYPATH'),

which is nice.