Error in installation of `Python-3.11.5-GCCcore-13.2.0.eb` with EasyBuild 4.9.2

[gkaf89]

When trying to install Python-3.11.5-GCCcore-13.2.0.eb with EasyBuild version 4.9.2, the installation fails due to an error in the installation of a file. The installation script seems to be using the log file path to store intermediate files which causes an error.

• EasyBuild command:

  eb --configfiles=iris_broadwell.cfg Python-3.11.5-GCCcore-13.2.0.eb

• Contents of configuration file iris_broadwell.cfg:

  [MAIN]
  # Enable debug log mode (default: False)
  #debug=
  # Enable info log mode (default: False)
  #info=
  # Enable quiet/warning log mode (default: False)
  #quiet=
  
  [basic]
  # Submit the build as a job (default: False)
  job=False
  # Enable dependency resolution, optionally consider additional paths to search for easyconfigs (type pathsep-separated list)
  robot=/work/projects/software_stack_alpha/backup/easybuild/easyconfigs
  # Additional paths to consider by robot for easyconfigs (--robot paths get priority) (type pathsep-separated list; default: /mnt/aiongpfs/projects/software_stack_alpha/backup/easybuild/easyconfigs:/work/projects/software_stack_alpha/easybuild/iris/2023b/epyc/software/EasyBuild/4.9.1/easybuild/easyconfigs)
  robot-paths=%(DEFAULT_ROBOT_PATHS)s
  # Additional locations to consider in --search (next to --robot and --robot-paths paths) (type pathsep-separated list)
  #search-paths=
  
  [config]
  # Temporary build path (default: /home/users/gkafanas/.local/easybuild/build)
  buildpath=/tmp/easybuild/iris/2023b/epyc/build
  # Change prefix for buildpath, installpath, sourcepath and repositorypath (used prefix for defaults /home/users/gkafanas/.local/easybuild)
  # NOTE: it seems that installpath is not determined by prefix
  prefix=/work/projects/software_stack_alpha/test/easybuild/iris/2023b/broadwell
  # Install path for software and modules (default: /home/users/gkafanas/.local/easybuild)
  installpath=%(prefix)s
  # Location where container recipe & image will be stored (default: /home/users/gkafanas/.local/easybuild/containers)
  containerpath=%(prefix)s/containers
  # The destination path for the packages built by package-tool (default: /home/users/gkafanas/.local/easybuild/packages)
  packagepath=%(prefix)s/packages
  # Module naming scheme to use (default: EasyBuildMNS)
  module-naming-scheme=CategorizedModuleNamingScheme
  # Repository path, used by repository (is passed as list of arguments to create the repository instance). For more info, use --avail-repositories. (type comma-separated list; default: /home/users/gkafanas/.local/easybuild/ebfiles_repo)
  repositorypath=%(prefix)s/ebfiles_repo
  # Path(s) to where sources should be downloaded (string, colon-separated) (default: /home/users/gkafanas/.local/easybuild/sources)
  sourcepath=%(prefix)s/sources
  # Location of Python module with hook implementations (type str)
  hooks=configuration/hooks/ulhpc.py
  
  # Directory name and format of the log file (type comma-separated tuple; default: easybuild,easybuild-%(name)s-%(version)s-%(date)s.%(time)s.log)
  logfile-format=%(prefix)s/logs/%%(name)s-%%(version)s/%%(date)s_%%(time)s,easybuild-%%(name)s-%%(version)s.log
  
  # Backend to use for submitting jobs (type choice; default: GC3Pie) (choices: GC3Pie, PbsPython, Slurm)
  job-backend=Slurm
  
  [container]
  [easyconfig]
  [github]
  [informative]
  [job]
  
  [override]
  # Wait interval (in seconds) to use when waiting for existing lock to be removed (type <class 'int'>; default: 60)
  #wait-on-lock-interval=
  # Maximum amount of time (in seconds) to wait until lock is released (0 means no waiting at all, exit with error; -1 means no waiting limit, keep waiting) (type <class 'int'>; default: 0)
  #wait-on-lock-limit=
  [package]
  [regtest]
  [software]
  [unittest]

• Error message produced during the installation step (sub-step: == taking care of extensions...):

  ERROR: Build of /work/projects/software_stack_alpha/test/easybuild/iris/2023b/broadwell/software/EasyBuild/4.9.2/easybuild/easyconfigs/p/Python/Python-3.11.5-GCCcore-13.2.0.eb failed (err: 'build failed (first 300 chars): Absolute path /work/projects/software_stack_alpha/test/easybuild/iris/2023b/broadwell/logs/%(name)s-%(version)s/20240728_023527/python passed to update_paths which only expects relative paths.')

Further details

During the failed installation sub-step the file

/work/projects/software_stack_alpha/test/easybuild/iris/2023b/broadwell/logs/%(name)s-%(version)s/%(date)s_%(time)s/python/sitecustomize.py

is created in the log directory (with the references in the log file path not resolved). This seems like a file that should be installed in the installation path according to the file contents:

# sitecustomize.py script installed by EasyBuild,
# to pick up Python packages installed with `--prefix` into folders listed in $EBPYTHONPREFIXES

import os
import site
import sys

# print debug messages when $EBPYTHONPREFIXES_DEBUG is defined
debug = os.getenv('EBPYTHONPREFIXES_DEBUG')

# use prefixes from $EBPYTHONPREFIXES, so they have lower priority than
# virtualenv-installed packages, unlike $PYTHONPATH

ebpythonprefixes = os.getenv('EBPYTHONPREFIXES')

if ebpythonprefixes:
    postfix = os.path.join('lib', 'python' + '.'.join(map(str, sys.version_info[:2])), 'site-packages')
    if debug:
        print("[EBPYTHONPREFIXES] postfix subdirectory to consider in installation directories: %s" % postfix)

    potential_sys_prefixes = (getattr(sys, attr, None) for attr in ("real_prefix", "base_prefix", "prefix"))
    sys_prefix = next(p for p in potential_sys_prefixes if p)
    base_paths = [p for p in sys.path if p.startswith(sys_prefix)]

    for prefix in ebpythonprefixes.split(os.pathsep):
        if debug:
            print("[EBPYTHONPREFIXES] prefix: %s" % prefix)
        sitedir = os.path.join(prefix, postfix)
        if os.path.isdir(sitedir):
            if debug:
                print("[EBPYTHONPREFIXES] adding site dir: %s" % sitedir)
            site.addsitedir(sitedir)

    # Move base python paths to the end of sys.path so modules can override packages from the core Python module
    sys.path = [p for p in sys.path if p not in base_paths] + base_paths

The installation fails when coping sitecustomize.py to its final path.

Following the installation error, the log files for Python are not created. Log files were created without issues for all packages on which Python depends.

[gkaf89]

Call stack produced with traceback.print_stack() at exceptions in lines 986 and 1460 of file easybuild-framework/easybuild/tools/module_generator.py:

  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/work/projects/software_stack_alpha/test/easybuild/iris/2023b/broadwell/software/EasyBuild/4.9.2/lib/python3.11/site-packages/easybuild/main.py", line 789, in <module>
    main(prepared_cfg_data=(init_session_state, eb_go, cfg_settings))
  File "/work/projects/software_stack_alpha/test/easybuild/iris/2023b/broadwell/software/EasyBuild/4.9.2/lib/python3.11/site-packages/easybuild/main.py", line 742, in main
    do_cleanup = process_easystack(options.easystack, args, logfile, testing, init_session_state, do_build)
  File "/work/projects/software_stack_alpha/test/easybuild/iris/2023b/broadwell/software/EasyBuild/4.9.2/lib/python3.11/site-packages/easybuild/main.py", line 289, in process_easystack
    do_cleanup &= process_eb_args([path], eb_go, cfg_settings, modtool, testing, init_session_state,
  File "/work/projects/software_stack_alpha/test/easybuild/iris/2023b/broadwell/software/EasyBuild/4.9.2/lib/python3.11/site-packages/easybuild/main.py", line 567, in process_eb_args
    ecs_with_res = build_and_install_software(ordered_ecs, init_session_state,
  File "/work/projects/software_stack_alpha/test/easybuild/iris/2023b/broadwell/software/EasyBuild/4.9.2/lib/python3.11/site-packages/easybuild/main.py", line 137, in build_and_install_software
    (ec_res['success'], app_log, err) = build_and_install_one(ec, init_env)
  File "/work/projects/software_stack_alpha/test/easybuild/iris/2023b/broadwell/software/EasyBuild/4.9.2/lib/python3.11/site-packages/easybuild/framework/easyblock.py", line 4264, in build_and_install_one
    result = app.run_all_steps(run_test_cases=run_test_cases)
  File "/work/projects/software_stack_alpha/test/easybuild/iris/2023b/broadwell/software/EasyBuild/4.9.2/lib/python3.11/site-packages/easybuild/framework/easyblock.py", line 4143, in run_all_steps
    self.run_step(step_name, step_methods)
  File "/work/projects/software_stack_alpha/test/easybuild/iris/2023b/broadwell/software/EasyBuild/4.9.2/lib/python3.11/site-packages/easybuild/framework/easyblock.py", line 3978, in run_step
    step_method(self)()
  File "/work/projects/software_stack_alpha/test/easybuild/iris/2023b/broadwell/software/EasyBuild/4.9.2/lib/python3.11/site-packages/easybuild/framework/easyblock.py", line 2880, in extensions_step
    fake_mod_data = self.load_fake_module(purge=True, extra_modules=build_dep_mods)
  File "/work/projects/software_stack_alpha/test/easybuild/iris/2023b/broadwell/software/EasyBuild/4.9.2/lib/python3.11/site-packages/easybuild/framework/easyblock.py", line 1708, in load_fake_module
    fake_mod_path = self.make_module_step(fake=True)
  File "/work/projects/software_stack_alpha/test/easybuild/iris/2023b/broadwell/software/EasyBuild/4.9.2/lib/python3.11/site-packages/easybuild/framework/easyblock.py", line 3751, in make_module_step
    txt += self.make_module_extra()
  File "/work/projects/software_stack_alpha/test/easybuild/iris/2023b/broadwell/software/EasyBuild/4.9.2/lib/python3.11/site-packages/easybuild/easyblocks/p/python.py", line 599, in make_module_extra
    txt += self.module_generator.prepend_paths('PYTHONPATH', self.pythonpath)
  File "/work/projects/software_stack_alpha/test/easybuild/iris/2023b/broadwell/software/EasyBuild/4.9.2/lib/python3.11/site-packages/easybuild/tools/module_generator.py", line 265, in prepend_paths
    return self.update_paths(key, paths, prepend=True, allow_abs=allow_abs, expand_relpaths=expand_relpaths)
  File "/work/projects/software_stack_alpha/test/easybuild/iris/2023b/broadwell/software/EasyBuild/4.9.2/lib/python3.11/site-packages/easybuild/tools/module_generator.py", line 1461, in update_paths
    traceback.print_stack()
ERROR: Build of /work/projects/software_stack_alpha/test/easybuild/iris/2023b/broadwell/software/EasyBuild/4.9.2/easybuild/easyconfigs/p/Python/Python-3.11.5-GCCcore-13.2.0.eb failed (err: 'build failed (first 300 chars): Absolute path /work/projects/software_stack_alpha/test/easybuild/iris/2023b/broadwell/logs/   %(name)s-%(version)s/20240729_095326/python passed to update_paths which only expects relative paths.') 

[gkaf89]

Working back from

"/work/projects/software_stack_alpha/test/easybuild/iris/2023b/broadwell/software/EasyBuild/4.9.2/lib/python3.11/site-packages/easybuild/easyblocks/p/python.py", line 599

the problem seems to be caused by the definition of the pythonpath class member variable of the EB_Python block:

# Used for EBPYTHONPREFIXES handler script
self.pythonpath = os.path.join(log_path(), 'python')

Should pythonpath be set to the log directory? Would using build_path instead of log_path make more sense?

[gkaf89]

Using build_path suffers from the same issue with the log_path, the function that uses the pythonpath field variable expects a relative path.

I created a pull request in the easyblocks repository: https://github.com/easybuilders/easybuild-easyblocks/pull/3400

[boegel]

I took a quick look, and it seems like the self.pythonpath in the Python easyblock is set up assuming that the log_path() function will return a relative path, while in your case it's returning an absolute path.

With a default EasyBuild configuration, I'm seeing:

$ python3 -c "from easybuild.tools.options import set_up_configuration; from easybuild.tools.config import log_path; set_up_configuration(silent=True); print(log_path())"
easybuild

When I mimic the relevant part of your configuration though, that changes:

export EASYBUILD_LOGFILE_FORMAT='/prefix/logs/%%(name)s-%%(version)s/%%(date)s_%%(time)s,easybuild-%%(name)s-%%(version)s.log'
python3 -c "from easybuild.tools.options import set_up_configuration; from easybuild.tools.config import log_path; set_up_configuration(silent=True); print(log_path())"
/prefix/logs/%(name)s-%(version)s/%(date)s_%(time)s

The reason the Python easyblock is using log_path() is to figure out the subdirectory of the installation directory in which the EasyBuild installation log file is stored (which by default is easybuild).

So in some sense this problem is self-inflicted, it only occurs because you've customized the logfile-format configuration setting.

That said, the Python easyblock should be robust against this, of course...

[boegel]

After giving this some more thought, I think the directory part of logfile-format is supposed to be a path that's relative to the software installation directory. EasyBuild isn't enforcing that though (it probably should).

[gkaf89]

It seems a bit restrictive the force the installation of the from log files in a path relative to the installation path.

This does not affects our case however, I can change the path to a relative path. It is already pointing inside the installation directory anyway.

[gkaf89]

After giving this some more thought, I think the directory part of logfile-format is supposed to be a path that's relative to the software installation directory. EasyBuild isn't enforcing that though (it probably should).

I think we are dealing with 2 issues here.

• logfile-format is expected to set a relative path, but this requirement is not documented or enforced. In my opinion it is a bit restrictive to force the log files into the directory specified by the easybuild prefix setting, but I would say this is a bug.
• The pythonpath field is set to the directory containing the log files. Since the pythonpath is used to store build artifacts, I believe it is an error to store build artifacts inside a directory used to hold log files.

Am I understanding the use of pythonpath correctly? Is there a more appropriate directory to store the python artifacts?

[boegel]

@gkaf89 Your understanding of the pythonpath variable is correct (we should probably rename it too, because it's a bit confusing).

I think you're right that we should reconsider where the sitecustomize.py script is being stored. That should definitely be somewhere in the installation directory.

That said, calling the easybuild subdirectory of the installation directory the "directory containing the log files" is somewhat incorrect, since other artifacts are also already stored in there, like a copy of the easyconfig file being used, patch files, the reprod directory that contains a copy of the easyblock, etc.

However, since sitecustomize.py is really part of the installation itself, it probably should be stored outside of the easybuild directory.

I guess storing it directory in the installation directory is not a good option, and then the question is in which (custom) subdirectory it should be stored (definitely not <installdir>/python/)...

Any suggestions here?