search

EESSI / software-layer

Software layer of the EESSI project
https://eessi.github.io/docs/software_layer
GNU General Public License v2.0
23 stars 46 forks source link

Permission denied errors when rebuilding (Python?) bundles with extensions that install binaries #556

Open bedroge opened 5 months ago

bedroge commented 5 months ago

When trying to rebuild Python, hatchling, and Python-bundle-PyPI in #546, we ran into weird permission issues for both hatchling and Python-bundle-PyPI. It's not clear yet what's causing it, but it seems to happen for (Python) bundles that include extensions that not only install files to lib, but also to bin.. The removal step seems to work fine and successfully removes the existing installation, but in the build phase the extension will suddenly see the old bin directory again (with read-only permissions), and fail with errors like:

Successfully built hatchling
Installing collected packages: hatchling
ERROR: Could not install packages due to an OSError.
Consider using the `--user` option or check the permissions.
Traceback (most recent call last):
  File "/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/software/Python/3.11.3-GCCcore-12.3.0/lib/python3.11/site-packages/pip/_internal/commands/install.py", line 449, in run
    installed = install_given_reqs(
                ^^^^^^^^^^^^^^^^^^^
  File "/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/software/Python/3.11.3-GCCcore-12.3.0/lib/python3.11/site-packages/pip/_internal/req/__init__.py", line 72, in install_given_reqs
    requirement.install(
  File "/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/software/Python/3.11.3-GCCcore-12.3.0/lib/python3.11/site-packages/pip/_internal/req/req_install.py", line 800, in install
    install_wheel(
  File "/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/software/Python/3.11.3-GCCcore-12.3.0/lib/python3.11/site-packages/pip/_internal/operations/install/wheel.py", line 731, in install_wheel
    _install_wheel(
  File "/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/software/Python/3.11.3-GCCcore-12.3.0/lib/python3.11/site-packages/pip/_internal/operations/install/wheel.py", line 648, in _install_wheel
    generated_console_scripts = maker.make_multiple(scripts_to_generate)
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/software/Python/3.11.3-GCCcore-12.3.0/lib/python3.11/site-packages/pip/_vendor/distlib/scripts.py", line 436, in make_multiple
    filenames.extend(self.make(specification, options))
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/software/Python/3.11.3-GCCcore-12.3.0/lib/python3.11/site-packages/pip/_internal/operations/install/wheel.py", line 429, in make
    return super().make(specification, options)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/software/Python/3.11.3-GCCcore-12.3.0/lib/python3.11/site-packages/pip/_vendor/distlib/scripts.py", line 425, in make
    self._make_script(entry, filenames, options=options)
  File "/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/software/Python/3.11.3-GCCcore-12.3.0/lib/python3.11/site-packages/pip/_vendor/distlib/scripts.py", line 325, in _make_script
    self._write_script(scriptnames, shebang, script, filenames, ext)
  File "/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/software/Python/3.11.3-GCCcore-12.3.0/lib/python3.11/site-packages/pip/_vendor/distlib/scripts.py", line 293, in _write_script
    self._fileop.write_binary_file(outname, script_bytes)
  File "/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/software/Python/3.11.3-GCCcore-12.3.0/lib/python3.11/site-packages/pip/_vendor/distlib/util.py", line 555, in write_binary_file
    os.remove(path)
PermissionError: [Errno 13] Permission denied: '/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/software/hatchling/1.18.0-GCCcore-12.3.0/bin/hatchling'
 (at easybuild/tools/run.py:682 in parse_cmd_output)

I've tried a lot of possible workarounds in both #546 and #555:

None of them solved the issue, though. So, in the end, I opted for working around it by adding write permissions to the affected installation directories on the Stratum 0, and then the rebuilds completed successfully.

bedroge commented 3 months ago

We're now seeing the same issue for our EESSI-extend module (see #578), which is a bundle. 

bedroge@x86-64-amd-zen3-node2 ~ $ ls -la /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/software/EESSI-extend/2023.06-easybuild/    
total 0
drwxrwxr-x 2 bedroge bedroge 42 Jun  7 11:54 .
drwxr-xr-x 3 bedroge bedroge 31 Jun  7 11:54 ..

# so it looks like the software was successfully removed, but you the old easybuild subdir is still there somehow:

bedroge@x86-64-amd-zen3-node2 ~ $ ls -la /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/software/EESSI-extend/2023.06-easybuild/easybuild/
total 62
dr-xr-xr-x 3 bedroge bedroge  4096 May  7 13:41 .
drwxrwxr-x 2 bedroge bedroge    42 Jun  7 11:54 ..
-r--r--r-- 1 bedroge bedroge  2754 May  7 13:41 EESSI-extend-2023.06-easybuild-easybuild-devel
-r--r--r-- 1 bedroge bedroge  7976 May  7 13:28 EESSI-extend-2023.06-easybuild.eb
-r--r--r-- 1 bedroge bedroge 22875 May  7 13:41 easybuild-EESSI-extend-2023.06-20240507.134135.log.bz2
-rw-rw-r-- 1 bedroge bedroge 19051 May  7 13:41 easybuild-EESSI-extend-2023.06-20240507.134135_test_report.md
dr-xr-xr-x 4 bedroge bedroge  4096 May  7 13:41 reprod

I still don't have a clue why it only happens for some easyconfigs...