spack install of astropy fails in extension-helpers

[lgarrison]

When installing astropy in spack, a curious situation comes up:

    File "setup.py", line 68, in <module>
      setup(ext_modules=get_extensions())
    File "/mnt/sw/nix/store/bcdmjrxn9z9zaa04k2gvgijic57vahjw-py-extension-helpers-1.0.0/lib/python3.9/site-packages/extension_helpers/_setup_helpers.py", line 97, in get_extensions
      shutil.copy(os.path.join(src_path, 'compiler.c'),
    File "/mnt/sw/nix/store/y80v9c1vpcqjlvxrkfxw5hyvrhjiixjr-python-3.9.15/lib/python3.9/shutil.py", line 427, in copy
      copyfile(src, dst, follow_symlinks=follow_symlinks)
    File "/mnt/sw/nix/store/y80v9c1vpcqjlvxrkfxw5hyvrhjiixjr-python-3.9.15/lib/python3.9/shutil.py", line 266, in copyfile
      with open(dst, 'wb') as fdst:
  PermissionError: [Errno 13] Permission denied: './astropy/_compiler.c'
  error: subprocess-exited-with-error

My diagnosis is this: first, extension-helpers is installed and spack decides to make it read-only; no problem. Then, astropy is built from source, calling extension-helpers' get_extensions() function twice: once in pip's "Preparing metadata" phase, and once to actually build it. The first time, extension-helpers/src/_compiler.c is copied to ./astropy/_compiler.c without issue. The second time, ./astropy/_compiler.c already exists and is read-only, because it was copied along with its 0444 permissions the first time. So the copy fails the second time.

I'm not 100% sure where the permissions on _compiler.c are being modified; I'm not even 100% sure it's spack. The build system is pretty complicated. But regardless, I think this issue would arise any time an installation of extension-helpers is made read-only, which seems reasonable, if uncommon.

I worked around this issue by modifying extension-helpers to simply remove the _compiler.c file before attempting to overwrite it. os.remove() doesn't care about read-only (like rm), but shutil.copy() does. Go figure!

This seems like a pretty safe, general change, so I wanted to see if (1) you agree this is a reasonable modification, and (2) you would be interested in a PR. The diff is simply:

diff --git a/extension_helpers/_setup_helpers.py b/extension_helpers/_setup_helpers.py
index 7e766da..8636873 100644
--- a/extension_helpers/_setup_helpers.py
+++ b/extension_helpers/_setup_helpers.py
@@ -94,8 +94,13 @@ def get_extensions(srcdir='.'):
     if len(ext_modules) > 0:
         main_package_dir = min(packages, key=len)
         src_path = os.path.join(os.path.dirname(__file__), 'src')
-        shutil.copy(os.path.join(src_path, 'compiler.c'),
-                    os.path.join(srcdir, main_package_dir, '_compiler.c'))
+        dst_file = os.path.join(srcdir, main_package_dir, '_compiler.c')
+        try:
+            # remove dst_file in case it exists but is read-only
+            os.remove(dst_file)
+        except FileNotFoundError:
+            pass
+        shutil.copy(os.path.join(src_path, 'compiler.c'), dst_file)
         ext = Extension(main_package_dir + '.compiler_version',
                         [os.path.join(main_package_dir, '_compiler.c')])
         ext_modules.append(ext)

More build log details

``` ==> py-astropy: Executing phase: 'install' ==> [2023-02-06-18:19:11.359931] '/mnt/sw/nix/store/y80v9c1vpcqjlvxrkfxw5hyvrhjiixjr-python-3.9.15/bin/python3.9' '-m' 'pip' '-vvv' '--no-input' '--no-cache-dir' '--disable-pip-version-check' 'install' '--no-deps' '--ignore-installed' '--no-build-isolation' '--no-warn-script-location' '--no-index' '--prefix=/mnt/sw/nix/store/p16j2flby89r9m29p2bnd7cagqp780sf-py-astropy-5.1' '--install-option=--use-system-libraries' '--install-option=--use-system-erfa' '--install-option=--use-system-wcslib' '--install-option=--use-system-cfitsio' '--install-option=--use-system-expat' '.' WARNING: Disabling all use of wheels due to the use of --build-option / --global-option / --install-option. Using pip 22.2.2 from /mnt/sw/nix/store/hmyviwx9ydli42d7al14dl0a644jr17f-py-pip-22.2.2/lib/python3.9/site-packages/pip (python 3.9) Non-user install due to --prefix or --target option Ignoring indexes: https://pypi.org/simple Created temporary directory: /dev/shm/nix-build-py-astropy-5.1.drv-1/pip-ephem-wheel-cache-jjoo0f0a Created temporary directory: /dev/shm/nix-build-py-astropy-5.1.drv-1/pip-build-tracker-gnsrljxg Initialized build tracking at /dev/shm/nix-build-py-astropy-5.1.drv-1/pip-build-tracker-gnsrljxg Created build tracker: /dev/shm/nix-build-py-astropy-5.1.drv-1/pip-build-tracker-gnsrljxg Entered build tracker: /dev/shm/nix-build-py-astropy-5.1.drv-1/pip-build-tracker-gnsrljxg Created temporary directory: /dev/shm/nix-build-py-astropy-5.1.drv-1/pip-install-tjs3ee_h Processing /dev/shm/nix-build-py-astropy-5.1.drv-1/nixbld1/spack-stage-py-astropy-5.1-p16j2flby89r9m29p2bnd7cagqp780sf/spack-src Added file:///dev/shm/nix-build-py-astropy-5.1.drv-1/nixbld1/spack-stage-py-astropy-5.1-p16j2flby89r9m29p2bnd7cagqp780sf/spack-src to build tracker '/dev/shm/nix-build-py-astropy-5.1.drv-1/pip-build-tracker-gnsrljxg' Created temporary directory: /dev/shm/nix-build-py-astropy-5.1.drv-1/pip-modern-metadata-xuibowpl Preparing metadata (pyproject.toml): started Running command Preparing metadata (pyproject.toml) /mnt/sw/nix/store/d9ksrdh7n65r9f6l2yfk9bh9scnxy03m-py-setuptools-62.6.0/lib/python3.9/site-packages/setuptools/config/setupcfg.py:463: SetuptoolsDeprecationWarning: The license_file parameter is deprecated, use license_files instead. warnings.warn(msg, warning_class) running dist_info creating /dev/shm/nix-build-py-astropy-5.1.drv-1/pip-modern-metadata-xuibowpl/astropy.egg-info writing /dev/shm/nix-build-py-astropy-5.1.drv-1/pip-modern-metadata-xuibowpl/astropy.egg-info/PKG-INFO writing dependency_links to /dev/shm/nix-build-py-astropy-5.1.drv-1/pip-modern-metadata-xuibowpl/astropy.egg-info/dependency_links.txt writing entry points to /dev/shm/nix-build-py-astropy-5.1.drv-1/pip-modern-metadata-xuibowpl/astropy.egg-info/entry_points.txt writing requirements to /dev/shm/nix-build-py-astropy-5.1.drv-1/pip-modern-metadata-xuibowpl/astropy.egg-info/requires.txt writing top-level names to /dev/shm/nix-build-py-astropy-5.1.drv-1/pip-modern-metadata-xuibowpl/astropy.egg-info/top_level.txt writing manifest file '/dev/shm/nix-build-py-astropy-5.1.drv-1/pip-modern-metadata-xuibowpl/astropy.egg-info/SOURCES.txt' listing git files failed - pretending there aren't any reading manifest file '/dev/shm/nix-build-py-astropy-5.1.drv-1/pip-modern-metadata-xuibowpl/astropy.egg-info/SOURCES.txt' reading manifest template 'MANIFEST.in' warning: no files found matching '*.templ' under directory 'astropy' warning: no files found matching 'astropy/astropy.cfg' warning: no files found matching '*' under directory 'scripts' warning: no files found matching '*' under directory 'static' no previously-included directories found matching 'astropy/_dev' no previously-included directories found matching 'docs/_build' no previously-included directories found matching 'build' warning: no previously-included files matching '*.pyc' found anywhere in distribution warning: no previously-included files matching '*.o' found anywhere in distribution adding license file 'LICENSE.rst' writing manifest file '/dev/shm/nix-build-py-astropy-5.1.drv-1/pip-modern-metadata-xuibowpl/astropy.egg-info/SOURCES.txt' creating '/dev/shm/nix-build-py-astropy-5.1.drv-1/pip-modern-metadata-xuibowpl/astropy-5.1.dist-info' Preparing metadata (pyproject.toml): finished with status 'done' Source in /dev/shm/nix-build-py-astropy-5.1.drv-1/nixbld1/spack-stage-py-astropy-5.1-p16j2flby89r9m29p2bnd7cagqp780sf/spack-src has version 5.1, which satisfies requirement astropy==5.1 from file:///dev/shm/nix-build-py-astropy-5.1.drv-1/nixbld1/spack-stage-py-astropy-5.1-p16j2flby89r9m29p2bnd7cagqp780sf/spack-src Removed astropy==5.1 from file:///dev/shm/nix-build-py-astropy-5.1.drv-1/nixbld1/spack-stage-py-astropy-5.1-p16j2flby89r9m29p2bnd7cagqp780sf/spack-src from build tracker '/dev/shm/nix-build-py-astropy-5.1.drv-1/pip-build-tracker-gnsrljxg' Created temporary directory: /dev/shm/nix-build-py-astropy-5.1.drv-1/pip-unpack-h8qbi0x4 Building wheels for collected packages: astropy Created temporary directory: /dev/shm/nix-build-py-astropy-5.1.drv-1/pip-wheel-y1fm63d2 Destination directory: /dev/shm/nix-build-py-astropy-5.1.drv-1/pip-wheel-y1fm63d2 Building wheel for astropy (pyproject.toml): started Running command Building wheel for astropy (pyproject.toml) Traceback (most recent call last): File "/mnt/sw/nix/store/hmyviwx9ydli42d7al14dl0a644jr17f-py-pip-22.2.2/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 363, in main() File "/mnt/sw/nix/store/hmyviwx9ydli42d7al14dl0a644jr17f-py-pip-22.2.2/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 345, in main json_out['return_val'] = hook(**hook_input['kwargs']) File "/mnt/sw/nix/store/hmyviwx9ydli42d7al14dl0a644jr17f-py-pip-22.2.2/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 261, in build_wheel return _build_backend().build_wheel(wheel_directory, config_settings, File "/mnt/sw/nix/store/d9ksrdh7n65r9f6l2yfk9bh9scnxy03m-py-setuptools-62.6.0/lib/python3.9/site-packages/setuptools/build_meta.py", line 244, in build_wheel return self._build_with_temp_dir(['bdist_wheel'], '.whl', File "/mnt/sw/nix/store/d9ksrdh7n65r9f6l2yfk9bh9scnxy03m-py-setuptools-62.6.0/lib/python3.9/site-packages/setuptools/build_meta.py", line 229, in _build_with_temp_dir self.run_setup() File "/mnt/sw/nix/store/d9ksrdh7n65r9f6l2yfk9bh9scnxy03m-py-setuptools-62.6.0/lib/python3.9/site-packages/setuptools/build_meta.py", line 174, in run_setup exec(compile(code, __file__, 'exec'), locals()) File "setup.py", line 68, in setup(ext_modules=get_extensions()) File "/mnt/sw/nix/store/bcdmjrxn9z9zaa04k2gvgijic57vahjw-py-extension-helpers-1.0.0/lib/python3.9/site-packages/extension_helpers/_setup_helpers.py", line 97, in get_extensions shutil.copy(os.path.join(src_path, 'compiler.c'), File "/mnt/sw/nix/store/y80v9c1vpcqjlvxrkfxw5hyvrhjiixjr-python-3.9.15/lib/python3.9/shutil.py", line 427, in copy copyfile(src, dst, follow_symlinks=follow_symlinks) File "/mnt/sw/nix/store/y80v9c1vpcqjlvxrkfxw5hyvrhjiixjr-python-3.9.15/lib/python3.9/shutil.py", line 266, in copyfile with open(dst, 'wb') as fdst: PermissionError: [Errno 13] Permission denied: './astropy/_compiler.c' error: subprocess-exited-with-error × Building wheel for astropy (pyproject.toml) did not run successfully. │ exit code: 1 ╰─> See above for output. note: This error originates from a subprocess, and is likely not a problem with pip. full command: /mnt/sw/nix/store/y80v9c1vpcqjlvxrkfxw5hyvrhjiixjr-python-3.9.15/bin/python3.9 /mnt/sw/nix/store/hmyviwx9ydli42d7al14dl0a644jr17f-py-pip-22.2.2/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py build_wheel /dev/shm/nix-build-py-astropy-5.1.drv-1/tmpudc3nck7 cwd: /dev/shm/nix-build-py-astropy-5.1.drv-1/nixbld1/spack-stage-py-astropy-5.1-p16j2flby89r9m29p2bnd7cagqp780sf/spack-src Building wheel for astropy (pyproject.toml): finished with status 'error' ERROR: Failed building wheel for astropy Failed to build astropy ERROR: Could not build wheels for astropy, which is required to install pyproject.toml-based projects Exception information: Traceback (most recent call last): File "/mnt/sw/nix/store/hmyviwx9ydli42d7al14dl0a644jr17f-py-pip-22.2.2/lib/python3.9/site-packages/pip/_internal/cli/base_command.py", line 167, in exc_logging_wrapper status = run_func(*args) File "/mnt/sw/nix/store/hmyviwx9ydli42d7al14dl0a644jr17f-py-pip-22.2.2/lib/python3.9/site-packages/pip/_internal/cli/req_command.py", line 247, in wrapper return func(self, options, args) File "/mnt/sw/nix/store/hmyviwx9ydli42d7al14dl0a644jr17f-py-pip-22.2.2/lib/python3.9/site-packages/pip/_internal/commands/install.py", line 431, in run raise InstallationError( pip._internal.exceptions.InstallationError: Could not build wheels for astropy, which is required to install pyproject.toml-based projects Removed build tracker: '/dev/shm/nix-build-py-astropy-5.1.drv-1/pip-build-tracker-gnsrljxg' Traceback (most recent call last): File "/mnt/sw/nix/store/x791w7gixpcfhqs0bdhgq9q33w8z6lvv-builder.py", line 75, in spack.installer.build_process(pkg, opts) File "/mnt/sw/nix/store/rz0ypy576hbkx883b12fj9v1sd4nfxs9-spack/lib/spack/spack/installer.py", line 2077, in build_process return installer.run() File "/mnt/sw/nix/store/rz0ypy576hbkx883b12fj9v1sd4nfxs9-spack/lib/spack/spack/installer.py", line 1941, in run self._real_install() File "/mnt/sw/nix/store/rz0ypy576hbkx883b12fj9v1sd4nfxs9-spack/lib/spack/spack/installer.py", line 2037, in _real_install phase_fn.execute() File "/mnt/sw/nix/store/x791w7gixpcfhqs0bdhgq9q33w8z6lvv-builder.py", line 61, in wrapPhase return f(*args) File "/mnt/sw/nix/store/rz0ypy576hbkx883b12fj9v1sd4nfxs9-spack/lib/spack/spack/builder.py", line 430, in execute self.phase_fn(pkg, pkg.spec, pkg.prefix) File "/mnt/sw/nix/store/rz0ypy576hbkx883b12fj9v1sd4nfxs9-spack/lib/spack/spack/builder.py", line 306, in _adapter return phase_fn(spec, prefix) File "/mnt/sw/nix/store/rz0ypy576hbkx883b12fj9v1sd4nfxs9-spack/lib/spack/spack/builder.py", line 150, in __forward return self.__getattr__(fn_name)(*args, **kwargs) File "/mnt/sw/nix/store/rz0ypy576hbkx883b12fj9v1sd4nfxs9-spack/lib/spack/spack/builder.py", line 72, in __call__ return self.phase_fn(self.builder.pkg, spec, prefix) File "/mnt/sw/nix/store/rz0ypy576hbkx883b12fj9v1sd4nfxs9-spack/lib/spack/spack/build_systems/python.py", line 463, in install pip(*args) File "/mnt/sw/nix/store/rz0ypy576hbkx883b12fj9v1sd4nfxs9-spack/lib/spack/spack/util/executable.py", line 238, in __call__ raise ProcessError("Command exited with status %d:" % proc.returncode, long_msg) spack.util.executable.ProcessError: Command exited with status 1: '/mnt/sw/nix/store/y80v9c1vpcqjlvxrkfxw5hyvrhjiixjr-python-3.9.15/bin/python3.9' '-m' 'pip' '-vvv' '--no-input' '--no-cache-dir' '--disable-pip-version-check' 'install' '--no-deps' '--ignore-installed' '--no-build-isolation' '--no-warn-script-location' '--no-index' '--prefix=/mnt/sw/nix/store/p16j2flby89r9m29p2bnd7cagqp780sf-py-astropy-5.1' '--install-option=--use-system-libraries' '--install-option=--use-system-erfa' '--install-option=--use-system-wcslib' '--install-option=--use-system-cfitsio' '--install-option=--use-system-expat' '.' builder for '/mnt/sw/nix/store/hzdw358lnnl132lvpcsamgiwsfxzil3s-py-astropy-5.1.drv' failed with exit code 1 error: build of '/mnt/sw/nix/store/hzdw358lnnl132lvpcsamgiwsfxzil3s-py-astropy-5.1.drv' failed ```

Thanks!

[astrofrog]

Thanks for the report! Another possible fix that would avoid having to call os.remove would be to check if the destination file is the same as the file being copied and to skip the copy if so?

[lgarrison]

Re-reading my own bug report, I'm now wondering if an even easier fix is to do the copy in a way that doesn't preserve the read-only permissions, which could be accomplished by using shutil.copyfile() instead of shutil.copy(). What would you think about that?