Cannot create environment view on the `work1` filesystem at FermiLab

[s-sajid-ali]

Steps to reproduce

Creating a view results in the following error:

bash-4.2$ export SPACK_DISABLE_LOCAL_CONFIG=true 
bash-4.2$ export SPACK_USER_CACHE_PATH=/tmp/spack
bash-4.2$ export SPACK_PYTHON=/usr/bin/python3
bash-4.2$ spack -ddd --verbose env create synergia-devel3-p100-001

Error message

==> Error: [Errno 18] Invalid cross-device link: '/work1/accelsim/spack-shared-v3/spack/var/spack/environments/synergia-devel3-p100-001/.spack-env/._view_link' -> '/work1/accelsim/spack-shared-v3/spack/var/spack/environments/synergia-devel3-p100-001/.spack-env/view'

While the environment is created, a view for the environment is not created. If I install packages as part of the environment and try to regenerate the view, I get the same error.

Information on your system

bash-4.2$ spack debug report
* **Spack:** 0.19.0.dev0
* **Python:** 3.6.8
* **Platform:** linux-scientific7-ivybridge
* **Concretizer:** clingo
bash-4.2$ 

Note that this is a fresh clone of spack on the develop branch.

General information

• [X] I have run spack debug report and reported the version of Spack/Python/Platform
• [X] I have searched the issues of this repo and believe this is not a duplicate
• [X] I have run the failing commands in debug mode and reported the output

[alalazo]

Where does it happen? Do you have a reproducer for this issue?

[s-sajid-ali]

This is on the work1 filesystem on a cluster at Fermilab. I have no issues creating spack environments on the Lustre filesystem : https://computing.fnal.gov/wilsoncluster/filesystems/

[s-sajid-ali]

Updating to the latest develop, I see a different error:

bash-4.2$ spack -d env create test
Traceback (most recent call last):
  File "/work1/accelsim/spack-shared-unused/spack/bin/spack", line 54, in <module>
    sys.exit(main())
  File "/work1/accelsim/spack-shared-unused/spack/lib/spack/spack_installable/main.py", line 35, in main
    import spack.main  # noqa: E402
  File "/work1/accelsim/spack-shared-unused/spack/lib/spack/spack/main.py", line 36, in <module>
    import spack.cmd
  File "/work1/accelsim/spack-shared-unused/spack/lib/spack/spack/cmd/__init__.py", line 26, in <module>
    import spack.environment as ev
  File "/work1/accelsim/spack-shared-unused/spack/lib/spack/spack/environment/__init__.py", line 332, in <module>
    from .environment import (
  File "/work1/accelsim/spack-shared-unused/spack/lib/spack/spack/environment/environment.py", line 42, in <module>
    import spack.user_environment as uenv
  File "/work1/accelsim/spack-shared-unused/spack/lib/spack/spack/user_environment.py", line 8, in <module>
    import spack.build_environment
  File "/work1/accelsim/spack-shared-unused/spack/lib/spack/spack/build_environment.py", line 51, in <module>
    import spack.build_systems.cmake
  File "/work1/accelsim/spack-shared-unused/spack/lib/spack/spack/build_systems/cmake.py", line 17, in <module>
    import spack.package_base
  File "/work1/accelsim/spack-shared-unused/spack/lib/spack/spack/package_base.py", line 59, in <module>
    from spack.installer import InstallError, PackageInstaller
  File "/work1/accelsim/spack-shared-unused/spack/lib/spack/spack/installer.py", line 47, in <module>
    import spack.binary_distribution as binary_distribution
  File "/work1/accelsim/spack-shared-unused/spack/lib/spack/spack/binary_distribution.py", line 44, in <module>
    import spack.relocate as relocate
  File "/work1/accelsim/spack-shared-unused/spack/lib/spack/spack/relocate.py", line 12, in <module>
    import macholib.mach_o
ImportError: bad magic number in 'macholib': b'\x03\xf3\r\n'
bash-4.2$ 

[alalazo]

Can you post:

$ spack debug report

?

[s-sajid-ali]

After a git clean -xfd, I see:

bash-4.2$ spack/bin/spack debug report
* **Spack:** 0.20.0.dev0
* **Python:** 3.6.8
* **Platform:** linux-scientific7-ivybridge
* **Concretizer:** clingo
bash-4.2$ 

And the error for creating the environment is:

bash-4.2$ spack -ddd env create test
==> Error: [Errno 18] Invalid cross-device link: '/work1/accelsim/spack-shared-unused/spack/var/spack/environments/test/.spack-env/._view_link' -> '/work1/accelsim/spack-shared-unused/spack/var/spack/environments/test/.spack-env/view'
bash-4.2$ 

[alalazo]

Hmm, this is weird. The structure you should get is something like:

$ tree -a /home/culpo/PycharmProjects/spack/var/spack/environments/test
/home/culpo/PycharmProjects/spack/var/spack/environments/test
├── .spack-env
│   ├── transaction_lock
│   ├── ._view
│   │   └── ltxmllvql77p5cdfu57vmha7d5jdhkux
│   └── view -> /home/culpo/PycharmProjects/spack/var/spack/environments/test/.spack-env/._view/ltxmllvql77p5cdfu57vmha7d5jdhkux
└── spack.yaml

Apparently Spack isn't able to make the view link. I'd need to have acces to not fly blind, but can you try to change this: https://github.com/spack/spack/blob/65ee864232ae3c0c2802c3d210c4fc618b8e96f2/lib/spack/llnl/util/filesystem.py#L174

into a call to shutil.move and see if it has any effect?

[s-sajid-ali]

into a call to shutil.move and see if it has any effect?

That seems to fix the issue as I no longer see any errors: