search

spack / spack

A flexible package manager that supports multiple versions, configurations, platforms, and compilers.
https://spack.io
Other
4.26k stars 2.26k forks source link

OS-Specific Configuration Scopes #2700

Open citibeth opened 7 years ago

citibeth commented 7 years ago

I'm looking to create a packages.yaml file specific to CentOS-7. I noticed that Spack provides platform-specific scopes. But when I looked at it, it seems that the scope is specific to just linux, not linux-centos7-x86_64, which is what I'm running.

I want an OS-specific scope so I can add things specific to my Linux distro; mostly, directing it to use system-supplied libraries for various packages that I don't want to build (on my system). For example:

   qt:
        paths:
            qt@4.8.5: /
        version: [4.8.5]
        buildable: False

    libxaw:
        paths:
            libxaw@1.0.12: /
        version: [1.0.12]
        buildable: False

Might I suggest a further expansion of scopes, to provide for OS-specific as well as platform-specific scopes? Now the default hierarchy would look like this on my system:

  1. defaults
  2. defaults/linux
  3. defaults/linux-centos7-x86_64
  4. site
  5. site/linux
  6. site/linux-centos7-x86_64
  7. user
  8. user/linux
  9. user/linux-centos7-x86_64
citibeth commented 7 years ago

Another use for this feature would be configuration issues specific to a particular version of macOS.

citibeth commented 7 years ago

OK, I have a sample file here for CentOS7 (below). I like where this is heading because it shows how we can integrate Spack with system-provided package managers. That's an issue we've kicked around a bunch, but have not yet found an easy way forward (see #2116 #541).

The idea here is as follows... something like the CentOS7-specific file below would be included with Spack in an OS-specific config scope. Note the # yum: lines, which show which Yum packages must be installed to provide the same functionality that comes from the Spack package in question.

Automated System Installs

Let's assume the # yum: functionality is added to the YAML grammer; and we will call it system-package, to be OS-neutral. If that is done, then Spack would be able to issue the yum commands needed to support any particular configuration. For example, the command spack system-installs <spec> would do as follows:

  1. Read and integrate the configuration (as described in http://spack.readthedocs.io/en/latest/configuration.html).
  2. Concretize <spec>
  3. Print out a list of all the system-package annotations that occur in the concretized DAG.

Now, the command spack system-installs git would print out:

curl
libcurl-devel

One can then use this in the command to install the system packages required for a spec:

sudo yum install `spack system-installs git`

Alternately, one can just do spack system-installs and install all system-package annotations listed in the configuration. This would be easier to implement, and probably almost as useful.

Overriding Config Scopes

We would ship Spack with as many overrides for each OS as we come up with. Depending on how we feel about the package, we either comment it out in the packages.yaml file we ship, or we leave it in. According to the following criteria:

  1. Security-sensitive stuff should always be left in. Spack should not be building openssl unless the user specifically wants it. Sysadmans are responsible for keeping system software up to date.

  2. Unless a user NEEDS to build a dependency, there is no point in having Spack build it. Stable core-level packages should be left in --- for example, libXaw. This will improve the user's build experience by shortening build times and avoiding possible build / concretization / circular dependency problems.

  3. Higher-level stuff that changes more frequently, we should probably ship but comment it out. If users want to use the system version, they can copy to a user-supplied config scope and uncomment it.

Package Versions

Things work best if Spack can supply the version of a package provided by the system. This will result in package versions being added to support this use; see #2703

# Use system version of certain packages on CentOS 7
#
# Try:
#     yum list installed | grep <package>
#     repoquery -l <package> | grep <expected filename>

packages:
    # Problems in the past installing Python2
    # See: https://github.com/LLNL/spack/issues/1100
    # yum: python
    python:
        paths:
            python@2.7.5: /usr
        version: [2.7.5]

    # Recommended for security reasons
    # Do not install OpenSSL as non-root user.
    # yum: openssl-devl
    openssl:
        paths:
            openssl@1.0.1e: /usr
        version: [1.0.1e]
        buildable: False

    # Recommended, unless your system doesn't provide Qt4
    # yum: qt
    qt:
        paths:
            qt@4.8.5: /usr
        version: [4.8.5]
        buildable: False

    # Recommended, unless your system doesn't provide X11 libs
    # yum: libXaw-devel
    libxaw:
        paths:
            libxaw@1.0.12: /usr
        version: [1.0.12]
        buildable: False

    # Matplotlib uses the system libpng anyway
    # yum: libpng-devel
    libpng:
        paths:
            libpng@1.5.13: /usr
        version: [1.5.13]
        buildable: False

    # yum: libssh2-devel
    libssh2:
        paths:
            libssh2@1.4.3: /usr
        version: [1.4.3]
        buildable: False

    # yum: curl libcurl-devel
    curl:
        paths:
            curl@7.29.0: /
        version: [7.29.0]
        buildable: False

    # yum: xz xz-libs
    xz:
        paths:
            xz@5.2.2: /usr
        version: [5.2.2]
        buildable: False
alalazo commented 2 years ago

Closing, since there has been no activity in a while. Feel free to reopen if you want to continue work on this.

marcmengel commented 2 years ago

This is implemented in #31028 , so I would appreciate if other interested folks would comment on whether my implementation (which allows i.e. $SPACK_ROOT/etc/spack/scientific7/packages.yaml) addresses their concerns.