packages.yaml / spack ignores external OpenCL, only provides pocl

I am new to spack :tada: and look for a build customization to use system's OpenCL instead of installing pocl. My current configuration provides a package called amd-opencl, that concretises the virtual package opencl by pointing to system's installation of OpenCL (coming via ROCm). However, spack does not apply my configuration (spack parsed the packages.yaml though), but only shows pocl. So what are the missing steps?

$ spack providers opencl
    pocl

packages.yaml

# /etc/spack/defaults/packages.yaml 
packages:
  amd-opencl:
    paths:
      amd-opencl@2.0 arch=linux-ubuntu16.04-x86_64: /opt/rocm/opencl
    buildable: False
  all:
    compiler: [gcc, intel, pgi, clang, xl, nag]
    providers:
<..snip..>
      opencl: [amd-opencl]
# or alternatively: /root/.spack/packages.yaml
packages:
  amd-opencl:
    paths:
      amd-opencl: /opt/rocm/opencl # with/without version/arch tried
    buildable: False
  all:
    providers:
      opencl: [amd-opencl]

I also have tried spack clean -a, spack reindex and also deleted the .spack/cache. spack reads the packages.yaml (when there is an error inside, spack stops install).

$ spack --version
0.11.2

$ ls /opt/rocm/opencl/*
/opt/rocm/opencl/bin:
x86_64

/opt/rocm/opencl/include:
CL  opencl-c.h

/opt/rocm/opencl/lib:
x86_64

spack compilers and mirrors

``` $ less .spack/linux/compilers.yaml compilers: - compiler: environment: {} extra_rpaths: [] flags: {} modules: [] operating_system: ubuntu16.04 paths: cc: /usr/bin/gcc-5 cxx: /usr/bin/g++-5 f77: null fc: null spec: gcc@5.4.0 target: x86_64 $ less .spack/linux/mirrors.yaml mirrors: manual: file://~/.spack/manual_mirror local_filesystem: file:///root/.spack/manual_mirror/ ```

spack find

``` -- linux-ubuntu16.04-x86_64 / gcc@5.4.0 ------------------------- binutils@2.28 bison@3.0.4 boost@1.65.1 bzip2@1.0.6 cmake@3.9.4 flex@2.6.4 gettext@0.19.8.1 help2man@1.47.4 hwloc@1.11.8 libedit@3.1-20170329 libelf@0.8.13 libpciaccess@0.13.5 libsigsegv@2.11 libtool@2.4.6 libxml2@2.9.4 llvm@4.0.1 m4@1.4.18 ncurses@6.0 openssl@1.0.2k pcre@8.40 pkg-config@0.29.2 py-appdirs@1.4.3 py-lit@0.5.0 py-packaging@16.8 py-pyparsing@2.2.0 py-setuptools@35.0.2 py-six@1.10.0 python@2.7.14 readline@7.0 sqlite@3.20.0 swig@3.0.12 tar@1.29 tcl@8.6.6 util-macros@1.19.1 xz@5.2.3 zlib@1.2.11 ```

OpenCL version:

$ /opt/rocm/opencl/bin/x86_64/clinfo | grep -i Version
  Platform Version:                              OpenCL 2.1 AMD-APP.internal (2574.0)
  Device OpenCL C version:                       OpenCL C 2.0 
  Driver version:                                2574.0 (HSA1.1,LC)
  Version:                                       OpenCL 1.2

Edit: relates to #7460

My current configuration provides a package called amd-opencl

Did you create a package.py for amd-opencl?

If not, Spack will not choose the amd-opencl external, even if it appears in a providers entry in packages.yaml: that is used to select amongst the Spack packages which advertise providing opencl (those with a provides statement in their package.py).

It may be of interest to be able to specify an external for a virtual package that does not correspond to an existing provider. I think the fastest way to achieve that is to allow spack create to generate a package.py file for you.

In the meantime, you can generate a package.py file, or a quicker hack would be to "pretend" that your implementation was pocl:

  pocl:
    paths:
      pocl@amd: /opt/rocm/opencl # with/without version/arch tried
    buildable: False

ah ok, thank you, now I see :) Is this how the package.py has to look like for external installs? If so, please add an example to the documentation (build customization?).

package.py

from spack import *

class Amdopencl(Package):
    """AMD OpenCL external provider."""

    provides('opencl@:2.0')
    url = "/opt/rocm/opencl/"
    version('2.0')

#    def install(self, spec, prefix):
#      spack_env.prepend_path('CMAKE_PREFIX_PATH', self.prefix.lib)
    def setup_dependent_environment(self, spack_env, run_env, dependent_spec):
        spack_env.prepend_path('CMAKE_PREFIX_PATH', self.prefix.lib)
        run_env.prepend_path('LD_LIBRARY_PATH', self.prefix.lib)
        run_env.set('OPENCL_ROOT', self.prefix)

The quicker hack works (overwrite pocl), too. However, a clean solution instead of overwriting the virtual package is preferred.

Edit: added CMAKE_PREFIX_PATH, etc. (have to read more about this spack part, e.g.OPENCL_ROOT is not set)

Notes:

~~package (e.g. clfft) does not include the env paths over opencl dependency (has to load Amdopencl directly)~~
- Edit: spack load --dependencies clfft
spack did remove /opt/rocm/opencl when reinstalling opencl with --overwrite (how to update path config after change in package.py?)
~~loading opencl loads Amdopencl and pocl together (but only want first one)~~
- Edit: load directly or let spack handle it via the installed dependencies
~~env vars only set, when Amdopencl is directly loaded~~
- Edit: follows from first note

spack did remove /opt/rocm/opencl when reinstalling opencl with --overwrite (how to update path config after change in package.py?)

That sounds bad - sorry! Thanks for adding the output of spack --version but could you also report the commit hash (the newest entry of git log); more generally, did you clone the tip of develop (most users generally do, so that's fine).

The url set in the package.py would not be used to locate the package (and you shouldn't have to set the url at all for recent Spack commits - let me know if that isn't working for you) - Spack should always be using the path listed in packages.yaml. Spack shouldn't attempt to build or download an external package (one with a paths entry).

Could you show me the current contents of your packages.yaml and the output of spack spec -I (whatever package depends on opencl)?

Is this how the package.py has to look like for external installs? If so, please add an example to the documentation (build customization?).

I might need some more specifics on which aspect is unclear in the documentation.

The CMAKE_PREFIX_PATH should already be handled automatically in spack. FYI spack_env refers to the build-time environment (when some package that depends on amdopencl is built) and run_env refers to the run-time environment.

If you want Spack to generate a module file which manages setting OPENCL_ROOT, then defining the package.py as you have done is the best way to go: this sort of detail would not be understood even if there were support for adding a packages.yaml entry for a virtual package implementation with no corresponding package.py.

loading opencl loads Amdopencl and pocl together (but only want first one)

If you you did spack module load opencl then this will find all packages that satisfy it - spack module load amdopencl should fix that for you: opencl is the virtual package and all installed implementations (pocl and amdopencl) would be included).

env vars only set, when Amdopencl is directly loaded

I think this should be happening based on the package.py you wrote. I'm wondering if you have a lingering module file from a previous install that needs to be refreshed: does spack module tcl refresh help?

FYI spack_env refers to the build-time environment

Just chiming in to get this off my chest: Can't we just change the name to build_env?

Thanks a lot for your replies :)

spack did remove /opt/rocm/opencl when reinstalling opencl with --overwrite (how to update path config after change in package.py?)

That sounds bad - sorry!

no problem in this case, but spack should not do this, it did not even inform me, that it is gone. After that reinstall, this package was invalid as the files have been gone. I can open a separate issue on this, if you want.

could you also report the commit hash (the newest entry of git log); more generally, did you clone the tip of develop (most users generally do, so that's fine).

tagged version releases/v0.11.2, so commit hash = c21c03f01a6d0deb23ff7389ed0566de7a1659b2.

The url set in the package.py would not be used to locate the package (and you shouldn't have to set the url at all for recent Spack commits - let me know if that isn't working for you)

An URL seems to be required, because otherwise a python error occurs. I show you that configuration:

# cat var/spack/repos/builtin/packages/Amdopencl/package.py
from spack import *
class Amdopencl(Package):
    """AMD OpenCL external provider."""
    provides('opencl@:2.0')
    #url = "/opt/rocm/opencl/"
    #version('2.0')
    #def install(self, spec, prefix):
    def setup_dependent_environment(self, spack_env, run_env, dependent_spec):
        spack_env.prepend_path('CMAKE_PREFIX_PATH', self.prefix.lib)
        run_env.prepend_path('LD_LIBRARY_PATH', self.prefix.lib)
        run_env.set('OPENCL_ROOT', self.prefix)

# cat ~/.spack/packages.yaml
packages:
  Amdopencl:
    paths:
      Amdopencl@2.0: /opt/rocm/opencl/
    buildable: False
  all:
    providers:
      opencl: [Amdopencl]

$ spack spec -I clfft
Input spec
--------------------------------
     clfft

Normalized
--------------------------------
     clfft
         ^cmake
         ^opencl@1.2:

Concretized
--------------------------------
==> Error: Class constructor failed for package 'builtin.Amdopencl'.

Caused by:
NoURLError: Package Amdopencl has no version with a URL.
  File "/opt/spack/lib/spack/spack/repository.py", line 836, in get
    self._instances[key] = package_class(copy)
  File "/opt/spack/lib/spack/spack/package.py", line 601, in __init__
    f = fs.for_package_version(self, self.version)
  File "/opt/spack/lib/spack/spack/fetch_strategy.py", line 949, in for_package_version
    attrs['url'] = pkg.url_for_version(version)
  File "/opt/spack/lib/spack/spack/package.py", line 719, in url_for_version
    raise NoURLError(cls)

I might need some more specifics on which aspect is unclear in the documentation.

First I did not know that a package.py is required (and where to place it) just for an external library and how to provide such package for virtual packages and how the package.py for external libraries has to look like. By reading some issues on this, I was under the assumption, only a package.yaml is sufficient for the minimal case, so I have not bothered the real package configuration thing. Furthermore, in the doc on build customization there is a section about external packages, but no word on package.py.

At the moment, my config for Amdopencl fails when url and the like are not provided. Maybe it is relaxed in develop, but we would like to stick with latest release version for now.

does spack module tcl refresh help?

spack module refresh refreshed the tcl files, good to know and helped a bit (clfft was installed against pocl before) :)
Here is the spack spec -I clfft (opencl package configuration is like above, just with enabled url and version).

``` $ spack spec -I clfft Input spec -------------------------------- clfft Normalized -------------------------------- clfft ^cmake ^opencl@1.2: Concretized -------------------------------- [+] clfft@2.12.2%gcc@5.4.0 build_type=RelWithDebInfo +client arch=linux-ubuntu16.04-x86_64 [+] ^Amdopencl@2.0%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64 [+] ^boost@1.65.1%gcc@5.4.0+atomic+chrono~clanglibcpp+date_time~debug+filesystem~graph~icu+iostreams+locale+log+math~mpi+multithreaded patches=2ab6c72d03dec6a4ae20220a9dfd5c8c572c5294252155b85c6874d97c323199 +program_options~python+random+regex+serialization+shared+signals~singlethreaded+system~taggedlayout+test+thread+timer~versionedlayout+wave arch=linux-ubuntu16.04-x86_64 [+] ^bzip2@1.0.6%gcc@5.4.0+shared arch=linux-ubuntu16.04-x86_64 [+] ^zlib@1.2.11%gcc@5.4.0+optimize+pic+shared arch=linux-ubuntu16.04-x86_64 [+] ^cmake@3.9.4%gcc@5.4.0~doc+ncurses+openssl+ownlibs~qt arch=linux-ubuntu16.04-x86_64 [+] ^ncurses@6.0%gcc@5.4.0 patches=4110a40613b800da2b2888c352b64c75a82809d48341061e4de5861e8b28423f,f84b2708a42777aadcc7f502a261afe10ca5646a51c1ef8b5e60d2070d926b57 ~symlinks arch=linux-ubuntu16.04-x86_64 [+] ^pkg-config@0.29.2%gcc@5.4.0+internal_glib arch=linux-ubuntu16.04-x86_64 [+] ^openssl@1.0.2k%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64 ```

When loading clfft I expected that the required includes for OpenCL are also loaded (from Amdopencl). My error was here, that --dependencies was missing ^^.

$ spack load --dependencies clfft
$ module list
Currently Loaded Modulefiles:
  1) Amdopencl-2.0-gcc-5.4.0-tbue4wx   3) zlib-1.2.11-gcc-5.4.0-5nus6kn     5) clfft-2.12.2-gcc-5.4.0-mpwvzcs
  2) bzip2-1.0.6-gcc-5.4.0-ufczdvs     4) boost-1.65.1-gcc-5.4.0-xxqnbqq

If you you did spack module load opencl then this will find all packages that satisfy it

Ok, would it be better, when you load a virtual package, that then only the first available one is loaded (provided in providers: opencl: [Amdopencl, pocl])? So it would be more generic, but I do not know at the moment, what the gain would be.

present situation:

url issue is still there (if left out then error occurs)
how to update path config after change in package.py without reinstalling
- ... or ... how to prevent uninstalling external data when calling reinstall (that is a critical issue imho)

... or ... how to prevent uninstalling external data when calling reinstall (that is a critical issue imho)

I can replicate this now: it is a bug with the spack install command when the --overwrite option is used: this command does not check whether a package is an external and replaces the directory. I can work on this tomorrow, generally for now if you explicitly split this into two steps like

spack uninstall foo
spack install foo

it will avoid this issue.

tagged version releases/v0.11.2, so commit hash = c21c03f

Got it - the URL was required at that point. Would you be interested in trying the latest develop? Releases for Spack are typically intermittent

I might need some more specifics on which aspect is unclear in the documentation.

First I did not know that a package.py is required (and where to place it) just for an external library and how to provide such package for virtual packages

Got it, https://spack.readthedocs.io/en/latest/tutorial_configuration.html#external-packages doesn't clarify that, although the section is long and there may be a better way of informing the user about this (perhaps an error message)

and how the package.py for external libraries has to look like.

This part can depend on the package (e.g. in your case you wanted to set the OPENCL_ROOT variable), I think most considerations from https://spack.readthedocs.io/en/latest/tutorial_advanced_packaging.html can apply, it may be worth clarifying this in some sort of dedicated "external packages" section.

Would you be interested in trying the latest develop? Releases for Spack are typically intermittent

alright, I tried and the URL is indeed no longer required :)

Got it, https://spack.readthedocs.io/en/latest/tutorial_configuration.html#external-packages doesn't clarify that, although the section is long and there may be a better way of informing the user about this (perhaps an error message)

Yes, an error message would be even better. And for the doc I could provide the OpenCL external package as an example

with the comment, that the URL will be no longer required after v0.11.2
with the link to advanced packaging for more details on packaging
with the comment about CMAKE_PREFIX_PATH, that spack already tries to fill it automatically

spack / spack