System:

macOS Catalina
intel i5 4260U
OpenMPI 4.1.5
Apple Clang 12.0.0
CMake 3.27.6
Conda 23.7.2
Python 3.11.5
spack 0.20.1

Steps to reproduce:

First I build Kokkos serial

git clone https://github.com/kokkos/kokkos
cd kokkos
mkdir build && cd build
cmake .. -DKokkos_ENABLE_SERIAL=ON -DCMAKE_CXX_STANDARD=17 -DKokkos_ENABLE_COMPILE_AS_CMAKE_LANGUAGE=ON
make -j4
sudo make install

Then I build pykokkos-base

git clone https://github.com/kokkos/pykokkos-base
cd pykokkos-base/
conda create --name pyk --file requirements.txt
conda activate pyk
python setup.py install -- -DENABLE_LAYOUTS=ON -DENABLE_MEMORY_TRAITS=OFF -DKokkos_ENABLE_SERIAL=ON -DCMAKE_CXX_STANDARD=17 -DKokkos_ENABLE_COMPILE_AS_CMAKE_LANGUAGE=ON

Then I install PyKokkos:

git clone https://github.com/kokkos/pykokkos
cd pykokkos
conda install -c conda-forge pybind11 patchelf
conda install pytest

And run the tests:

python runtests.py

But I get the following error:

dlopen(/Users/keithhbova/anaconda3/envs/pyk/lib/python3.11/site-packages/pykokkos_base-0.0.7-py3.11-macosx-10.15-x86_64.egg/kokkos/libpykokkos.cpython-311-darwin.so, 2): Library not loaded: @rpath/libkokkossimd.3.7.dylib
  Referenced from: /Users/keithhbova/anaconda3/envs/pyk/lib/python3.11/site-packages/pykokkos_base-0.0.7-py3.11-macosx-10.15-x86_64.egg/kokkos/libpykokkos.cpython-311-darwin.so
  Reason: image not found
Traceback (most recent call last):
  File "/Users/keithhbova/anaconda3/envs/pyk/lib/python3.11/site-packages/pykokkos_base-0.0.7-py3.11-macosx-10.15-x86_64.egg/kokkos/__init__.py", line 133, in <module>
    from . import libpykokkos  # noqa: F811
    ^^^^^^^^^^^^^^^^^^^^^^^^^
ImportError: dlopen(/Users/keithhbova/anaconda3/envs/pyk/lib/python3.11/site-packages/pykokkos_base-0.0.7-py3.11-macosx-10.15-x86_64.egg/kokkos/libpykokkos.cpython-311-darwin.so, 2): Library not loaded: @rpath/libkokkossimd.3.7.dylib
  Referenced from: /Users/keithhbova/anaconda3/envs/pyk/lib/python3.11/site-packages/pykokkos_base-0.0.7-py3.11-macosx-10.15-x86_64.egg/kokkos/libpykokkos.cpython-311-darwin.so
  Reason: image not found
Traceback (most recent call last):
  File "/Users/keithhbova/Desktop/kokkos_stuff/pykokkos/runtests.py", line 14, in <module>
    from tests import _logging_probe
  File "/Users/keithhbova/Desktop/kokkos_stuff/pykokkos/tests/_logging_probe.py", line 13, in <module>
    import pykokkos
  File "/Users/keithhbova/Desktop/kokkos_stuff/pykokkos/pykokkos/__init__.py", line 5, in <module>
    from pykokkos.core import CompilationDefaults, Runtime
  File "/Users/keithhbova/Desktop/kokkos_stuff/pykokkos/pykokkos/core/__init__.py", line 1, in <module>
    from pykokkos.interface import *
  File "/Users/keithhbova/Desktop/kokkos_stuff/pykokkos/pykokkos/interface/__init__.py", line 2, in <module>
    from .atomic.atomic_fetch_op import (
  File "/Users/keithhbova/Desktop/kokkos_stuff/pykokkos/pykokkos/interface/atomic/atomic_fetch_op.py", line 6, in <module>
    from ..views import View 
    ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/keithhbova/Desktop/kokkos_stuff/pykokkos/pykokkos/interface/views.py", line 15, in <module>
    from pykokkos.bindings import kokkos
  File "/Users/keithhbova/Desktop/kokkos_stuff/pykokkos/pykokkos/bindings/__init__.py", line 1, in <module>
    import kokkos
  File "/Users/keithhbova/anaconda3/envs/pyk/lib/python3.11/site-packages/pykokkos_base-0.0.7-py3.11-macosx-10.15-x86_64.egg/kokkos/__init__.py", line 133, in <module>
    from . import libpykokkos  # noqa: F811
    ^^^^^^^^^^^^^^^^^^^^^^^^^
ImportError: dlopen(/Users/keithhbova/anaconda3/envs/pyk/lib/python3.11/site-packages/pykokkos_base-0.0.7-py3.11-macosx-10.15-x86_64.egg/kokkos/libpykokkos.cpython-311-darwin.so, 2): Library not loaded: @rpath/libkokkossimd.3.7.dylib
  Referenced from: /Users/keithhbova/anaconda3/envs/pyk/lib/python3.11/site-packages/pykokkos_base-0.0.7-py3.11-macosx-10.15-x86_64.egg/kokkos/libpykokkos.cpython-311-darwin.so
  Reason: image not found

I'm sure I'm doing something wrong. I don't have much experience with conda so I'm not sure where to begin building the python bindings. My goal is to get a working python program on my laptop that can run a conjugate gradient solver on a cpu cluster (amd) or a gpu cluster (nvidia). I was thinking about using mpi4py for the communication. For example, I'd like to be able to write something like this:

import numpy as np

def conjGrad(A,x,b,tol,N):
    r = b - A.dot(x)
    p = r.copy()
    for i in range(N):
        Ap = A.dot(p)
        alpha = np.dot(p,r)/np.dot(p,Ap)
        x = x + alpha*p
        r = b - A.dot(x)
        if np.sqrt(np.sum((r**2))) < tol:
            break
        else:
            beta = -np.dot(r,Ap)/np.dot(p,Ap)
            p = r + beta*p
    return x

...but call it inside a parallel for--and be able to launch the kernel in either Serial, OpenMP or CUDA. Ideally, I'd like to be able to do a git clone, cmake build and pip install on each target system. Thanks, any help or guidance would be greatly appreciated!

tylerjereddy commented 1 year ago

Do we actually support MacOS? IIRC our primary support platform is Linux for now. Sometimes it helps to follow the CI setup steps first, although those don't include GPU support: https://github.com/kokkos/pykokkos/blob/main/.github/workflows/main_ci.yml

NaderAlAwar commented 1 year ago

Unfortunately I can't reproduce the error as I don't have a Mac. I may be able to help though. It seems the issue is that it cannot find the dynamic library file libkokkossimd.3.7.dylib, which should have been installed into your conda environment when you installed pykokkos-base.

As a first step, can you first check whether that dynamic library file exists? It should be located in this directory

/Users/keithhbova/anaconda3/envs/pyk/lib/python3.11/site-packages/pykokkos_base-0.0.7-py3.11-macosx-10.15-x86_64.egg/lib

If the lib directory does not exist, replace it with lib64 in the above path. Let me know if you have trouble finding the file

kokkos / pykokkos