Open aminiussi opened 2 years ago
jrmadsen
commented
2 years ago I do not understand why you want to link to libpykokkos-core in the first place. Install libpykokkos-core alongside your code. Link to Kokkos::Kokkos and use Kokkos views in your code and as long as there is an import kokkos somewhere (which you could do in your bindings via py::module::import("kokkos")), it should "just work".
jrmadsen
commented
2 years ago If I do not link with libpykokkos-core, the python test script will fail, as it won't find the Kokkos views binding.
Do you have an import kokkos somewhere in the python test script?
aminiussi
commented
2 years ago Hi,
I installed libkkokos-core as a submodule:
aminiussi@login1:/work/aminiussi/fargo/pybind11/rel$ ls external/pykokkos-base/kokkos/
__init__.py libpykokkos.cpython-36m-x86_64-linux-gnu.so __pycache__ test utility.py
aminiussi@login1:/work/aminiussi/fargo/pybind11/rel$ At the begining of th test script:
import sys
sys.path.append("/work/aminiussi/fargo/pybind11/rel/src/fargOCApy")
sys.path.append("/work/aminiussi/fargo/pybind11/rel/external/pykokkos-base")
import argparse
import kokkos
import fargOCApy as fargo
[...]but then the test script will fail with:
63: Traceback (most recent call last):
63: File "/work/aminiussi/fargo/pybind11/rel/test/fargocapy/explore_seq.py", line 52, in <module>
63: vradial = simulation.disk.velocity.radial.field
63: TypeError: Unable to convert function return value to a Python type! The signature was
63: (arg0: fargOCApy.ScalarField) -> Kokkos::View<double const***, Kokkos::LayoutRight, Kokkos::HostSpace>
1/1 Test #63: py_explore_seq ...................***Failed 0.26 sec
So maybe I'm missing something in the instantiation chain.
The py11bind based export code is:
namespace py11 {
void exportScalarField(py::module_& m) {
py::class_<ScalarField, std::shared_ptr<ScalarField>>(m, "ScalarField")
.def_property_readonly("field",
[](ScalarField const& v) {
return Kokkos::Experimental::as_python_type(v.data());
},
"return a view copy",
py::return_value_policy::copy)
.def("__repr__",
[](ScalarField const& d) {
return "fargOCApy.ScalarField";
});
}with v.data() returning a Kokko::View<double***> so it looks like some code related with Kokko::View<double***> wasn't instantiated.
aminiussi
commented
2 years ago Hi Any hint ? Thanks
aminiussi
commented
2 years ago with v.data() returning a Kokko::View<double> so it looks like some code related with Kokko::View<double> wasn't instantiated.
Actually, v.data() returns a Kokko::View<double const***>, when I run the test, I have message:
TypeError: Unregistered type : Kokkos::View<double const***, Kokkos::LayoutRight, Kokkos::HostSpace>Is there a way to get the list of registered types ?
aminiussi
commented
2 years ago Ok so the problem is with the const
Kokkos::View<double const***, Kokkos::LayoutRight, Kokkos::HostSpace>
If I change the code to return a Kokkos::View<double***, Kokkos::LayoutRight, Kokkos::HostSpace> it works just fine.
Is this an expected behavior ?
jrmadsen
commented
2 years ago Yes, they are different types and bindings are not built for const data types.
aminiussi
commented
2 years ago Is there a documented way to explicitly select/instantiate (with cmake help?) the type to instantiate ?
aminiussi
commented
2 years ago Also, now, when in release mode on linux, pykokkos bulid fails with:
cd /gpfswork/rech/oth/roth005/fargo/vanilla/gpu/rel/external/pykokkos-base && /gpfs7kro/gpfslocalsup/spack_soft/cmake/3.21.3/gcc-8.4.1-x74fppdpetkjc5r4jgzwielde7c7o2w3/bin/cmake -E cmake_link_script CMakeFiles/libpykokkos.dir/link.txt --verbose=1
/gpfswork/rech/oth/roth005/fargo/vanilla/external/kokkos/bin/kokkos_launch_compiler /gpfswork/rech/oth/roth005/fargo/vanilla/external/kokkos/bin/nvcc_wrapper /usr/bin/g++ /usr/bin/g++ -fPIC -fopenmp -O3 -DNDEBUG -flto -fno-fat-lto-objects -DKOKKOS_DEPENDENCE -fopenmp -arch=sm_70 -shared -o kokkos/libpykokkos.cpython-36m-x86_64-linux-gnu.so CMakeFiles/libpykokkos-core.dir/src/libpykokkos.cpp.o CMakeFiles/libpykokkos-core.dir/src/backend_version.cpp.o CMakeFiles/libpykokkos-core.dir/src/enumeration.cpp.o CMakeFiles/libpykokkos-core.dir/src/available.cpp.o CMakeFiles/libpykokkos-core.dir/src/common.cpp.o CMakeFiles/libpykokkos-core.dir/src/tools.cpp.o CMakeFiles/libpykokkos-core.dir/src/pool_variants/pools.cpp.o CMakeFiles/libpykokkos-core.dir/src/variants/atomics.cpp.o CMakeFiles/libpykokkos-core.dir/src/variants/views.cpp.o src/pool_variants/CMakeFiles/libpykokkos-pool-variants.dir/xorshift64_pool.cpp.o src/pool_variants/CMakeFiles/libpykokkos-pool-variants.dir/xorshift1024_pool.cpp.o src/variants/CMakeFiles/libpykokkos-variants.dir/concrete_view_int16_right_managed.cpp.o src/variants/CMakeFiles/libpykokkos-variants.dir/concrete_view_int16_left_managed.cpp.o src/variants/CMakeFiles/libpykokkos-variants.dir/concrete_view_int32_right_managed.cpp.o src/variants/CMakeFiles/libpykokkos-variants.dir/concrete_view_int32_left_managed.cpp.o src/variants/CMakeFiles/libpykokkos-variants.dir/concrete_view_int64_right_managed.cpp.o src/variants/CMakeFiles/libpykokkos-variants.dir/concrete_view_int64_left_managed.cpp.o src/variants/CMakeFiles/libpykokkos-variants.dir/concrete_view_uint16_right_managed.cpp.o src/variants/CMakeFiles/libpykokkos-variants.dir/concrete_view_uint16_left_managed.cpp.o src/variants/CMakeFiles/libpykokkos-variants.dir/concrete_view_uint32_right_managed.cpp.o src/variants/CMakeFiles/libpykokkos-variants.dir/concrete_view_uint32_left_managed.cpp.o src/variants/CMakeFiles/libpykokkos-variants.dir/concrete_view_uint64_right_managed.cpp.o src/variants/CMakeFiles/libpykokkos-variants.dir/concrete_view_uint64_left_managed.cpp.o src/variants/CMakeFiles/libpykokkos-variants.dir/concrete_view_float32_right_managed.cpp.o src/variants/CMakeFiles/libpykokkos-variants.dir/concrete_view_float32_left_managed.cpp.o src/variants/CMakeFiles/libpykokkos-variants.dir/concrete_view_float64_right_managed.cpp.o src/variants/CMakeFiles/libpykokkos-variants.dir/concrete_view_float64_left_managed.cpp.o src/variants/CMakeFiles/libpykokkos-variants.dir/dynamic_view_int16_right_managed.cpp.o src/variants/CMakeFiles/libpykokkos-variants.dir/dynamic_view_int16_left_managed.cpp.o src/variants/CMakeFiles/libpykokkos-variants.dir/dynamic_view_int32_right_managed.cpp.o src/variants/CMakeFiles/libpykokkos-variants.dir/dynamic_view_int32_left_managed.cpp.o src/variants/CMakeFiles/libpykokkos-variants.dir/dynamic_view_int64_right_managed.cpp.o src/variants/CMakeFiles/libpykokkos-variants.dir/dynamic_view_int64_left_managed.cpp.o src/variants/CMakeFiles/libpykokkos-variants.dir/dynamic_view_uint16_right_managed.cpp.o src/variants/CMakeFiles/libpykokkos-variants.dir/dynamic_view_uint16_left_managed.cpp.o src/variants/CMakeFiles/libpykokkos-variants.dir/dynamic_view_uint32_right_managed.cpp.o src/variants/CMakeFiles/libpykokkos-variants.dir/dynamic_view_uint32_left_managed.cpp.o src/variants/CMakeFiles/libpykokkos-variants.dir/dynamic_view_uint64_right_managed.cpp.o src/variants/CMakeFiles/libpykokkos-variants.dir/dynamic_view_uint64_left_managed.cpp.o src/variants/CMakeFiles/libpykokkos-variants.dir/dynamic_view_float32_right_managed.cpp.o src/variants/CMakeFiles/libpykokkos-variants.dir/dynamic_view_float32_left_managed.cpp.o src/variants/CMakeFiles/libpykokkos-variants.dir/dynamic_view_float64_right_managed.cpp.o src/variants/CMakeFiles/libpykokkos-variants.dir/dynamic_view_float64_left_managed.cpp.o -Wl,-rpath,/gpfswork/rech/oth/roth005/fargo/vanilla/gpu/rel/external/kokkos/containers/src:/gpfswork/rech/oth/roth005/fargo/vanilla/gpu/rel/external/kokkos/core/src: ../kokkos/containers/src/libkokkoscontainers.so.3.6.00 ../kokkos/core/src/libkokkoscore.so.3.6.00 /gpfslocalsys/cuda/11.2/lib64/stubs/libcuda.so /gpfslocalsys/cuda/11.2/lib64/libcudart.so /usr/lib64/libdl.so
/tmp/ccaFaZFz.s: Assembler messages:
/tmp/ccaFaZFz.s:43: Error: symbol `fatbinData' is already defined
/tmp/ccaFaZFz.s:1264: Error: symbol `fatbinData' is already defined
/tmp/ccaFaZFz.s:2485: Error: symbol `fatbinData' is already defined
/tmp/ccaFaZFz.s:3706: Error: symbol `fatbinData' is already defined
...
lto-wrapper: fatal error: /usr/bin/g++ returned 1 exit status
compilation terminated.
/usr/bin/ld: error: lto-wrapper failed
collect2: error: ld returned 1 exit status
make[2]: *** [external/pykokkos-base/CMakeFiles/libpykokkos.dir/build.make:172: external/pykokkos-base/kokkos/libpykokkos.cpython-36m-x86_64-linux-gnu.so] Error 1
make[2]: Leaving directory '/gpfsdswork/projects/rech/oth/roth005/fargo/vanilla/gpu/rel'
make[1]: *** [CMakeFiles/Makefile2:2469: external/pykokkos-base/CMakeFiles/libpykokkos.dir/all] Error 2
make[1]: Leaving directory '/gpfsdswork/projects/rech/oth/roth005/fargo/vanilla/gpu/rel'
make: *** [Makefile:146: all] Error 2So, this one seems to be fixed be setting ENABLE_THIN_LTO to Off.
What is the rationale for setting it On by default ? it is not the default for pybind11 in general. Does it really improve performances that much ?
Also, I could not see it in the documentation.
Thanks
jrmadsen
commented
2 years ago So, this one seems to be fixed be setting ENABLE_THIN_LTO to Off.
What is the rationale for setting it On by default ?
It's not.
it is not the default for pybind11 in general.
Actually it is. Pybind11 enables LTO by default when you use pybind11_add_module. You have to pass NO_EXTRAS to disable it.
Does it really improve performances that much ?
AFAIR, the real benefit is the reduced size of the binary.
jrmadsen
commented
2 years ago Is there a documented way to explicitly select/instantiate (with cmake help?) the type to instantiate ?
No. Providing instantiations for all possible const-qualified types would be a massively unnecessary compilation overhead. And Python doesn't have a concept of const data so having const data wouldn't even translate. You just need a simple template metaprogramming wrapper to provide you with the view type with the const remove and do a static cast.
jrmadsen
commented
2 years ago #include <KokkosExp_InterOp.hpp> // provides Kokkos::Experimental::python_view_type
// provide an additional layer which removes the const from the data type
template <typename ViewT>
struct python_view_type;
template <template <typename...> class ViewT, typename DataT, typename... ExtraT>
struct python_view_type<ViewT<DataT, ExtraT....>>
{
// key part:
// std::remove_const_t<DataT>
using type = Kokkos::Experimental::python_view_type_t<ViewT<std::remove_const_t<DataT>, ExtraT...>>;
};
template <typename ViewT>
using python_view_type_t = typename python_view_type<ViewT>::type;
template <typename ViewT>
inline auto as_python_type(ViewT _view)
{
return static_cast<python_view_type_t<ViewT>>(_view);
}whenever your bindings return a view to python, e.g. given Kokkos::View<const double***> myview:
m.def("foo", []() {
// does something and returns view
return myview;
}, "Returns a view");you just wrap as_python_type around the returned view, e.g.:
m.def("foo", []() {
// does something and returns view without const
return as_python_type(myview);
}, "Returns a view");it is pretty much the same as doing a const_cast.
Hi,
I have an object model using Kokkos views I want to export to python using pybind11. The export code is in fargOCApy library. So I added the following link directive in my CMake files:
fargoseqrtcontains the object model.If I do that, the link fails with:
A verbose build indicates that the pool.o object does indeed appears twice on the link command:
If I do not link with
libpykokkos-core, the python test script will fail, as it won't find the Kokkos views binding.Am I doing something wrong ?
Thx