Open stonebig opened 7 years ago
At the moment we have not provided one. We could do this via the install_headers
directive.
My idea on this was to create an xtensor-pip
package that would vendor xtensor and xtensor-python and install the headers the install_headers
pip directive.
The drawback is that the headers are then (typically) installed under the python include directory - which is genrally different from the general include directory of the prefix (include
vs include/pythonX.Y
).
Maybe we could do this with python wheels' data files instead. If you have a suggestion, that would be great?
cc @ghisvail .
If there is a reliable means with python wheels to install into the prefix include directory, that might be the solution, and we could simply add a setup.py to the repository.
Any update on this?
One could do it in pybind11
or numpy
style. First, define header files as package data : they will be copied in the package directory during install. Then define a method get_include
that returns the path to the local include files, see eg. get_include in pybind11.
Of course this does not follow the classical system way to do it, but it is indeed pretty convenient and OS independent in a python context.
Hi, I created a pull request where I copied the respective files from pybind11. Not finished yet but works for me already: #236
I drafted a meta package that provides the headers.
https://github.com/StefanUlbrich/xtensor-cpp
Maybe this is a useful extension to the xtensor-stack?
Hi @StefanUlbrich ,
Thank you for tackling this! I'm not sure yet, but I wonder if building the wheel with scikit-build and CMake could simplify the process:
@SylvainCorlay any thought on this?
@JohanMabille what about conan? most of the xtensor stack is already there
I am not sure if these packages are still maintained. Anyway this is orthogonal to pip, right? I mean, having a package for conan is independent from having / building a wheel for pip, or is there some bridge that makes it easy to have one when we already have the other?
@JohanMabille The packages are still maintained but you are right. My idea was to have a simple package that can be installed and a numeric extension can be set up with minimal effort using setuptools. CMake is of course standard but might be overkill for those not familiar with it–I personally never became comfortable with it when I met it 10yrs ago. In addition, some projects might require lapack and blas installation and static linking (xtensor-blas). Do you think a cookie cutter would be more adequate? I am more than willing to contribute to an "official" endorsed "starter kit"
Maybe this thread is a good way to start?
CMake is of course standard but might be overkill for those not familiar with it
I totally agree, especially for people coming from Python and not having a lot of experience with C++ projects. What I meant is that we (developers and package maintainers) should leverage on existing CMake to build packages in order to make the process easier to maintain (no need to duplicate list of files, build flags, etc).
Do you think a cookie cutter would be more adequate?
We already provide a [cookie cutter] for authoring python extensions based on xtensor, but it might be broken (I haven't tried it for a while) and anyway it deserves some love ;). I think this (cookie cutter) and a wheel are complementary since the cookie cutter does not vendor the dependencies.
I am more than willing to contribute to an "official" endorsed "starter kit"
Thanks that is much appreciated! We have already experienced with scikit-build for building a more complicated stack. The process should be simpler for xtensor since it's a header-only library an it as fewer dependencies.
We split the R-bindings and Julia bindings in two packages, one pure C++ package, and one that vendors the C++ package and its dependencies (see https://github.com/xtensor-stack/Xtensor.R and https://github.com/xtensor-stack/xtensor-r for instance). As per in person conversation with @SylvainCorlay, we should do the same with xtensor-python
: keep the headers and pure C++ stuff in xtensor-python
and move the Python-related stuff in a new repo xtensor.py
, that would vendor the dependencies and build the wheel based on scikit-build
Let us know what you prefer, you can create this xtensor.py repo and iterate on it until you have soething that works, then transfer it to the xtensor-stack organization, or we can create it directly here. In both cases I would be super happy to help with CMake and scikit-build.
We already provide a [cookie cutter] for authoring python extensions based on xtensor, but it might be broken (I haven't tried it for a while) and anyway it deserves some love ;)
CI is set up for the cookiecutters. We should definitely give it a run.
We should do the same with xtensor-python: keep the headers and pure C++ stuff in xtensor-python and move the Python-related stuff in a new repo xtensor.py, that would vendor the dependencies and build the wheel based on scikit-build.
👍 we have done this for both Julia and R.
Hey everyone, Any update on this?
We are swamped with a lot of other tasks and don't have the time to implement it for now. We can provide some guidance if you want to tackle this.
Just switched jobs. I'd love to continue within the next few weeks.
That would be great to have, so that nowadays we could just use scikit-build-core with:
pyproject.toml
[build-system]
requires = ["scikit-build-core", "pybind11", "xtensor-python"]
build-backend = "scikit_build_core.build"
[project]
name = "my_python_extension"
version = "0.0.1"
dependencies = ["numpy"]
CMakeLists.txt
And then python -m pip install .
I'm already using scikit-build-core with xtensor(-python) installed via conda/mamba, which works well although it is slightly less convenient (I need to run python -m pip install . --no-build-isolation
).
I did some experiment here: https://github.com/benbovy/xtensor-python-wheel
I used scikit-build-core together with simple pyproject.toml
files and CMake's FetchContent
to create wheels for the xtensor projects. It is actually very straightforward! The repository also contains a small example that is built from the installed wheels in a github action.
Caveats:
FetchContent
... It is probably possible via ExternalProject_Add
but it is a bit more work.xtensor_python-0.26.1-py3-none-manylinux_2_35_x86_64.whl
whereas we want something like xtensor_python-0.26.1-py3-none-any.whl
. Not sure this is supported by scikit-build-core, but it would actually be useful for header-only (non-python) libraries supporting cmake.CMakeLists.txt
, we need to add find_package()
for each of the xtensor projects (configured targets do not work)
maybe it's impossible...