Can't initialize pyarray

[semin-park]

I've been trying to get xtensor-python working for the last couple days with no luck.

I have xtl, xtensor, pybind11 all installed (installed from the master branch) and working correctly. Now when I try #include "xtensor-python/pyarray.hpp", this builds alright as well. But when I try to initialize any xt::pyarray at all, I get a runtime segmentation fault.

This is the full code I have:

#define FORCE_IMPORT_ARRAY                // numpy C api loading
#include "xtensor-python/pyarray.hpp"     // Numpy bindings

int main() {
    xt::pyarray<double> a;
}

Above code gives me a segmentation fault. When I check the stack trace, I see that the error is actually occurring on the pybind11 side, so maybe it's because pybind11 and my numpy.h are incompatible or something?

I would really like to use xtensor-python so any help would be highly appreciated!!

[JohanMabille]

You need to initialize the Python interpreter and load numpy:

#include <Python.h>
#define FORCE_IMPORT_ARRAY                // numpy C api loading
#include "xtensor-python/pyarray.hpp"     // Numpy bindings

int main(int argc, char* argv[])
{
    Py_Initialize();
    xt::import_numpy();

    // Add your code here

    Py_Finalize();
    return 0;
}

[SylvainCorlay]

@SPark9625 there are two scenarios for using xtensor-python or pybind11

1. You are writing a Python extension module, and producing a shared library that can be loaded from python. This is the case where you make use of the PYBIND11_MODULE.

2. You are embedding python in your C++ application and producing an executable. In this case, you instantiate the Python interpreter.

In both cases, xtensor-python requires a running Python interpreter.

[wolfv]

I guess this could work as well:

https://pybind11.readthedocs.io/en/stable/advanced/embedding.html

#include <pybind11/embed.h> // everything needed for embedding
namespace py = pybind11;

int main() {
    py::scoped_interpreter guard{}; // start the interpreter and keep it alive
    xt::import_numpy();

    // Add your code here
    py::print("Hello, World!"); // use the Python API
    return 1;
}

where the scoped interpreter is responsible for initializing and finalizing upon destruction.

[semin-park]

Wow thank you! That solved my problem!

[wolfv]

By the way, you can also use xtensor without python (just in case).

Instead of using a pyarray and pytensor, the equivalent pure-C++ containers are xarray and xtensor.

If you use the pure-C++ ones, you don't need to bother with the Python interpreter, or linking to Python / NumPy. But of course, it depends on what you want to do.

[semin-park]

I'm trying to train a board-game-playing AI using the AlphaGo Zero paper. Most of the code that I've written are in Python. However, since training is closely related to the speed of the game engine, I rewrote just the game engine in C++ using vectors to simulate a numpy ndarray.

This was before I got to know xtensor-python though. Now I'm planning to replace my custom Tensor class with xt::pytensor. For my use case, I am right to use pyarray/pytensor right?

Or can I somehow use xtensor/xarrays throughout my code and only convert it into a pytensor/pyarray when returning to python?

[wolfv]

Usually, we try to write generic code using xexpressions and templates, and only use implementation specific containers when binding to another language. If your custom tensor class is pure C++ then you could just as well use an xtensor.

The point of the pyarray and pytensor are that they completely share their memory with NumPy.

You could look at an example of this in this xtensor-showcase repo (I'll publish a blog post regarding this tomorrow, hopefully). https://github.com/wolfv/xtensor-showcase