how to make xtensor pickleable in python?

[xiuliren]

I am following the example of pybind11: https://pybind11.readthedocs.io/en/stable/advanced/classes.html#pickling-support

my code is like:

.def(py::pickle(
            [](const VectorCloud &vc) { // __getstate__
                // Return a tuple that fully encodes the state of the object
                return py::make_tuple( vc.get_points(), vc.get_vectors(), vc.get_kd_tree() );
            },
            [](py::tuple tp) { // __setstate__
                if (tp.size() != 3)
                    throw std::runtime_error("Invalid state!");
                // create a new C++ instance
                VectorCloud vc( tp[0].cast<Points>(), tp[1].cast<Points>(), 
                                tp[2].cast<KDTree>());
                return vc;
            }
        ));

The Points is xtensor<float, 2>

the cast to xtensor<float,2> do not compile and get an error:

/opt/anaconda3/envs/reneu/include/python3.7m/pybind11/cast.h: In instantiation of ‘typename pybind11::detail::make_cas
ter<T>::cast_op_type<typename std::add_rvalue_reference<_Tp>::type> pybind11::detail::cast_op(pybind11::detail::make_c
aster<T>&&) [with T = xt::xtensor_container<xt::uvector<float, std::allocator<float> >, 2, xt::layout_type::row_major,
 xt::xtensor_expression_tag>; typename pybind11::detail::make_caster<T>::cast_op_type<typename std::add_rvalue_referen
ce<_Tp>::type> = xt::xtensor_container<xt::uvector<float, std::allocator<float> >, 2, xt::layout_type::row_major, xt::
xtensor_expression_tag>&; pybind11::detail::make_caster<T> = pybind11::detail::type_caster<xt::xtensor_container<xt::u
vector<float, std::allocator<float> >, 2, xt::layout_type::row_major, xt::xtensor_expression_tag>, void>; typename pyb
ind11::detail::intrinsic_type<T>::type = xt::xtensor_container<xt::uvector<float, std::allocator<float> >, 2, xt::layo
ut_type::row_major, xt::xtensor_expression_tag>]’:
/opt/anaconda3/envs/reneu/include/python3.7m/pybind11/cast.h:1604:22:   required from ‘T pybind11::cast(const pybind11
::handle&) [with T = xt::xtensor_container<xt::uvector<float, std::allocator<float> >, 2, xt::layout_type::row_major, 
xt::xtensor_expression_tag>; typename std::enable_if<(! std::is_base_of<pybind11::detail::pyobject_tag, typename std::
remove_reference<_Tp>::type>::value), int>::type <anonymous> = 0]’
/opt/anaconda3/envs/reneu/include/python3.7m/pybind11/cast.h:1659:74:   required from ‘T pybind11::object::cast() cons
t & [with T = xt::xtensor_container<xt::uvector<float, std::allocator<float> >, 2, xt::layout_type::row_major, xt::xte
nsor_expression_tag>]’
/opt/anaconda3/envs/reneu/include/python3.7m/pybind11/pytypes.h:463:80:   required from ‘T pybind11::detail::accessor<
Policy>::cast() const [with T = xt::xtensor_container<xt::uvector<float, std::allocator<float> >, 2, xt::layout_type::
row_major, xt::xtensor_expression_tag>; Policy = pybind11::detail::accessor_policies::tuple_item]’
src/main.cpp:75:52:   required from here
/opt/anaconda3/envs/reneu/include/python3.7m/pybind11/cast.h:876:43: error: ‘std::remove_reference<pybind11::detail::t
ype_caster<xt::xtensor_container<xt::uvector<float, std::allocator<float> >, 2, xt::layout_type::row_major, xt::xtenso
r_expression_tag>, void>&>::type’ {aka ‘class pybind11::detail::type_caster<xt::xtensor_container<xt::uvector<float, s
td::allocator<float> >, 2, xt::layout_type::row_major, xt::xtensor_expression_tag>, void>’} has no member named ‘opera
tor pybind11::detail::xtensor_type_caster_base<xt::xtensor_container<xt::uvector<float, std::allocator<float> >, 2, xt
::layout_type::row_major, xt::xtensor_expression_tag> >::cast_op_type<xt::xtensor_container<xt::uvector<float, std::al
locator<float> >, 2, xt::layout_type::row_major, xt::xtensor_expression_tag>&&>’
  875 |     return std::move(caster).operator
      |            ~~~~~~~~~~~~~~~~~~~~~~~~~~      
  876 |         typename make_caster<T>::template cast_op_type<typename std::add_rvalue_reference<T>::type>();
      |         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
error: command 'gcc' failed with exit status 1

It seems a cast error, any idea or example to fix it?

[wolfv]

Maybe it would work if you assign the xtensor first to a pytensor or pyarray? Or if that's not enough, then you might be able to extract the underlying numpy array. Even though I think it should be enough as soon as you have a pytensor or pyarray.

[xiuliren]

thanks! you are right. transforming to pytensor works!