pybind / pybind11

Seamless operability between C++11 and Python
https://pybind11.readthedocs.io/
Other
15.8k stars 2.12k forks source link

Add dtype::normalized_num and dtype::num_of #5429

Closed MaartenBaert closed 5 days ago

MaartenBaert commented 2 weeks ago

Description

This commit adds two functions, dtype::normalized_num and dtype::num_of, which make it possible to use a switch statement for dynamic dispatch based on the dtype of an array, like this:

switch (arr.dtype().normalized_num()) {
    case py::dtype::num_of<int8_t>():
        return foo<int8_t>(arr);
    case py::dtype::num_of<uint8_t>():
        return foo<uint8_t>(arr);
    case py::dtype::num_of<int16_t>():
        return foo<int16_t>(arr);
    case py::dtype::num_of<uint16_t>():
        return foo<uint16_t>(arr);
// ...

dtype::normalized_num does the same as dtype::num, except the type number is normalized to match the one used in npy_format_descriptor. This is needed because dtype::num can return different values for equivalent types, e.g. even though long may be equivalent to int or long long, they still have different type numbers (at least when the dtype is constructed by type number, rather than with dtype::of<T>()). Without normalization, this leads to strange bugs where the switch statement inexplicably rejects arrays from certain sources despite superficially having the correct dtype. E.g. on x86_64 (linux), dtypes long (7) and longlong (9) both appear as int64 but have different type numbers:

In [1]: np.dtype('long')
Out[1]: dtype('int64')

In [2]: np.dtype('longlong')
Out[2]: dtype('int64')

In [3]: np.dtype('long').num
Out[3]: 7

In [4]: np.dtype('longlong').num
Out[4]: 9

dtype::num_of<T> is simply the constexpr equivalent of dtype::of<T>().num(), so it can be used for cases in a switch statement.

While developing this, I ran into an inconsistency in the behavior of is_fmt_numeric which is relevant for the implementation of dtype::normalized_num, I have reported this as a separate issue: #5428

Suggested changelog entry:

Add dtype::normalized_num and dtype::num_of
rwgk commented 5 days ago

Thanks @MaartenBaert, and thanks a lot for the review @seberg!