[Format] Add metadata for single and double precision complex numbers

[asfimport]

Numerical computing libraries like NumPy and TensorFlow feature complex64 and complex128 numbers

Reporter: Wes McKinney / @wesm

PRs and other links:

• GitHub Pull Request #10452
• GitHub Pull Request #10565

_{Note: This issue was originally created as ARROW-638. Please see the migration documentation for further details.}

[asfimport]

Julien Le Dem / @julienledem:

I see several possibilities:

• first class complex types: but we have to curate types so that we don't create too many of them.
• have two float columns: but I guess we want the two values next to each other
• have one float column where odd and even indices are the components of the complex value: which I guess would be the same binary representation as an array of numpy.complex values
• embed floats in a fixed_width_byte_array: we lose meaning in the metadata but also allows a zero copy import of a numpy array.
• define a generic row/struct/compound type which is a fixed width representation of several values: Basically the same binary representation as the previous one but associating metadata to it.

[asfimport]

Wes McKinney / @wesm: As a practical matter, complex64 and complex128 probably need to be compatible with std::complex, which is implemented as a C struct with the values next to each other in memory – this is used in TensorFlow for example:

https://github.com/tensorflow/tensorflow/blob/6dfa434252b1cc3f6c523edc329853b3e8e4911c/tensorflow/core/framework/numeric_types.h#L33

in C++, this boils down to

  template<typename _Tp>
    struct complex
    {

<SNIP>

    private:
      _Tp _M_real;
      _Tp _M_imag;
    };

[asfimport]

Wes McKinney / @wesm: This might be simpler to handle as an ExtensionType in C++ to start

[jorisvandenbossche]

Mailing list discussion about this from 2021: https://www.mail-archive.com/dev@arrow.apache.org/msg23352.html

[kou]

lists.apache.org URL: https://lists.apache.org/thread/bngbnnhyq7lkyx8cg7l2qs1msd0ngg82