search

microsoft / yardl

Tooling for streaming instrument data
https://microsoft.github.io/yardl/
MIT License
29 stars 5 forks source link

Error in Python when serializing complex arrays that are not C-contiguous #148

Closed naegelejd closed 1 month ago

naegelejd commented 2 months ago

Given a model containing arrays of complex numbers, e.g.

MyProtocol: !protocol
  sequence:
    array: complexfloat[coils, samples]

If the NumPy array being serialized is not "C-contiguous", e.g.

with demo.BinaryMyProtocolWriter("out.bin") as w:
    w.write_array(
        np.zeros((2, 128), dtype=np.complex64, order="F")
    )

yardl's binary serializer throws the following error:

Traceback (most recent call last):
  File "/workspaces/yardl/joe/issue-#000/python/test.py", line 25, in <module>
    main()
  File "/workspaces/yardl/joe/issue-#000/python/test.py", line 7, in main
    w.write_array(np.zeros((2, 128), dtype=np.complex64, order="F"))
  File "/workspaces/yardl/joe/issue-#000/python/demo/protocols.py", line 52, in write_array
    self._write_array(value)
  File "/workspaces/yardl/joe/issue-#000/python/demo/binary.py", line 35, in _write_array
    _binary.NDArraySerializer(_binary.complexfloat32_serializer, 2).write(self._stream, value)
  File "/workspaces/yardl/joe/issue-#000/python/demo/_binary.py", line 1235, in write
    self._write_data(stream, value)
  File "/workspaces/yardl/joe/issue-#000/python/demo/_binary.py", line 1129, in _write_data
    self.element_serializer.write_numpy(stream, element)
  File "/workspaces/yardl/joe/issue-#000/python/demo/_binary.py", line 346, in write_numpy
    stream.write(self._struct, value)
  File "/workspaces/yardl/joe/issue-#000/python/demo/_binary.py", line 129, in write
    formatter.pack_into(self._buffer, self._offset, *args)
struct.error: pack_into expected 2 items for packing (got 1)
naegelejd commented 2 months ago

This is caused by the following events:

  1. NDArraySerializerBase checks if the array is trivially serializable: https://github.com/microsoft/yardl/blob/ea42e477ae78939b4327ca690f7c82bf1e2b2162/tooling/internal/python/static_files/_binary.py#L1125-L1129
  2. It is not, because it is not C-contiguous: https://github.com/microsoft/yardl/blob/ea42e477ae78939b4327ca690f7c82bf1e2b2162/tooling/internal/python/static_files/_binary.py#L1147-L1151
  3. Therefore, NDArraySerializerBase calls the ComplexFloat32Serializer.write_numpy method, which is implemented by StructSerializer, and uses struct.Struct.pack_into with the format string <ff, which is not valid for a complex number. https://github.com/microsoft/yardl/blob/ea42e477ae78939b4327ca690f7c82bf1e2b2162/tooling/internal/python/static_files/_binary.py#L624-L626