feat: Remove sdk, use all python

[tlambert03]

This re-implements the complete sdk in python, with many thanks to the great folks at LIM for all their help.

cleaned up version of #122 closes #122

[codecov[bot]]

Codecov Report

Merging #135 (77d87ed) into main (2f1f137) will increase coverage by 4.83%. The diff coverage is 91.55%.

@@            Coverage Diff             @@
##             main     #135      +/-   ##
==========================================
+ Coverage   86.55%   91.38%   +4.83%     
==========================================
  Files          10       13       +3     
  Lines        1145     1962     +817     
==========================================
+ Hits          991     1793     +802     
- Misses        154      169      +15     

Impacted Files	Coverage Δ
src/nd2/_util.py	86.66% <71.42%> (-3.90%)	:arrow_down:
src/nd2/nd2file.py	89.06% <80.58%> (-4.81%)	:arrow_down:
src/nd2/_legacy/_legacy.py	85.28% <81.81%> (ø)
src/nd2/_pysdk/_chunk_decode.py	84.44% <84.44%> (ø)
src/nd2/_binary.py	85.41% <85.41%> (ø)
src/nd2/_pysdk/_pysdk.py	91.18% <91.18%> (ø)
src/nd2/_clx_xml.py	92.10% <92.10%> (ø)
src/nd2/_pysdk/_parse.py	95.67% <95.67%> (ø)
src/nd2/structures.py	95.84% <96.90%> (-3.66%)	:arrow_down:
src/nd2/_clx_lite.py	97.02% <97.02%> (ø)
... and 3 more

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

[tlambert03]

some internal notes for myself... to keep track of what's going on here (copied here from https://github.com/tlambert03/nd2/pull/122#issue-1501961857)

• nd2._pysdk._pysdk.ND2Reader is the new pure class that takes the place of the cython sdk wrapper _sdk.latest.ND2Reader

• as a first pass, I'm re-implementing in python all of the metadata-reading methods from the sdk wrapper (those that returned structured objects). These include:

  class ND2Reader:
      @property
      def attributes(self) -> structures.Attributes: ...
      def metadata(self) -> structures.Metadata: ...
      def frame_metadata(self, seq_index: int) -> structures.FrameMetadata: ...
      def experiment(self) -> list[structures.ExpLoop]: ...
      def text_info(self) -> structures.TextInfo:

• they each generally begin with parsing a specific chunk using ND2Reader._decode_chunk:

  attributes -> b"ImageAttributesLV!"
  _raw_image_metadata -> b"ImageMetadataSeqLV|0!"
  text_info -> b"ImageTextInfoLV!"
  experiment -> b"ImageMetadataLV!"

• _decode_chunk loads the raw bytes for the key from the chunk map and then dispatches to one of two parsers based on the nd2 version, each of which is supposed to return a dict[str, Any]:

  • nd2 version < 3.0 uses nd2._xml.parse_variant_xml
  • nd2 version > 3.0 uses nd2._pysdk._decode.decode_CLxLiteVariant_json

• there are then a number of functions in the nd2._pysdk._parse module that take the parsed data for each metadata section and convert them to "structured" objects (i.e. data class objects from the nd2.structures module). It's possible that the return type of these functions might be relaxed to simply return another "cleaned up dict" that could be passed to these data structures. but for now the actual data structure is returned. The "main" functions include:

  • nd2._pysdk._parse.load_metadata
  • nd2._pysdk._parse.load_exp_loop
  • nd2._pysdk._parse.load_attributes
  • nd2._pysdk._parse.load_text_info
  • nd2._pysdk._parse.load_global_metadata
  • nd2._pysdk._parse.load_metadata
  • nd2._pysdk._parse.load_frame_metadata

• reading of image data itself is probably the simplest... it pulls the file offset for the requested frame index from the frame offsets (keys in the chunk map that start with b"ImageDataSeq|"), and then returns a numpy array from the memmaped file at the corresponding offset. If the frame is compressed, it uses zlib.decompress on the data

[tlambert03]

Everything is working and tests are passing. Going to merge this and do a little more cleanup in follow up PRs