search

storpipfugl / pykdtree

Fast kd-tree implementation in Python
GNU Lesser General Public License v3.0
206 stars 48 forks source link

pickle support #81

Open adizhol opened 1 year ago

adizhol commented 1 year ago

Please add pickle support.

Now there's an error:

  pickle.dump(
  File "stringsource", line 2, in pykdtree.kdtree.KDTree.__reduce_cython__
TypeError: no default __reduce__ due to non-trivial __cinit__
djhoese commented 1 year ago

What do you mean by "now"? This has always been this way I assume. If you have an idea for implementing a fix then feel free to make a pull request or at least propose it here.

djhoese commented 1 year ago

I realize now that by "now" you mean "currently it produces an error". Understood.

I'm curious what is your use case?

adizhol commented 1 year ago

hmmm... I have a class that contains a kdtree object. I want to save the object to disk for re-use later.

djhoese commented 1 year ago

We actually have a similar "need" in the Pytroll/Satpy community where we use dask and pykdtree KDTree. Our usage works fine in dask's threaded context when everything is on the same system and we don't need to serialize anything, but in the distributed context of multiple processes or multiple nodes when we have to serialize things we crash the processing because dask can't serialize the KDTree.

The hard part of serializing the KDTree is that the actual tree part of the structure is a low-level C struct:

https://github.com/storpipfugl/pykdtree/blob/7e2eab73ae688ebf8b2e22ded2b0a09f31e313a3/pykdtree/_kdtree_core.c.mako#L40-L58

If the algorithm was implemented as a contiguous block of memory then it'd be pretty simple to say "pretend this C pointer points to an array of numbers, let numpy serialize it", but it isn't that simple. I'm not sure the easiest way to go about this without rewriting large chunks of the C code. I suppose we could write a C function that traverses the C structs and converts them into a single array. I wonder how that effects look ups...

That said, @storpipfugl and @mraspaud, what do you think about performance and memory usage if the C code was rewritten to allocate a block of memory for all the nodes at once...if that's even possible? If it is algorithm-wise wouldn't that be way faster than the possibly non-contiguous pieces of memory for every Node struct?