How to deal with C pointers

[antoine-levitt]

FFT plans for instance have C pointers that currently can't be saved (JLD2 gives an error). If I understand correctly, the correct option is to define a custom serializer, but if I have to do that in my package for FFTW structs that's type piracy, right? Is there a good solution to this? Should there be a SerializationBase package that FFTW could depend on and that would define this API?

[JonasIsensee]

Hi @antoine-levitt ,

I'm not quite sure what you are trying to achieve. Even if you could serialize a pointer it would be meaningless when loading in a new session, right?

Apart from that, yes, a custom serializer seems reasonable. I'm not sure whether that qualifies as type piracy since you would have to define the SerializedFFTPlan struct which thus you would own.

Note however that in the latest release custom serialization is not yet supported. You are welcome to play around with #245 which already works but needs additional testing and I need to figure out the performance penalties of the changes involved.

[antoine-levitt]

The struct that contains it has the information needed, so you can just write a serializer that would ignore the pointer, and a deserializer that would just create a new plan.

My actual use case is that I have a struct A I want to serialize with a lot of fields, among them FFT plans. If I write @save a with a of type A, it'll call something like serialize(p::FFTPlan), no? In which case I would need to overload that which is piracy. The alternative of course is to write a serializer for A which would take care of that.

[JonasIsensee]

That is not wrong but also not entirely correct. JLD2 custom serialization works by converting structs prior to serialization. Here's an example

struct FFTPlan
    pointer::Ptr
    data::Float64
end

struct SerializedFFTPlan
    data::Float64
end
Base.convert(::Type{SerializedFFTPlan}, x::FFTPlan) = SerializedFFTPlan(x.data)
Base.convert(::Type{FFTPlan}, x::SerializedFFTPlan) = FFTPlan(somecomputation(x.data), x.data)

JLD2.writeas(::Type{FFTPlan}) = SerializedFFTPlan

now any FFTPlan will be converted to a SerializedFFTPlan prior to storing and then converted back upon loading. This also works if the FFTPlan is only a field of some other struct. (Again only works with #245)

[antoine-levitt]

OK but the piracy is still there at JLD2.writeas(::Type{FFTPlan}) = SerializedFFTPlan. I think that's a pretty common problem in the julia ecosystem, and the only way to solve that without having FFTW depend on JLD2 is some kind of common lightweight SerializationBase package. That's premature though.

[JonasIsensee]

Fair enough. I'd say that might be acceptable, though. If JLD2 custom serialization became common place we'd have to rethink that but in the medium term future I'd say that's perfectly fine.

[JonasIsensee]

v0.3.0 with custom serialization is out now.

I'll agree, that is a very common and unavoidable problem. I suppose that you're right. If you wanted to do this properly, you would need some FFTWJLD2Serialization package that defines all this.

Until then, my suggestions from above are probably the best bet.