search

JuliaData / Feather.jl

Read and write feather files in pure Julia
https://juliadata.github.io/Feather.jl/stable
Other
109 stars 27 forks source link

A DataFrame column of eltype "Date" changes to "Arrow.Datestamp" after reading it from a saved feather file #99

Open mwsohn opened 5 years ago

mwsohn commented 5 years ago

Has this been reported as an issue before? Is there a way to get around this issue?

ExpandingMan commented 5 years ago

You can just use convert on the Arrow types.

I do agree that the current behavior can be a little bit awkward, I've been planning on improving it at some point but unfortunately there's quite a bit of ambiguity in how this would be done.

So, what's going on here, is that Feather only officially supports saving dates and times in the arrow date and time formats. It would be trivial to instead use the Julia format, but this would make Julia feathers incompatible with other languages. Therefore, I decided to automatically convert only Date, Time, and DateTime (incidentally, see #98).

It's quite clear what the best default behavior would be for loading. Certainly Julia Dates types are infinitely more useful than the Arrow ones, but we also don't necessarily want to allocate a converted array.

I think my prefered behavior is to have a special type of Arrow array that wraps the raw ones and converts types on demand. I haven't implemented this yet, but, when I do, we'll probably make columns use these wrappers by default.

In the meantime you can of course convert the Arrow types.

(Also, our documentation really sucks regardless! We definitely need to improve it)