search

JuliaData / Feather.jl

Read and write feather files in pure Julia
https://juliadata.github.io/Feather.jl/stable
Other
109 stars 27 forks source link

Support for units #102

Closed cstjean closed 5 years ago

cstjean commented 5 years ago

I have a DataFrame with some Unitful units:

3×2 DataFrame
│ Row │ x     │ y         │
│     │ Int64 │ Quantity… │
├─────┼───────┼───────────┤
│ 1   │ 1     │ 3 m       │
│ 2   │ 2     │ 4 m       │
│ 3   │ 3     │ 5 m       │

Is there any way to store the units in feather format? I seem to remember that one of Feather's goals was to attach metadata to the columns, did it go anywhere?

ExpandingMan commented 5 years ago

I'm pretty sure that's not what Wes means by metadata, I think he's strictly refering to metadata that tells you about the binary formats of the data. Currently there is no way of storing units metadata according to the Feather standard. In principle it would be easy enough to store the underlying binary data of the Julia Unitful types (though I think we may have broken this with some of the more recent updates), but that's not something we wanted to encourage because we didn't want people storing lots of Julia-specific Feather files and it still wouldn't help you with your problem since you'd have to use metadata outside of Feather to convert the values.

My suggestion for you right now is that you somehow standardize your column names such that you can uniquely determine the units from those. The next alternative would be JLD2.

(Regardless, this is a feature that I too would really like to have... I have some ideas, but I'm far from the point where I want to stick my foot in my mouth and say something!)

I'm going to close this issue, as it can't go anywhere unless they update the standard.

cstjean commented 5 years ago

Wes' blog post is not very clear, but:

By "metadata-free", I mean that in storing data in these formats, you lose type information (for example: factor levels) that may be impossible to recover.

Factor levels are metadata similar to units, so I think that's what he was referring to.

(Regardless, this is a feature that I too would really like to have... I have some ideas, but I'm far from the point where I want to stick my foot in my mouth and say something!)

Looking forward to it!