JuliaData / Feather.jl

Read and write feather files in pure Julia
https://juliadata.github.io/Feather.jl/stable
Other
109 stars 27 forks source link

use_mmap default values #61

Closed Nosferican closed 6 years ago

Nosferican commented 6 years ago

Currently the default value for use_mmap is false for Windows and true for all other OS. In my limited understanding and experience (I am using a Macbook), MacOS does not present the capabilities (at least by default) and might be more common in Linux. Should the logic be false for all systems except Linux rather?

quinnj commented 6 years ago

What makes you think it doesn't work on mac? I use a mac and developed this package using it; mmap is pretty standard unix, which means macs are fully compatible.

Nosferican commented 6 years ago

This is more my personal understanding, but can you send me a resource so I can look up on how enable it on my Macbook? Mine is from last year and doesn't seem to be working with Feather.read.

Also, in the code it seems that the OS is used for the default value in Source, but it has a hardcoded true in the keyword argument for read. Is this intentional or could the default be informed by the OS as well?

quinnj commented 6 years ago

It'd be helpful if you posted the actual errors you're seeing; I imagine they're not related to mmap. There's nothing to "enable", mmap is builtin to any unix-based OS, including OSX. If you could post the code you're running and the errors you're seeing, I can help further diagnose what's going on.

Nosferican commented 6 years ago

I assumed it was related to my machine and not the code. If my system is suppose to have it, then it might be worth checking out.

using Feather
using DataFrames
df = Feather.read(joinpath(Pkg.dir("Feather"), "test/newdata/iris.feather"),
    use_mmap = true)

and the error message is

type Void has no field mmap

Stacktrace:
 [1] #Source#7(::Bool, ::Bool, ::Bool, ::Type, ::String) at /Users/jbsc/.julia/v0.7/Feather/src/Feather.jl:104
 [2] (::getfield(Core, Symbol("#kw#Type")))(::Array{Any,1}, ::Type{Feather.Source}, ::String) at ./<missing>:0
 [3] #read#27(::Bool, ::Bool, ::Bool, ::Bool, ::Dict{Int64,Function}, ::Function, ::String, ::Type{DataFrames.DataFrame}) at /Users/jbsc/.julia/v0.7/Feather/src/Feather.jl:278
 [4] (::getfield(Feather, Symbol("#kw##read")))(::Array{Any,1}, ::typeof(Feather.read), ::String, ::Type{DataFrames.DataFrame}) at ./<missing>:0 (repeats 2 times)

Using,

Julia                         0.7.0-DEV.2098
Feather                       0.2.5+
CategoricalArrays             0.2.3
DataFrames                    0.10.1+
DataStreams                   0.2.1
FlatBuffers                   0.3.1
Nulls                         0.1.2+
WeakRefStrings                0.3.3+

Setting use_mmap = false works perfectly.

davidanthoff commented 6 years ago

I guess the real question is why it is disabled on Windows :) Windows does support memory mapped files, right?