search

JuliaDataReaders / DataReaders.jl

A Julia library to get remote data via Requests.jl and get DataFrame (from DataFrames.jl) or TimeArray (from TimeSeries.jl)
MIT License
10 stars 7 forks source link

Cache works fine with one symbol... not with several symbols #12

Open femtotrader opened 8 years ago

femtotrader commented 8 years ago 
include("src/DataReaders.jl")

using DataReaders
using RequestsCache: CachedSession

session = CachedSession(cache_name="cache.jld", backend="jld", expire_after=Base.Dates.Day(1))
#session = nothing

dr = DataReader("yahoo", session=session)

dt_start = DateTime("2015-04-01")
dt_end = DateTime("2015-04-15")

symb = DataSymbol("MSFT")
data = get(dr, symb, dt_start, dt_end)

we get

RequestsCache.CachedSessionType("cache.jld","jld",1 day)
execute_local
Read key='5733033982959674190' from 'cache.jld'
Not expired
10x7 DataFrames.DataFrame
│ Row │ Date       │ Open  │ High  │ Low   │ Close │ Volume   │ Adj_Close │
┝━━━━━┿━━━━━━━━━━━━┿━━━━━━━┿━━━━━━━┿━━━━━━━┿━━━━━━━┿━━━━━━━━━━┿━━━━━━━━━━━┥
│ 1   │ 2015-04-01 │ 40.6  │ 40.76 │ 40.31 │ 40.72 │ 36865300 │ 39.6383   │
│ 2   │ 2015-04-02 │ 40.66 │ 40.74 │ 40.12 │ 40.29 │ 37487500 │ 39.2198   │
│ 3   │ 2015-04-06 │ 40.34 │ 41.78 │ 40.18 │ 41.55 │ 39223700 │ 40.4463   │
│ 4   │ 2015-04-07 │ 41.61 │ 41.91 │ 41.31 │ 41.53 │ 28809400 │ 40.4268   │
│ 5   │ 2015-04-08 │ 41.46 │ 41.69 │ 41.04 │ 41.42 │ 24753400 │ 40.3197   │
│ 6   │ 2015-04-09 │ 41.25 │ 41.62 │ 41.25 │ 41.48 │ 25723900 │ 40.3781   │
│ 7   │ 2015-04-10 │ 41.63 │ 41.95 │ 41.41 │ 41.72 │ 28022000 │ 40.6118   │
│ 8   │ 2015-04-13 │ 41.4  │ 42.06 │ 41.39 │ 41.76 │ 30276700 │ 40.6507   │
│ 9   │ 2015-04-14 │ 41.8  │ 42.03 │ 41.39 │ 41.65 │ 24244400 │ 40.5436   │
│ 10  │ 2015-04-15 │ 41.76 │ 42.46 │ 41.68 │ 42.26 │ 27343600 │ 41.1374   │

disconnect internet (wireless) connection

data = get(dr, symb, dt_start, dt_end)

we get 

RequestsCache.CachedSessionType("cache.jld","jld",1 day)
execute_local
Read key='5733033982959674190' from 'cache.jld'
Not expired
10x7 DataFrames.DataFrame
│ Row │ Date       │ Open  │ High  │ Low   │ Close │ Volume   │ Adj_Close │
┝━━━━━┿━━━━━━━━━━━━┿━━━━━━━┿━━━━━━━┿━━━━━━━┿━━━━━━━┿━━━━━━━━━━┿━━━━━━━━━━━┥
│ 1   │ 2015-04-01 │ 40.6  │ 40.76 │ 40.31 │ 40.72 │ 36865300 │ 39.6383   │
│ 2   │ 2015-04-02 │ 40.66 │ 40.74 │ 40.12 │ 40.29 │ 37487500 │ 39.2198   │
│ 3   │ 2015-04-06 │ 40.34 │ 41.78 │ 40.18 │ 41.55 │ 39223700 │ 40.4463   │
│ 4   │ 2015-04-07 │ 41.61 │ 41.91 │ 41.31 │ 41.53 │ 28809400 │ 40.4268   │
│ 5   │ 2015-04-08 │ 41.46 │ 41.69 │ 41.04 │ 41.42 │ 24753400 │ 40.3197   │
│ 6   │ 2015-04-09 │ 41.25 │ 41.62 │ 41.25 │ 41.48 │ 25723900 │ 40.3781   │
│ 7   │ 2015-04-10 │ 41.63 │ 41.95 │ 41.41 │ 41.72 │ 28022000 │ 40.6118   │
│ 8   │ 2015-04-13 │ 41.4  │ 42.06 │ 41.39 │ 41.76 │ 30276700 │ 40.6507   │
│ 9   │ 2015-04-14 │ 41.8  │ 42.03 │ 41.39 │ 41.65 │ 24244400 │ 40.5436   │
│ 10  │ 2015-04-15 │ 41.76 │ 42.46 │ 41.68 │ 42.26 │ 27343600 │ 41.1374   │

but it doesn't work with several symbols

include("src/DataReaders.jl")

using DataReaders
using RequestsCache: CachedSession

session = CachedSession(cache_name="cache.jld", backend="jld", expire_after=Base.Dates.Day(1))

dr = DataReader("yahoo", session=session)

dt_start = DateTime("2015-04-01")
dt_end = DateTime("2015-04-15")

symbols = DataSymbols(["IBM", "MSFT"])
data = get(dr, symbols, dt_start, dt_end)

it shows

RequestsCache.CachedSessionType("cache.jld","jld",1 day)
execute_local
Read key='4926608604750823822' from 'cache.jld'
HDF5-DIAG: Error detected in HDF5 (1.8.15-patch1) thread 0:
  #000: H5O.c line 246 in H5Oopen(): unable to open object
    major: Symbol table
    minor: Can't open object
  #001: H5O.c line 1357 in H5O_open_name(): object not found
    major: Symbol table
    minor: Object not found
  #002: H5Gloc.c line 430 in H5G_loc_find(): can't find object
    major: Symbol table
    minor: Object not found
  #003: H5Gtraverse.c line 861 in H5G_traverse(): internal path traversal failed
    major: Symbol table
    minor: Object not found
  #004: H5Gtraverse.c line 641 in H5G_traverse_real(): traversal operator failed
    major: Symbol table
    minor: Callback failed
  #005: H5Gloc.c line 385 in H5G_loc_find_cb(): object '4926608604750823822' doesn't exist
    major: Symbol table
    minor: Object not found
LoadError RequestsCache.CachedSessionType("cache.jld","jld",1 day)
execute_remote get http://ichart.finance.yahoo.com/table.csv Any[(:query,Dict{AbstractString,AbstractString}("f"=>"2015","g"=>"d","c"=>"2015","e"=>"15","ignore"=>".csv","b"=>"1","s"=>"IBM","a"=>"3","d"=>"3")),(:timeout,30.0)]
Write RequestsCache.CachedResponse(2016-05-06T10:30:54.538,Response(200 OK, 15 headers, 766 bytes in body)) with key='4926608604750823822' to 'cache.jld'
Write to RequestsCache.CachedSessionType("cache.jld","jld",1 day)
RequestsCache.CachedSessionType("cache.jld","jld",1 day)
execute_local
Read key='5733033982959674190' from 'cache.jld'
HDF5-DIAG: Error detected in HDF5 (1.8.15-patch1) thread 0:
  #000: H5O.c line 246 in H5Oopen(): unable to open object
    major: Symbol table
    minor: Can't open object
  #001: H5O.c line 1357 in H5O_open_name(): object not found
    major: Symbol table
    minor: Object not found
  #002: H5Gloc.c line 430 in H5G_loc_find(): can't find object
    major: Symbol table
    minor: Object not found
  #003: H5Gtraverse.c line 861 in H5G_traverse(): internal path traversal failed
    major: Symbol table
    minor: Object not found
  #004: H5Gtraverse.c line 641 in H5G_traverse_real(): traversal operator failed
    major: Symbol table
    minor: Callback failed
  #005: H5Gloc.c line 385 in H5G_loc_find_cb(): object '5733033982959674190' doesn't exist
    major: Symbol table
    minor: Object not found
LoadError RequestsCache.CachedSessionType("cache.jld","jld",1 day)
execute_remote get http://ichart.finance.yahoo.com/table.csv Any[(:query,Dict{AbstractString,AbstractString}("f"=>"2015","g"=>"d","c"=>"2015","e"=>"15","ignore"=>".csv","b"=>"1","s"=>"MSFT","a"=>"3","d"=>"3")),(:timeout,30.0)]
Write RequestsCache.CachedResponse(2016-05-06T10:31:01.176,Response(200 OK, 15 headers, 705 bytes in body)) with key='5733033982959674190' to 'cache.jld'
Write to RequestsCache.CachedSessionType("cache.jld","jld",1 day)
DataStructures.OrderedDict{DataReaders.DataSymbol,DataFrames.DataFrame} with 2 entries:
  DataReaders.DataSymbol("IBM")  => 10x7 DataFrames.DataFrame…
  DataReaders.DataSymbol("MSFT") => 10x7 DataFrames.DataFrame…

disconnect internet

julia> data = get(dr, symbols, dt_start, dt_end)
RequestsCache.CachedSessionType("cache.jld","jld",1 day)
execute_local
Read key='4926608604750823822' from 'cache.jld'
HDF5-DIAG: Error detected in HDF5 (1.8.15-patch1) thread 0:
  #000: H5O.c line 246 in H5Oopen(): unable to open object
    major: Symbol table
    minor: Can't open object
  #001: H5O.c line 1357 in H5O_open_name(): object not found
    major: Symbol table
    minor: Object not found
  #002: H5Gloc.c line 430 in H5G_loc_find(): can't find object
    major: Symbol table
    minor: Object not found
  #003: H5Gtraverse.c line 861 in H5G_traverse(): internal path traversal failed
    major: Symbol table
    minor: Object not found
  #004: H5Gtraverse.c line 641 in H5G_traverse_real(): traversal operator failed
    major: Symbol table
    minor: Callback failed
  #005: H5Gloc.c line 385 in H5G_loc_find_cb(): object '4926608604750823822' doesn't exist
    major: Symbol table
    minor: Object not found
LoadError RequestsCache.CachedSessionType("cache.jld","jld",1 day)
execute_remote get http://ichart.finance.yahoo.com/table.csv Any[(:query,Dict{AbstractString,AbstractString}("f"=>"2015","g"=>"d","c"=>"2015","e"=>"15","ignore"=>".csv","b"=>"1","s"=>"IBM","a"=>"3","d"=>"3")),(:timeout,30.0)]
ERROR: getaddrinfo callback: unknown node or service (EAI_NONAME)
 in getaddrinfo at /Applications/Julia-0.4.3.app/Contents/Resources/julia/lib/julia/sys.dylib
 in open_stream at /Users/femto/.julia/v0.4/Requests/src/streaming.jl:184
 in do_stream_request at /Users/femto/.julia/v0.4/Requests/src/Requests.jl:346
 in do_request at /Users/femto/.julia/v0.4/Requests/src/Requests.jl:286
 in execute_remote at /Users/femto/.julia/v0.4/RequestsCache/src/RequestsCache.jl:122
 in execute_local at /Users/femto/.julia/v0.4/RequestsCache/src/RequestsCache.jl:141
 in execute at /Users/femto/.julia/v0.4/RequestsCache/src/RequestsCache.jl:151
 in get_response at /Users/femto/.julia/v0.4/DataReaders/src/DataReaders.jl:98
 in get at /Users/femto/.julia/v0.4/DataReaders/src/yahoo/daily.jl:15
 in get_several_symbols_to_ordereddict at /Users/femto/.julia/v0.4/DataReaders/src/DataReaders.jl:84
 in get at /Users/femto/.julia/v0.4/DataReaders/src/yahoo/daily.jl:24