search

dysonance / Temporal.jl

Time series implementation for the Julia language focused on efficiency and flexibility
Other
100 stars 25 forks source link

Error using Yahoo connector #6

Closed Iscalc closed 7 years ago

Iscalc commented 7 years ago

First, I would like to thank you very much for this package.

Today started getting the following error message. After

using Temporal a = yahoo("AAPL", from="2010-06-09", thru=string(Dates.today()), freq='w')

I get

connect: connection timed out (ETIMEDOUT) in yieldto(::Task, ::ANY) at .\event.jl:136 in wait() at .\event.jl:169 in wait(::Condition) at .\event.jl:27 in stream_wait(::TCPSocket, ::Condition, ::Vararg{Condition,N}) at .\stream.jl:44 in wait_connected(::TCPSocket) at .\stream.jl:265 in connect at .\stream.jl:960 [inlined] in connect(::IPv4, ::Int64) at .\socket.jl:677 in open_stream(::HttpCommon.Request, ::MbedTLS.SSLConfig, ::Float64, ::Nullable{URIParser.URI}, ::Nullable{URIParser.URI}) at C:\JuliaPro-0.5.0.5\pkgs-0.5.0.5\v0.5\Requests\src\streaming.jl:186 in #do_stream_request#23(::Dict{AbstractString,AbstractString}, ::Void, ::Void, ::Void, ::Array{Requests.FileParam,1}, ::Void, ::Dict{Any,Any}, ::Bool, ::Int64, ::Array{HttpCommon.Response,1}, ::MbedTLS.SSLConfig, ::Bool, ::Bool, ::Bool, ::Nullable{URIParser.URI}, ::Nullable{URIParser.URI}, ::Requests.#do_stream_request, ::URIParser.URI, ::String) at C:\JuliaPro-0.5.0.5\pkgs-0.5.0.5\v0.5\Requests\src\Requests.jl:381 in do_stream_request(::URIParser.URI, ::String) at C:\JuliaPro-0.5.0.5\pkgs-0.5.0.5\v0.5\Requests\src\Requests.jl:344 in #do_request#22(::Array{Any,1}, ::Function, ::URIParser.URI, ::String) at C:\JuliaPro-0.5.0.5\pkgs-0.5.0.5\v0.5\Requests\src\Requests.jl:311 in #get#29(::Array{Any,1}, ::Function, ::URIParser.URI) at C:\JuliaPro-0.5.0.5\pkgs-0.5.0.5\v0.5\Requests\src\Requests.jl:444 in #get#28(::Array{Any,1}, ::Function, ::String) at C:\JuliaPro-0.5.0.5\pkgs-0.5.0.5\v0.5\Requests\src\Requests.jl:443 in #yahoo#64(::String, ::String, ::Char, ::Function, ::String) at C:\JuliaPro-0.5.0.5\pkgs-0.5.0.5\v0.5\Temporal\src\io.jl:291 in (::Temporal.#kw##yahoo)(::Array{Any,1}, ::Temporal.#yahoo, ::String) at .\<missing>:0

haavardhvarnes commented 7 years ago

I have the same problem. Copying the yahoo function and made some changes and it works. But parsing csv sometimes fails. Hope more can take a look at this problem. My adjusted functions below:

function getyahoocrumb() ustr = "https://ca.finance.yahoo.com/quote/^GSPC/history?p=^GSPC"; response = Requests.get(ustr); str = readstring(response); m = match(r"\"user\":{\"crumb\":\"(.*?)\"", str) ; crumb = m[1] mcookies = cookies(response) return crumb, mcookies end

function yahoo2(symb::String; from::String="1900-01-01", thru::String=string(Dates.today()), freq::Char='d', crumb = getyahoocrumb()) @assert freq in ['d','w','m','v'] "Argument freq must be in ['d','w','m','v']" @assert from[5] == '-' && from[8] == '-' "Argument from has invalid date format." @assert thru[5] == '-' && thru[8] == '-' "Argument thru has invalid date format." dtFrom = Dates.DateTime(from) dtThru = Dates.DateTime(thru)

# symb = "XOM"
# dtFrom = Dates.DateTime("2016-01-01")
# dtThru = Dates.DateTime("2017-05-18")
per1 = Int(floor(Dates.datetime2unix(dtFrom)))
per2 = Int(floor(Dates.datetime2unix(dtThru)))

crumb = "Df1WvGdhjEg"
crumb, mcookies = getyahoocrumb()
qrystr= string("https://query1.finance.yahoo.com/v7/finance/download/",symb,"?period1=", per1, "&period2=", per2,"&interval=1d&events=history&crumb=", crumb)
# println(qrystr)

# indata = Temporal.csvresp(get(qrystr))
indata = Temporal.csvresp(get(qrystr, cookies = mcookies))

sleep(0.8)
return ts(indata[1], indata[2], indata[3][2:end])

end

AntonioSaragga commented 7 years ago

It seems that yahoo is discontinuing its service.

dysonance commented 7 years ago

@AntonioSaragga Could you provide a source suggesting this is the case? Best I've found is an answer on this forum. I know R's quantmod package has been experiencing this issue lately, but not sure if it's resolved (and given that Joshua Ulrich is a very proactive package maintainer, that would tell me this is a non-trivial issue).

dysonance commented 7 years ago

@haavardhvarnes Thanks for taking a crack at getting it to work again, that's much appreciated! I'll try it out see if it's possible to get it worked out. If Yahoo hasn't discontinued the service and we can get this solved I'll try to merge that into the package ASAP.

dysonance commented 7 years ago

Seeing some workarounds for packages in other languages out there. This project implements a "temporary fix" for the function in the Pandas package in Python that downloads data from the Yahoo Finance API. See this Stack Overflow thread for some insights around the work done on R's quantmod package to solve the issue. If anyone has experience with either of these please provide thoughts/feedback on how they've been working.

dysonance commented 7 years ago

Alright all, with the great help of @haavardhvarnes's initial workaround above, I have managed to get a fix put in place for the Yahoo Finance data downloading functionality. There are some changes to the arguments required to fit the new URL formatting needs:

I'm about to send through an update / PR to Julia's METADATA repo, so hopefully this fix should be available to the wider community with a Pkg.update() soon. Until then, feel free to pull the changes from git to your local package directory and you should be all set.

Thanks again everyone for the input on this issue and for helping out the legwork on a workaround. Hopefully this issue will be solved for the time being. However, it did also raise an interesting note in my mind that it would be good to have more options for market data sources going forward (Google, etc.). If anyone would like to contribute with any functionality for sources other than Yahoo and Quandl, I'd be happy to work with you on it. Cheers.