DataLoad: Issue with retrieving stock quotes from yahoo

[GoogleCodeExporter]

I'm getting all kinds of data issues with using the following link to retrieve 
stock quotes:
http://download.finance.yahoo.com/d/quotes.csv?f=sl1d1t1c1ohgv&e=.csv&s=MSFT,GE

One example where I'm seeing bad data: JCI (entity id 260). The value 36.030 
appears periodically through the time frame of 8/2/2011 - 8/4/2011.

I may need to use an alternative format for retrieving data from yahoo - like 
the YQL (yahoo query language).

Original issue reported on code.google.com by hastapas...@gmail.com on 4 Aug 2011 at 9:55

[GoogleCodeExporter]

I'm seeing this issue with several other tickers, like CPWR (entity id 134) and 
value 9.280. 

One option is to set up an independent mechanism to download the data and see 
if the same pattern occurs as with DataLoad. It's possible that the issue is 
with DataLoad (maybe threading related?) but I think this is low probability 
since it was working fine for a while and nothing has really changed recently.

Original comment by hastapas...@gmail.com on 4 Aug 2011 at 10:03

[GoogleCodeExporter]

I should be able to use MultiTest to observe the data independently.

Original comment by hastapas...@gmail.com on 4 Aug 2011 at 10:13

[GoogleCodeExporter]

Multitest is now running and monitoring the data.

Original comment by hastapas...@gmail.com on 5 Aug 2011 at 1:23

[GoogleCodeExporter]

Have not been able to reproduce this problem anywhere other than DataLoad in 
production. DataLoad in test doesn't even show the problem. 

I sent a message to yahoo support. I also opened up a thread on stackoverflow:
http://stackoverflow.com/questions/6958908/strange-issue-with-obtaining-stock-qu
otes-from-yahoo-finance

Another data issue was reported with yahoo finance on stack overflow:
http://stackoverflow.com/questions/6778138/yahoofinance-csv-file-error-in-price-
5-05544704e8-or-similar-numbers-and-date

Original comment by hastapas...@gmail.com on 5 Aug 2011 at 11:43

[GoogleCodeExporter]

I set up a task using YQL in test and I'm seeing other issues. It seems to have 
some kind of bandwidth limiter that prevents someone from making many requests 
in a short period of time.

2 other options to investigate: Do a bulk download of all tickers using either 
csv or yql.

Original comment by hastapas...@gmail.com on 5 Aug 2011 at 11:45

[GoogleCodeExporter]

I'm feeling even stronger that there is a bandwidth limiter when using yql to 
retrieve data by individual tickers. One option is to put in a pause between 
retrievals.

Issue I'm seeing with bulk retrieval is limit on length of URL (2083).

Original comment by hastapas...@gmail.com on 7 Aug 2011 at 9:22

[GoogleCodeExporter]

According to this thread, Google is unable to provide stock quotes through 
their finance API:

http://groups.google.com/group/google-finance-apis/browse_thread/thread/d5862f2d
06551c23/f7b8778bff263c55?lnk=gst&q=stock+quote#f7b8778bff263c55

Original comment by hastapas...@gmail.com on 7 Aug 2011 at 9:46

[GoogleCodeExporter]

Another option is to add a batch retrieval size to the entity loop.

Original comment by hastapas...@gmail.com on 7 Aug 2011 at 9:47

[GoogleCodeExporter]

Now seeing issue where date is correct but the time is off. Just in code to 
resubmit the url in that case as well.

Original comment by hastapas...@gmail.com on 23 Sep 2011 at 7:54

[GoogleCodeExporter]

Original comment by hastapas...@gmail.com on 25 Sep 2011 at 3:46

• Added labels: Priority-High
• Removed labels: Priority-Critical

[GoogleCodeExporter]

Issue 365 has been merged into this issue.

Original comment by hastapas...@gmail.com on 16 Dec 2011 at 3:42

[GoogleCodeExporter]

THIS WILL BE THE THREAD FOR MONITORING INDIVIDUAL STOCK QUOTE ISSUES AND THE 
ASSOCIATED WORKAROUNDS.

FOR DATA ACCURACY ISSUES WITH INDEXES SEE Issue 417.

Original comment by hastapas...@gmail.com on 16 Dec 2011 at 3:44

[GoogleCodeExporter]

Two issues bumping into each other.

1) Created a workaround where a Yahoo URL is resubmitted if a quote with an 
earlier data is returned. There was an issue where sporadically the wrong 
date/data was returned and if you resubmitted, you would generally get back 
valid data on the next attempt.

2) When a stock is dropped from the S&P, yahoo returns the trade value from the 
last valid data which ends up being days ago. This results in the url being 
submitted over and over again because of item #1.

Options:
1) Get rid of the work around for #1. (Haven't been following how often it 
still occurs... should set up some kind of additional tracking).
2) Try to find out when stocks are dropped from the S&P ASAP. This requires 
more effort for monitoring and may still result in data being missed.
3) Try to come up with some kind of hybrid workaround that deals with both 
situations. Maybe count the number of resubmit attempts? Or is there some way 
to differentiate between the 2 issues programmatically?

Original comment by hastapas...@gmail.com on 16 Dec 2011 at 3:45

[GoogleCodeExporter]

Still need to come up with a better way to handle yahoo data or find a 
completely different source. Will probably wait for eagle on this.

Original comment by hastapas...@gmail.com on 7 Oct 2012 at 12:36