DataLoad: Tracking thread for yahoo index data issues

[GoogleCodeExporter]

Getting zero for a bunch of yahoo indexes. ^W5HCR is the first one I'm focusing 
on.

Here's the URL:

http://query.yahooapis.com/v1/public/yql?q=select%20Symbol%2CLastTradePriceOnly%
2CLastTradeDate%2CLastTradeTime%20from%20yahoo.finance.quotes%20where%20symbol%2
0in%20%28%22^W5HCR%22%29%0A%09%09&env=http%3A%2F%2Fdatatables.org%2Falltables.en
v

Original issue reported on code.google.com by hastapas...@gmail.com on 7 Nov 2011 at 1:33

[GoogleCodeExporter]

I opened a data inaccuracy ticket here:
http://help.yahoo.com/l/us/yahoo/finance/forms_index.html

Original comment by hastapas...@gmail.com on 7 Nov 2011 at 1:40

[GoogleCodeExporter]

I'll handle this one.

Original comment by hastapas...@gmail.com on 1 Dec 2011 at 11:12

• Changed state: Started

[GoogleCodeExporter]

This appears to have been resolved by yahoo.

Original comment by hastapas...@gmail.com on 6 Dec 2011 at 11:41

• Changed state: Done

[GoogleCodeExporter]

Reopening this....

Original comment by hastapas...@gmail.com on 13 Dec 2011 at 4:41

[GoogleCodeExporter]

Two issues bumping into each other.

1) Created a workaround where a Yahoo URL is resubmitted if a quote with an 
earlier data is returned. There was an issue where sporadically the wrong 
date/data was returned and if you resubmitted, you would generally get back 
valid data on the next attempt.

2) When a stock is dropped from the S&P, yahoo returns the trade value from the 
last valid data which ends up being days ago. This results in the url being 
submitted over and over again because of item #1.

Options:
1) Get rid of the work around for #1. (Haven't been following how often it 
still occurs... should set up some kind of additional tracking).
2) Try to find out when stocks are dropped from the S&P ASAP. This requires 
more effort for monitoring and may still result in data being missed.
3) Try to come up with some kind of hybrid workaround that deals with both 
situations. Maybe count the number of resubmit attempts? Or is there some way 
to differentiate between the 2 issues programmatically?

Original comment by hastapas...@gmail.com on 13 Dec 2011 at 4:48

[GoogleCodeExporter]

Just witnessed the yahoo resubmit issue with task 24 so it is still a problem 
(not sure how often it happens with task 10).

Original comment by hastapas...@gmail.com on 13 Dec 2011 at 5:19

• Changed state: Started

[GoogleCodeExporter]

THIS IS NOW THE THREAD FOR YAHOO INDEX DATA INACCURACY ISSUES.

FOR YAHOO INDIVIDUAL STOCK QUOTE ISSUES SEE Issue 350.

Original comment by hastapas...@gmail.com on 16 Dec 2011 at 3:45

[GoogleCodeExporter]

Created a test group that only contains four tickers: 
(^IDDOW,^MNCEFT,^DJUBSSI,^DWCIDE)

Now running this every 15 minutes. May change to running it every hour.

Also eventually want to set up MultiTest again from another machine. Wondering 
if maybe yahoo is limiting requests.

Original comment by hastapas...@gmail.com on 16 Dec 2011 at 4:25

[GoogleCodeExporter]

Here's the csv url for those 4 tickers:

http://query.yahooapis.com/v1/public/yql?q=select%20Symbol%2CLastTradePriceOnly%
2CLastTradeDate%2CLastTradeTime%20from%20yahoo.finance.quotes%20where%20symbol%2
0in%20%28%22^IDDOW,^MNCEFT,^DJUBSSI,^DWCIDE%22%29%0A%09%09&env=http%3A%2F%2Fdata
tables.org%2Falltables.env

Original comment by hastapas...@gmail.com on 16 Dec 2011 at 5:06

[GoogleCodeExporter]

Doesn't appear that yahoo is limiting the requests. Get 0 values from laptop as 
well.

Original comment by hastapas...@gmail.com on 16 Dec 2011 at 5:07

[GoogleCodeExporter]

Getting the following error when I try to submit a yahoo ticket (haven't seen 
an email yet):

Oops! System Error:

We're sorry! There was a problem submitting your update to incident #.
Please reply to the email we sent to let us know if your issue was resolved.

Original comment by hastapas...@gmail.com on 21 Dec 2011 at 12:53

[GoogleCodeExporter]

These evening I started getting no quote data at all:

<query yahoo:count="0" yahoo:created="2011-12-21T03:46:48Z" 
yahoo:lang="en-US"><results/></query><!-- total: 133 --><!-- 
engine1.yql.sp2.yahoo.com -->

And this was causing the following to be displayed in the console:
2011-12-20 20:37:33,451 [Thread-894] INFO  [Task:24] DG55.30:Bad Yahoo Data, 
Resubmitting URL
2011-12-20 20:37:33,451 [Thread-894] INFO  [Task:24] DG55.66:Data length only 
241

Original comment by hastapas...@gmail.com on 21 Dec 2011 at 3:48

[GoogleCodeExporter]

Original comment by hastapas...@gmail.com on 8 Mar 2012 at 1:52

• Added labels: Priority-High
• Removed labels: Priority-Critical

[GoogleCodeExporter]

Original comment by hastapas...@gmail.com on 18 Oct 2012 at 7:15

• Added labels: Priority-Critical
• Removed labels: Priority-High