downloading historical Yahoo data gives ParseException

[GoogleCodeExporter]

From the e-mail list, mvalovski reports:

I installed truetrade from source, with eclipse and mysql and removed
the jdbc piping (it wouldn't work otherwise)
After that I added JNJ as a stock, went to "download historical data"
and selected yahoo daily data for 2006-2007
After a couple of seconds the "retrieving data" page disappears and
I'm back to the main truetrade page, with no data in the central part
of the window, and when I look in eclipse there's that exception in
the error log (lower part of the window)

-----
> > 00:05:421[ERROR]YahooHistDataBuilder - Could not build historical
> > price list for Instrument[symbol=JNJ,type=stock]
> > java.text.ParseException: Unparseable date: "22-May-07"
> >         at java.text.DateFormat.parse(Unknown Source)
> >         at
> > com.ats.providers.YahooHistDataBuilder.getEODData(
> > YahooHistDataBuilder.java:
> > 80)
> >         at
> > com.ats.engine.ImportDataManager.downloadHistData(ImportDataManager.java:
> > 66)
> >         at com.ats.client.wizards.DownloadHistDataWizard
> > $1.run(DownloadHistDataWizard.java:70)
> >         at org.eclipse.jface.operation.ModalContext
> > $ModalContextThread.run(ModalContext.java:113)
> > 00:05:421[DEBUG]YahooHistDataBuilder - Found: BarSeries
> > [span=daily,instrument=Instrument[symbol=JNJ,type=stock],bars=[]]
> > "

Original issue reported on code.google.com by tyrotra...@gmail.com on 24 May 2007 at 1:12

[GoogleCodeExporter]

Hi there,

I have changed YahooHistDataBuilder.java to do a download in CSV format 
(faster? more
reliable?) rather than through parsing HTML tables. For this I needed to 
include code
for a CSV parser which I took from http://opencsv.sourceforge.net/ (no idea 
whether
there is a standard one included in Java, i didn't find out about it).

As I know Java only from doing small hacks such as this and never used eclipse 
before
i prefer to post this code here. Please tell me if there is a better way of 
posting
my code; code you definately want to triple-check before including.

package com.ats.providers;

import java.text.DateFormat;
import java.text.DecimalFormat;
import java.text.NumberFormat;
import java.text.SimpleDateFormat;
import java.util.Calendar;
import java.util.Date;
import java.util.GregorianCalendar;

import org.apache.log4j.Logger;

import com.ats.utils.CSVReader;
import java.io.FileReader;
import java.io.IOException;
import java.io.StringWriter;
import java.util.List;

import com.ats.platform.Bar;
import com.ats.platform.BarSeries;
import com.ats.platform.BarType;
import com.ats.platform.Instrument;
import com.ats.platform.TimeSpan;
import com.meterware.httpunit.GetMethodWebRequest;
import com.meterware.httpunit.WebConversation;
import com.meterware.httpunit.WebLink;
import com.meterware.httpunit.WebRequest;
import com.meterware.httpunit.WebResponse;
import com.meterware.httpunit.WebTable;

import java.io.BufferedInputStream;
import java.io.BufferedReader;
import java.io.InputStreamReader;

public class YahooHistDataBuilder {
    private static final Logger logger = Logger.getLogger(YahooHistDataBuilder.class);

    private static final YahooHistDataBuilder instance = new YahooHistDataBuilder();

    private static final NumberFormat monthForm = new DecimalFormat("00");

    private static final DateFormat yahooDateForm = new SimpleDateFormat("yyyy-MM-dd");

    private int RowCount;

    WebConversation wc;

    private YahooHistDataBuilder() {
        wc = new WebConversation();
    }

    public static final YahooHistDataBuilder getInstance() {
        return instance;
    }

    public synchronized BarSeries getEODData(Instrument stock, int numDays) {
        if( ! stock.isStock() ) {
            // cannot gather non-stock data
            return null;
        }
        BarSeries series = new BarSeries(stock,BarType.time, TimeSpan.daily);
        try {

            // build request URL
            Calendar endCal = new GregorianCalendar();
            Calendar startCal = (Calendar)endCal.clone();
            startCal.add(Calendar.DATE, -1 * numDays);

            String urlString = "http://ichart.finance.yahoo.com/table.csv?s="
                + stock.getSymbol()
                + "&a=" + monthForm.format(startCal.get(Calendar.MONTH))
                + "&b=" + startCal.get(Calendar.DAY_OF_MONTH)
                + "&c=" + startCal.get(Calendar.YEAR)
                + "&d=" + monthForm.format(endCal.get(Calendar.MONTH))
                + "&e=" + endCal.get(Calendar.DAY_OF_MONTH)
                + "&f=" + endCal.get(Calendar.YEAR)
                + "&g=d" + "&ignore=.csv";

            System.out.println(urlString);

            WebRequest req = new GetMethodWebRequest( urlString );
            WebResponse resp = wc.getResponse(req);

            BufferedReader in = new BufferedReader(new InputStreamReader(resp.getInputStream()));
             CSVReader reader = new CSVReader(in);
                String [] nextLine;
                RowCount = 1;
                while ((nextLine = reader.readNext()) != null) {
                    if (RowCount != 1) {
                    Bar hist = new Bar(BarType.time, TimeSpan.daily);
                    hist.setBeginTime(yahooDateForm.parse((nextLine[0])));
                    hist.setEndTime(new Date(hist.getBeginTime().getTime() +
TimeSpan.daily.getSpanInMillis() - 1));
                    hist.setOpen(Double.parseDouble(nextLine[1]));
                    hist.setHigh(Double.parseDouble(nextLine[2]));
                    hist.setLow(Double.parseDouble(nextLine[3]));
                    hist.setClose(Double.parseDouble(nextLine[4]));
                    String vol = nextLine[5];
                    vol = vol.replaceAll(",","");
                    hist.setVolume(Integer.parseInt(vol));

                    series.addHistory(hist);

                    }
                    RowCount = RowCount + 1;
                }

        } catch( Exception e ) {
            logger.error("Could not build historical price list for " + stock, e);
        }
        logger.debug("Found: " + series);

        return series;
    }

}

Original comment by damiaan....@econ.kuleuven.be on 28 May 2007 at 2:47

[GoogleCodeExporter]

for the csv parser to work, you have to include the code from
http://opencsv.sourceforge.net/ in "com.ats.utils.CSVReader"

Original comment by damiaan....@econ.kuleuven.be on 28 May 2007 at 3:04

[GoogleCodeExporter]

I haven't tried the CSV reader, but I did add a check for bad data such as 
dividends
so the existing YahooHistDataBuilder is working.  I just checked it in.

Original comment by tyrotra...@gmail.com on 30 May 2007 at 10:56

• Changed state: Fixed