HistoricalData

[SeaRoth]

Did Yahoo remove the historicalData portion of their query?

[tghilardi]

I am about to try this api. Have you tried calling the yahoo api with https? They no longer work with http. Please let me know if it works so I can try it myself.

[SeaRoth]

-There is an example at the bottom of the readme with historicalData -I changed it to https and I'm still receiving the same error -I have had the same error happen with another php library as well so I think Yahoo canned the historical data or they changed its parameters / link

https://query.yahooapis.com/v1/public/yql?env=store%3A%2F%2Fdatatables.org%2Falltableswithkeys&format=json&q=select%20*%20from%20yahoo.finance.historicaldata%20where%20startDate=%272014-01-01%27%20and%20endDate=%272014-01-10%27%20and%20symbol=%27YHOO%27

[tghilardi]

Got it. No clue why. Yahoo probably changed other things as well. If you can work with csv, this still works http://download.finance.yahoo.com/d/quotes.csv?s=^GSPC&f=ld1

[SeaRoth]

I thank you for the link - that is a quote btw, and not historical data ;-)

[lesaff]

I am experiencing the same thing. Their front end says that it's an issue on their part but who knows.

The new historical data CSV download is

https://query1.finance.yahoo.com/v7/finance/download/AAPL?period1=1492859959&period2=1495451959&interval=1d&events=history&crumb=oi13r.ljwKk

And it requires authentication now. I still couldn't find any documentation on the new API (v7 I presume)

[lesaff]

Update: It looks like Yahoo FInance has discontinued Historical Data API. See official statement here https://forums.yahoo.net/t5/Yahoo-Finance-help/ichart-stopped-working-2-days-ago/m-p/251679#M3154

[loekvangool]

It was only a matter of time I suppose. Are there alternatives?

[lesaff]

Google Finance provides limited CSV download but they don't have Canadian exchanges.

If someone can translate Python code to PHP, perhaps this helps? Somehow this guy manages to decode the new v7 data feed crumbs authentication.

https://github.com/mementum/backtrader/commit/98e3ccc2b65ec09dd1a3a885b30ab1d0dcccf93f

[SeaRoth]

Here's how I've been getting around the errors:

1. Google Finance in CSV http://www.google.com/finance/historical?q=NASDAQ%3A' . $mSymbol . '&output=csv

2. Yahoo Finance in CSV https://query1.finance.yahoo.com/v7/finance/download/aapl?period1=1487685205&period2=1495457605&interval=1d&events=history&crumb=dU2hYSfAy9E

• With Yahoo Finance you have to do the http wrapper / POST otherwise you get errors

I hope this helps

[lesaff]

Thanks @SeaRoth Here's the CURL code if anyone interested

<?php

// Get cURL resource
$ch = curl_init();

// Set url
curl_setopt($ch, CURLOPT_URL, 'https://query1.finance.yahoo.com/v7/finance/download/AAPL?period1=1487685205&period2=1495457605&interval=1d&events=history&crumb');

// Set method
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'POST');

// Set options
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);

// Send the request & save response to $resp
$resp = curl_exec($ch);

if(!$resp) {
  die('Error: "' . curl_error($ch) . '" - Code: ' . curl_errno($ch));
} else {
  echo "Response HTTP Status Code : " . curl_getinfo($ch, CURLINFO_HTTP_CODE);
  echo "\nResponse HTTP Body : " . $resp;
}

// Close request to clear up some resources
curl_close($ch);

[Harrymon12]

I have moved to MarketXLS after this change, much more reliable data

[mchojrin]

@lesaff I tried your code but got an error for lack of authentication... is there something I'm missing?

Thanks

[lesaff]

@mchojrin, the auth comes in the form of session cookie. As soon as you make the POST request, Yahoo will send you back cookie code and that is your authentication. Once done, you don't need to do anything else until next session.

[mchojrin]

Just tried this:

<?php
$cookie_jar = tempnam('/tmp','cookie');

$ch = curl_init();
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_COOKIEJAR, $cookie_jar);

$d1 = new DateTime('-59 days');
$d2 = new DateTime('-60 days');

echo 'Cookie jar: '.$cookie_jar.PHP_EOL;

echo 'First curl'.PHP_EOL;
curl_setopt($ch, CURLOPT_URL, 'https://finance.yahoo.com/quote/'.$argv[1].'/history?p='.$argv[1]);
curl_setopt($ch, CURLOPT_COOKIEJAR, $cookie_jar);
curl_exec($ch);
echo 'Cookie: '.PHP_EOL.file_get_contents($cookie_jar).PHP_EOL;

echo 'Second cURL'.PHP_EOL;
curl_setopt($ch, CURLOPT_URL, 'https://query1.finance.yahoo.com/v7/finance/download/'.$argv[1].'?period1='.$d1->format('U').'&period2='.$d2->format('U').'&interval=1d&events=history&crumb');

// Set method
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'POST');

// Send the request & save response to $resp
$resp = curl_exec($ch);

if(!$resp) {
  die('Error: "' . curl_error($ch) . '" - Code: ' . curl_errno($ch));
} else {
  echo "Response HTTP Status Code : " . curl_getinfo($ch, CURLINFO_HTTP_CODE).PHP_EOL;
//  echo "\nResponse HTTP Body : " . $resp;
}

// Close request to clear up some resources
curl_close($ch);

echo 'Cookie: '.PHP_EOL.file_get_contents($cookie_jar).PHP_EOL;

But I'm still not getting it :(

I tried copying a cURL command with FireBug and got:

curl 'https://query1.finance.yahoo.com/v7/finance/download/GOOG?period1=1497893878&period2=1500485878&interval=1d&events=history&crumb=saRHVV4h4dU' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' -H 'Accept-Encoding: gzip, deflate, br' -H 'Accept-Language: en-US,en;q=0.5' -H 'Connection: keep-alive' -H 'Cookie: B=8emaer9cmv64t&b=3&s=6c;' -H 'Host: query1.finance.yahoo.com' -H 'Referer: https://finance.yahoo.com/quote/GOOG/history?p=GOOG' -H 'Upgrade-Insecure-Requests: 1' -H 'User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:54.0) Gecko/20100101 Firefox/54.0'

Which works fine (Had to add | gzip -d) to get the actual results but ok.

The only difference I can see is the value of the param crumb but I don't know how I should fill it...

[mchojrin]

Got my answer here.

[janwalther]

Any plans on integrating this into scheb/yahoo-finance-api library?

[scheb]

I tried out some things, this is working for me:

<?php
$cookie_jar = tempnam(__DIR__, 'cookie');

$ch = curl_init();
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_COOKIEJAR, $cookie_jar);

$d1 = new DateTime('-30days');
$d2 = new DateTime('today');

echo 'Cookie jar: '.$cookie_jar.PHP_EOL;

echo 'First curl'.PHP_EOL;

curl_setopt($ch, CURLOPT_VERBOSE, true);
curl_setopt($ch, CURLOPT_URL, 'https://finance.yahoo.com/lookup?s='.$argv[1]);
curl_setopt($ch, CURLOPT_COOKIEJAR, $cookie_jar);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);

$resp = curl_exec($ch);

echo 'Cookie: '.PHP_EOL.file_get_contents($cookie_jar).PHP_EOL;

preg_match('#CrumbStore":{"crumb":"(?<crumb>.+?)"}#', $resp, $match);
$crumb = json_decode('"'.$match['crumb'].'"'); // Crumb is a JSON encoded string
echo 'Crumb: ' . $crumb.PHP_EOL;

$url = 'https://query1.finance.yahoo.com/v7/finance/download/'.urlencode($argv[1]).'?period1='.$d1->format('U').'&period2='.$d2->format('U').'&interval=1d&events=history&crumb='.urlencode($crumb);
echo $url.PHP_EOL;

echo 'Second cURL'.PHP_EOL;
curl_setopt($ch, CURLOPT_URL, $url);

// Send the request & save response to $resp
$resp = curl_exec($ch);

if(!$resp) {
    die('Error: "' . curl_error($ch) . '" - Code: ' . curl_errno($ch));
} else {
    echo "Response HTTP Status Code : " . curl_getinfo($ch, CURLINFO_HTTP_CODE).PHP_EOL;
    echo "\nResponse HTTP Body : " . $resp;
}

// Close request to clear up some resources
curl_close($ch);

echo 'Cookie: '.PHP_EOL.file_get_contents($cookie_jar).PHP_EOL;

They put quite some work into protecting it. You first need to aquire a session by visiting the page, keep the session cookie, parse out a "crumb" value from the page source. In order to get the historical data, do another request, which passes that "crumb" value and the cookie. The "crumb" value seems to be just some random string, which is bound to the session. It doesn't change for different stocks.

What I have to say, I doubt that this endpoint is meant to be a public API. It's just some internal endpoint of the Yahoo Finance website and using it probably violates their TOS. Since it's not public it also could change any time.

[scheb]

New major release (2.0.0) of the library brings back support for all broken endpoints.