Closed blais closed 3 years ago
blais
commented
7 years ago Original comment by Zhuoyun Wei (Bitbucket: wzyboy, GitHub: wzyboy).
I cannot find any docs on Google Finance API. The original link in the docstring is now broken as well. The top Google search result is a StackOverflow post that says the API is deprecated since 2011.
Should we deprecate the Google Finance driver or make some workaround fixes for it?
blais
commented
7 years ago Original comment by Zhuoyun Wei (Bitbucket: wzyboy, GitHub: wzyboy).
Related: the Yahoo Finance driver is currently broken. The domain name is unresovalbe: https://bitbucket.org/blais/beancount/src/621cec5ed38bcd128a3502a3b5c367f283deffe2/beancount/prices/sources/yahoo.py?at=default&fileviewer=file-view-default#yahoo.py-203
It seems they took down the service as well :-(
blais
commented
7 years ago Original comment by Zhuoyun Wei (Bitbucket: wzyboy, GitHub: wzyboy).
Here is a quick-and-dirty script to fetch prices from Google and print them as "price" directives. It does not fit into the existing beancount.prices framework but if anyone else needs, they can use it to finish the job.
#!/usr/bin/env python
'''A quick-and-dirty script to fetch prices from Google Finance API.'''
import sys
import csv
import argparse
import requests
from dateutil.parser import parse as parse_datetime
from beancount.core import data
from beancount.core.number import D
from beancount.core.amount import Amount
from beancount.parser.printer import print_entries
from beancount.loader import load_file
from beancount.ops import holdings
class Price:
def fetch_all(self, symbols=[]):
entries = []
for symbol in symbols:
prices = self.get_prices_from_google(symbol)
_entries = self.convert_prices(prices)
entries.extend(_entries)
entries.sort()
return entries
def get_prices_from_google(self, symbol):
# Get prices of a stock for the past year
print('Fetching prices for {} ...'.format(symbol), file=sys.stderr)
params = {
'q': symbol,
'output': 'csv',
}
url = 'http://finance.google.com/finance/historical'
try:
resp = requests.get(url, params=params)
except Exception as e:
print('Error fetching prices for {}:\n{}'.format(symbol, e), file=sys.stderr)
return []
if not resp.ok:
print('Error fetching prices for {}:\n{}\n{}'.format(symbol, resp, resp.text), file=sys.stderr)
return []
csv_data = resp.text
csv_lines = csv_data.splitlines()
# We only care about close price
csv_reader = csv.DictReader(csv_lines)
prices = [
(symbol, parse_datetime(row['Date']).date(), row['Close'])
for row in csv_reader
]
return prices
def convert_prices(self, prices):
'''Convert prices to Beancount "Price" entries'''
meta = data.new_metadata('<price>', 0)
entries = [
data.Price(
meta=meta,
date=price[1],
currency=price[0],
amount=Amount(D(price[2]), 'USD')
)
for price in prices
]
return entries
def extract_symbols(self, filename):
entries, _, _ = load_file(filename)
symbols = set(
h.currency for h in holdings.get_final_holdings(entries)
if h.account.startswith('Assets:') and 'Positions' in h.account
)
return symbols
def main():
argparser = argparse.ArgumentParser()
symbol_source = argparser.add_mutually_exclusive_group(required=True)
symbol_source.add_argument('--from-file', help='read symbols from Beancount file')
symbol_source.add_argument('--symbols', metavar='SYMBOL', nargs='+', help='read symbols from command line')
argparser.add_argument('-l', '--list', action='store_true', help='only list symbols to fetch prices for')
args = argparser.parse_args()
price = Price()
if args.from_file:
symbols = price.extract_symbols(args.from_file)
elif args.symbols:
symbols = args.symbols
if not args.list:
entries = price.fetch_all(symbols)
print_entries(entries)
else:
print(symbols)
if __name__ == '__main__':
main()Original comment by Zhuoyun Wei (Bitbucket: wzyboy, GitHub: wzyboy).
I started to get 403 errors about one week ago. I have updated the quick script to IEX source instead:
#!/usr/bin/env python
'''A quick-and-dirty script to fetch prices from (deprecated) Google Finance
API. Because all price sources in bean-price are broken and bean-price itself
is too complicated.'''
import sys
import csv
import argparse
import requests
from dateutil.parser import parse as parse_datetime
from beancount.core import data
from beancount.core.number import D
from beancount.core.amount import Amount
from beancount.parser import printer
from beancount.loader import load_file
from beancount.ops import holdings
def get_prices_from_google(symbol):
# Get prices of a stock for the past year
print('Fetching prices for {} ...'.format(symbol), file=sys.stderr)
params = {
'q': symbol,
'output': 'csv',
}
url = 'http://finance.google.com/finance/historical'
try:
resp = requests.get(url, params=params)
except Exception as e:
print('Error fetching prices for {}:\n{}'.format(symbol, e), file=sys.stderr)
return []
if not resp.ok:
print('Error fetching prices for {}:\n{}\n{}'.format(symbol, resp, resp.text), file=sys.stderr)
return []
csv_data = resp.text
csv_lines = csv_data.splitlines()
# We only care about close price
csv_reader = csv.DictReader(csv_lines)
prices = [
(symbol, parse_datetime(row['Date']).date(), row['Close'])
for row in csv_reader
]
return prices
def get_prices_from_iex(symbol, date_range='ytd'):
print('Fetching prices for {} ...'.format(symbol), file=sys.stderr)
url = f'https://api.iextrading.com/1.0/stock/{symbol}/chart/{date_range}'
json_data = requests.get(url).json()
prices = [
(symbol, parse_datetime(item['date']).date(), item['close'])
for item in json_data
]
return prices
def convert_prices(prices):
'''Convert prices to Beancount "Price" entries'''
meta = data.new_metadata('<price>', 0)
entries = [
data.Price(
meta=meta,
date=price[1],
currency=price[0],
amount=Amount(D('{:.2f}'.format(price[2])), 'USD')
)
for price in prices
]
return entries
def extract_symbols(filename):
entries, _, _ = load_file(filename)
symbols = set(
h.currency for h in holdings.get_final_holdings(entries)
if h.account.startswith('Assets:') and 'Positions' in h.account
)
return symbols
def fetch_all(fetch_func, symbols):
symbols = symbols or []
entries = []
for symbol in symbols:
prices = fetch_func(symbol)
_entries = convert_prices(prices)
entries.extend(_entries)
entries.sort()
return entries
def main():
argparser = argparse.ArgumentParser()
symbol_source = argparser.add_mutually_exclusive_group(required=True)
symbol_source.add_argument('-f', '--from-file', help='read symbols from Beancount file')
symbol_source.add_argument('-s', '--symbols', metavar='SYMBOL', nargs='+', help='read symbols from command line')
argparser.add_argument('-l', '--list', action='store_true', help='only list symbols to fetch prices for')
args = argparser.parse_args()
if args.from_file:
symbols = extract_symbols(args.from_file)
elif args.symbols:
symbols = args.symbols
if not args.list:
entries = fetch_all(get_prices_from_iex, symbols)
printer.print_entries(entries)
else:
print(symbols)
if __name__ == '__main__':
main()Original comment by Martin Blais (Bitbucket: blais, GitHub: blais).
I confirmed by accessing in an non-automated way that it's really gone indeed. I'm deleting it right now.
Earlier today I discovered the IEX feed and I filed a ticket (https://bitbucket.org/blais/beancount/issues/267/implement-price-source-for-iex) to implement one. It'll show up soon. I'll likely also merge the Quandl pull request, that looks like a nice feed as well.
Original report by Zhuoyun Wei (Bitbucket: wzyboy, GitHub: wzyboy).
Hi,
I found a bug where
bean-pricealways returns today's price no matter what date I provided. I root the cause here: https://bitbucket.org/blais/beancount/src/621cec5ed38bcd128a3502a3b5c367f283deffe2/beancount/prices/sources/google.py?at=default&fileviewer=file-view-default#google.py-181:184The Google Finance API now (since when? I have no idea) always returns a full year's data (from ~365 days ago to today), ignoring
startdateandenddateparameters. So fetching the most recent record (line 1) is always to get today's price.