mrhappyasthma / IsThisStockGood

A tool for evaluating companies using the Rule #1 investing principles.
http://www.isthisstockgood.com
22 stars 10 forks source link

Morningstar analysis URLs no longer work. #20

Closed mrhappyasthma closed 3 years ago

mrhappyasthma commented 4 years ago

The direct URLs no longer work. Seems to be a refactoring of their website.

However I found out that local library cards get free premium access, so I should be able to reuse that ID to make premium requests.

The biggest drawback is that I need to send a dummy request for the ticker symbol in order to parse out the unique ticker ID that the API requires. I have not found a more robust way to do this.

This also is subject to breaking if the url ever changes.

Background on this approach: https://stackoverflow.com/a/57449982/1366973

mrhappyasthma commented 4 years ago

Example code:

import re
import urllib2

ticker = 'goog'

# This is a pretty fragile way to extract the Morningstar Ticker ID
find_ticker_id_regex = re.compile(r'byId:{"(.*?)"')
find_ticker_url = 'https://www.morningstar.com/stocks/xnas/{}/price-fair-value'.format(ticker)
req = urllib2.Request(find_ticker_url)
resp = urllib2.urlopen(req)
data = resp.read()
ticker_id = find_ticker_id_regex.findall(data)[0]

# Fair value (privlidged)
fair_value_url = 'https://api-global.morningstar.com/sal-service/v1/stock/morningstarTake/v3/{}/analysisData?locale=en&access_token=gIZECmgrLiGStrr5OGQ2W85RvQl0&clientId=MIRC_SALStockModule'.format(ticker_id)

# Quote (privlidged)
quote_url = 'https://api-global.morningstar.com/sal-service/v1/stock/realTime/v3/{}/data?secExchangeList=&random=0.5332403090320754&access_token=gIZECmgrLiGStrr5OGQ2W85RvQl0&clientId=MIRC_SALStockModule'.format(ticker_id)

# Name, PE, Forward PE, star rating (privlidged)
url = 'https://api-global.morningstar.com/sal-service/v1/stock/header/v2/data/{}/securityInfo?access_token=gIZECmgrLiGStrr5OGQ2W85RvQl0&clientId=MIRC_SALStockModule'.format(ticker_id)

company_profile_description_url = 'https://api-global.morningstar.com/sal-service/v1/stock/companyProfile/{}?languageId=en&locale=en&clientId=MIRC_SALStockModule&access_token=gIZECmgrLiGStrr5OGQ2W85RvQl0'.format(ticker_id)

insider_info_url = 'https://api-global.morningstar.com/sal-service/v1/stock/insiders/keyExecutives/{}/data?access_token=gIZECmgrLiGStrr5OGQ2W85RvQl0&clientId=MIRC_SALStockModule'.format(ticker_id)

dividend_and_buyback_url = 'https://api-global.morningstar.com/sal-service/v1/stock/dividends/v4/{}/data?access_token=gIZECmgrLiGStrr5OGQ2W85RvQl0&clientId=MIRC_SALStockModule'.format(ticker_id)

req = urllib2.Request(url)
req.add_header('Origin', 'http://library.morningstar.com')
req.add_header('Referer', 'http://library.morningstar.com/')
req.add_header('User-Agent', 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36')
req.add_header('X-API-REALTIME-E', <redacted>)  # This needs filed in based on your unique ID.
req.add_header('X-API-RequestId', <redacted>)  # This needs filled in based on your unique ID.
req.add_header('X-SAL-ContentType', 'nNsGdN3REOnPMlKDShOYjlk6VYiEVLSdpfpXAm7o2Tk=')
resp = urllib2.urlopen(req)
content = resp.read()
print content
mrhappyasthma commented 4 years ago

As I feared, the login header fields expire. I'll need to find a way to simulate the log in flow to extract these values.

mrhappyasthma commented 4 years ago

Removed the broken URLs in https://github.com/mrhappyasthma/IsThisStockGood/commit/d00bed99ba438f29bcaad785655d002c14136051

mrhappyasthma commented 3 years ago

TODO: remove this - https://github.com/mrhappyasthma/IsThisStockGood/blob/52381ff15babee7b0ba0fe4c73428866e324ba35/templates/js/search.js#L59

mrhappyasthma commented 3 years ago

Since I don't have a good way to authenticate, I'm just gonna remove the lingering code and call this obsolete.

mrhappyasthma commented 3 years ago

Removed the last of the code in https://github.com/mrhappyasthma/IsThisStockGood/commit/9b1081319a434cc777f064c12f8d4cd7fb41fa7b