search

jimmysitu / msfinance

msfinance offers Pythonic way to download market data from morningstar.com
https://msfinance.readthedocs.io/en/latest/
Apache License 2.0
10 stars 2 forks source link

Automation is Impeded by 'Are you a robot' Check #2

Open wyattschwanbeck opened 1 week ago

wyattschwanbeck commented 1 week ago

Not sure if this is something you've had to deal with from MS but the current implementation doesn't account for the 'Are you a robot' check. I managed to come up with a workaround using debug=True and replacing line 85 in stocks.py with an input prompt telling the user to complete the puzzle.

jimmysitu commented 6 days ago

@wyattschwanbeck , it should be work now with 5609c1a Could you help me test it?

Thanks

wyattschwanbeck commented 6 days ago

Tested via the latest version and found the issue still persists. It may be necessary to keep debug mode on and essentially pause for a user to complete the puzzles needed to continue if you want to still use selenium to get the clean exports that MS provides. Alternatively, i did find method that just requires parsing tons and tons of performance IDs based on the ticker symbol. I don't know if morningstar regularly changes these performance IDs or even updates them for existing tickers but I managed to parse a bunch by manually updating the aws-waf token and using the public API key via the url: url = f"https://api-global.morningstar.com/sal-service/v1/stock/newfinancials/{performanceId}/{reportType}/detail?dataType=A&reportType=A&locale=en&languageId=en&locale=en&clientId=MDC&component=sal-equity-financials-details&version=4.30.0"

It'd be cool if we could map all those performance IDs and figure out if they regularly change across all global ticker symbols. However, I'd also be happy to continue to test further iterations you should you find a way to get around those robot checks using selenium though.

Please keep me posted, Cheers!