search

caiobran / mstables

MorningStar.com scraper that consolidates tens of thousands of financial records into a SQLite relational database. Class 'dataframes' easily converts the SQLite data into pandas DataFrames (see Jupyter notebook for examples)
MIT License
180 stars 46 forks source link

Running option 1 fetchs 0 stocks, funds, etfs... then option 2 creates an error #15

Open jobordu opened 1 year ago

jobordu commented 1 year ago

Is the scrapper functionnal ? When running option 1, it fetchs 0 stocks, funds, etfs...


====================== Welcome to msTables ======================

Available actions:

0 - Change database file name (current name = 'mstables.sqlite')
1 - Create database tables and import latest symbols
2 - Download Morningstar data into database
3 - Erase all records from database tables
4 - Delete all database tables
5 - Erase all downloaded history from 'Fetched_urls' table
6 - Create a database back-up file

=================================================================
Enter action no.:
1

Please wait, database tables are being created ...

Fetching list of stock from MorningStar.com

Parsing list of stock
List of stock length = 0

Fetching list of cefs from MorningStar.com
Parsing list of cefs
List of cefs length = 0

Fetching list of funds from MorningStar.com
Parsing list of funds
List of funds length = 0

Fetching list of etfs from MorningStar.com
Parsing list of etfs
List of etfs length = 0

Total length = 0

~ The following 34 database tables were successfully created:

  - ColHeaders
  - Companies
  - Countries
  - Currencies
  - Exchanges
  - Fetched_urls
  - Industries
  - InsiderTransactions
  - Insiders
  - MSfinancials
  - MSheader
  - MSpricehistory
  - MSratio_cashflow
  - MSratio_efficiency
  - MSratio_financial
  - MSratio_growth
  - MSratio_profitability
  - MSreport_bs_qt
  - MSreport_bs_yr
  - MSreport_cf_qt
  - MSreport_cf_yr
  - MSreport_is_qt
  - MSreport_is_yr
  - MSvaluation
  - Master
  - Sectors
  - SecurityTypes
  - StockStyles
  - StockTypes
  - Tickers
  - TimeRefs
  - TransactionType
  - URLs
  - YahooQuote

~ Execution Time    1.66 sec

Then running option 2:

=================================================================
====================== Welcome to msTables ======================

Available actions:

0 - Change database file name (current name = 'mstables.sqlite')
1 - Create database tables and import latest symbols
2 - Download Morningstar data into database
3 - Erase all records from database tables
4 - Delete all database tables
5 - Erase all downloaded history from 'Fetched_urls' table
6 - Create a database back-up file

=================================================================
Enter action no.:
2

Qty. of records to be updated:
10000
\e[KCreating URL list for API 1 ...
\e[KCreating URL list for API 2 ...
\e[KCreating URL list for API 3 ...
\e[KCreating URL list for API 4 ...
\e[KCreating URL list for API 5 ...
\e[KCreating URL list for API 6 ...
\e[KCreating URL list for API 7 ...
\e[KCreating URL list for API 8 ...
\e[KCreating URL list for API 9 ...
\e[KCreating URL list for API 10 ...
\e[KCreating URL list for API 11 ...
\e[KCreating URL list for API 12 ...
\e[KCreating URL list for API 13 ...
\e[KCreating URL list for API 14 ...
\e[KCreating URL list for API 15 ...
\e[KCreating URL list for API 16 ...
\e[K
Qty. of records pending update per API no.:

      Pending
API          
1      88,737
2      88,737
3      88,737
4     138,312
5     138,312
6     138,312
7     138,312
8     138,312
9      34,032
10    138,312
11    138,312
12    138,312
13    138,312
14    138,312
15    138,312
16    138,312

Total URL requests pending =    1,959,987
Total URL requests planned =      160,000

Run 1 / 67  (150 requests per API per run = 2400 requests per run)

multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/Users/jonathanborduas/anaconda3/lib/python3.10/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/Users/jonathanborduas/Documents/development/mstables/fetch.py", line 377, in fetch_api
    num = ticker_list[url_id]['{}:{}'.format(exch_id, ticker_id)]
KeyError: 1
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/jonathanborduas/Documents/development/mstables/main.py", line 161, in <module>
    main(db_file)
  File "/Users/jonathanborduas/Documents/development/mstables/main.py", line 104, in main
    start = fetch.fetch(db_file['path'])
  File "/Users/jonathanborduas/Documents/development/mstables/fetch.py", line 302, in fetch
    results.append(r.next(timeout=5))
  File "/Users/jonathanborduas/anaconda3/lib/python3.10/multiprocessing/pool.py", line 873, in next
    raise value
KeyError: 1
caiobran commented 1 year ago

Hi @jobordu , unfortunately the API's are no longer working so the tool is not function as is.