[IMPROVE] Replace Yahoo Finance functionalities with a different source

[JerBouma]

This is relevant since yet again Yahoo Finance has broken down (partly), see the following ticket: https://github.com/ranaroussi/yfinance/issues/1407 (also mentioned in #4168)

The hypothesis here is that anything related to market data (dividends, stock splits, close, volume etc) are fine since they are freely available to download on Yahoo Finance. Anything else could potentially break anytime. See "Download" button here: https://finance.yahoo.com/quote/MSFT/history?p=MSFT

The ones that will most likely need replacing to another source are:

• stocks/fa/sust (actually broken) Removed ✅
• stocks/quote --source YahooFinance Removed ✅
• stocks/etf/summary (actually broken) Removed ✅
• stocks/etf/weights (actually broken) --> https://financialmodelingprep.com/api/v3/etf-sector-weightings/SPY?apikey=KEY. Replaced with FMP ✅
  • We can also replace stocks/etf/holdings since StockAnalysis is also scraper (or make it secondary) --> https://financialmodelingprep.com/api/v3/etf-sector-weightings/SPY?apikey=KEY
  • We can extend the menu with stocks/etf/countries --> https://financialmodelingprep.com/api/v3/etf-country-weightings/SPY?apikey=KEY
• stocks/etf/news (broken, not entirely important so can also be removed) Removed Yahoo Finance dependency. ✅
• portfolio/load (no data for benchmark available) All Yahoo Finance info statements taken out of the equation. ✅
• portfolio/po --> "Other Optimization Techniques" will fail due to Yahoo Finance. I strongly advise to remove all of these other methods, these are just to play with the technique and are absolutely not great optimisation techniques. Let's stick to the methods deployed by, an expert at the topic: https://github.com/dcajasn/Riskfolio-Lib, and not invent our own. Most removed or changed ✅

The ones that can be deleted since we have an alternative already (but not a priority since not the main source):

• stocks/fa/overview --source Yahoo Finance (can just be deleted)
• stocks/fa/revfc --source YahooFinance (can just be deleted)
• stocks/quote --source YahooFinance (can just be deleted)
• stocks/fa/earnings --source YahooFinance (can just be deleted)
• stocks/fa/income --source YahooFinance (can just be deleted)
• stocks/fa/balance --source YahooFinance (can just be deleted)
• stocks/fa/cash --source YahooFinance (can just be deleted)

The ones that seem to have worked extensively thus lower priority.

• stocks/fa/divs --> https://financialmodelingprep.com/api/v3/historical-price-full/stock_dividend/AAPL?api_key=API_KEY
• stocks/fa/splits --> https://financialmodelingprep.com/api/v3/historical-price-full/stock_split/AAPL?apikey=KEY
• stocks/fa/shrs (could be broken at some point)
• stocks/fa/mgmt (could be broken at some point)

We also have in stocks/disc the following:

│     gainers            show latest top gainers                                         [YahooFinance]                                                                                            │
│     losers             show latest top losers                                          [YahooFinance]                                                                                            │
│     ugs                undervalued stocks with revenue and earnings growth above 25%   [YahooFinance]                                                                                            │
│     gtech              tech stocks with revenue and earnings growth more than 25%      [YahooFinance]                                                                                            │
│     active             most active stocks by intraday trade volumest                   [YahooFinance]                                                                                            │
│     ulc                potentially undervalued large cap stocks                        [YahooFinance]                                                                                            │
│     asc                small cap stocks with earnings growth rates better than 25%     [YahooFinance]

And in dps:

    shorted            show most shorted stocks                                        [YahooFinance]  

Since they haven't broken yet, no need to replace.

[jmaslek]

To get the allocations of SECTORS for spy, we can do:

pd.read_html("https://www.ssga.com/us/en/intermediary/etfs/funds/spdr-sp-500-etf-trust-spy")[-1]

[MrAsimZahid]

Found the following internal API to get financials

https://query2.finance.yahoo.com/ws/fundamentals-timeseries/v1/finance/timeseries/ALB?lang=en-US&region=US&symbol=ALB&padTimeSeries=true&type=annualTaxEffectOfUnusualItems,trailingTaxEffectOfUnusualItems,annualTaxRateForCalcs,trailingTaxRateForCalcs,annualNormalizedEBITDA,trailingNormalizedEBITDA,annualNormalizedDilutedEPS,trailingNormalizedDilutedEPS,annualNormalizedBasicEPS,trailingNormalizedBasicEPS,annualTotalUnusualItems,trailingTotalUnusualItems,annualTotalUnusualItemsExcludingGoodwill,trailingTotalUnusualItemsExcludingGoodwill,annualNetIncomeFromContinuingOperationNetMinorityInterest,trailingNetIncomeFromContinuingOperationNetMinorityInterest,annualReconciledDepreciation,trailingReconciledDepreciation,annualReconciledCostOfRevenue,trailingReconciledCostOfRevenue,annualEBITDA,trailingEBITDA,annualEBIT,trailingEBIT,annualNetInterestIncome,trailingNetInterestIncome,annualInterestExpense,trailingInterestExpense,annualInterestIncome,trailingInterestIncome,annualContinuingAndDiscontinuedDilutedEPS,trailingContinuingAndDiscontinuedDilutedEPS,annualContinuingAndDiscontinuedBasicEPS,trailingContinuingAndDiscontinuedBasicEPS,annualNormalizedIncome,trailingNormalizedIncome,annualNetIncomeFromContinuingAndDiscontinuedOperation,trailingNetIncomeFromContinuingAndDiscontinuedOperation,annualTotalExpenses,trailingTotalExpenses,annualRentExpenseSupplemental,trailingRentExpenseSupplemental,annualReportedNormalizedDilutedEPS,trailingReportedNormalizedDilutedEPS,annualReportedNormalizedBasicEPS,trailingReportedNormalizedBasicEPS,annualTotalOperatingIncomeAsReported,trailingTotalOperatingIncomeAsReported,annualDividendPerShare,trailingDividendPerShare,annualDilutedAverageShares,trailingDilutedAverageShares,annualBasicAverageShares,trailingBasicAverageShares,annualDilutedEPS,trailingDilutedEPS,annualDilutedEPSOtherGainsLosses,trailingDilutedEPSOtherGainsLosses,annualTaxLossCarryforwardDilutedEPS,trailingTaxLossCarryforwardDilutedEPS,annualDilutedAccountingChange,trailingDilutedAccountingChange,annualDilutedExtraordinary,trailingDilutedExtraordinary,annualDilutedDiscontinuousOperations,trailingDilutedDiscontinuousOperations,annualDilutedContinuousOperations,trailingDilutedContinuousOperations,annualBasicEPS,trailingBasicEPS,annualBasicEPSOtherGainsLosses,trailingBasicEPSOtherGainsLosses,annualTaxLossCarryforwardBasicEPS,trailingTaxLossCarryforwardBasicEPS,annualBasicAccountingChange,trailingBasicAccountingChange,annualBasicExtraordinary,trailingBasicExtraordinary,annualBasicDiscontinuousOperations,trailingBasicDiscontinuousOperations,annualBasicContinuousOperations,trailingBasicContinuousOperations,annualDilutedNIAvailtoComStockholders,trailingDilutedNIAvailtoComStockholders,annualAverageDilutionEarnings,trailingAverageDilutionEarnings,annualNetIncomeCommonStockholders,trailingNetIncomeCommonStockholders,annualOtherunderPreferredStockDividend,trailingOtherunderPreferredStockDividend,annualPreferredStockDividends,trailingPreferredStockDividends,annualNetIncome,trailingNetIncome,annualMinorityInterests,trailingMinorityInterests,annualNetIncomeIncludingNoncontrollingInterests,trailingNetIncomeIncludingNoncontrollingInterests,annualNetIncomeFromTaxLossCarryforward,trailingNetIncomeFromTaxLossCarryforward,annualNetIncomeExtraordinary,trailingNetIncomeExtraordinary,annualNetIncomeDiscontinuousOperations,trailingNetIncomeDiscontinuousOperations,annualNetIncomeContinuousOperations,trailingNetIncomeContinuousOperations,annualEarningsFromEquityInterestNetOfTax,trailingEarningsFromEquityInterestNetOfTax,annualTaxProvision,trailingTaxProvision,annualPretaxIncome,trailingPretaxIncome,annualOtherIncomeExpense,trailingOtherIncomeExpense,annualOtherNonOperatingIncomeExpenses,trailingOtherNonOperatingIncomeExpenses,annualSpecialIncomeCharges,trailingSpecialIncomeCharges,annualGainOnSaleOfPPE,trailingGainOnSaleOfPPE,annualGainOnSaleOfBusiness,trailingGainOnSaleOfBusiness,annualOtherSpecialCharges,trailingOtherSpecialCharges,annualWriteOff,trailingWriteOff,annualImpairmentOfCapitalAssets,trailingImpairmentOfCapitalAssets,annualRestructuringAndMergernAcquisition,trailingRestructuringAndMergernAcquisition,annualSecuritiesAmortization,trailingSecuritiesAmortization,annualEarningsFromEquityInterest,trailingEarningsFromEquityInterest,annualGainOnSaleOfSecurity,trailingGainOnSaleOfSecurity,annualNetNonOperatingInterestIncomeExpense,trailingNetNonOperatingInterestIncomeExpense,annualTotalOtherFinanceCost,trailingTotalOtherFinanceCost,annualInterestExpenseNonOperating,trailingInterestExpenseNonOperating,annualInterestIncomeNonOperating,trailingInterestIncomeNonOperating,annualOperatingIncome,trailingOperatingIncome,annualOperatingExpense,trailingOperatingExpense,annualOtherOperatingExpenses,trailingOtherOperatingExpenses,annualOtherTaxes,trailingOtherTaxes,annualProvisionForDoubtfulAccounts,trailingProvisionForDoubtfulAccounts,annualDepreciationAmortizationDepletionIncomeStatement,trailingDepreciationAmortizationDepletionIncomeStatement,annualDepletionIncomeStatement,trailingDepletionIncomeStatement,annualDepreciationAndAmortizationInIncomeStatement,trailingDepreciationAndAmortizationInIncomeStatement,annualAmortization,trailingAmortization,annualAmortizationOfIntangiblesIncomeStatement,trailingAmortizationOfIntangiblesIncomeStatement,annualDepreciationIncomeStatement,trailingDepreciationIncomeStatement,annualResearchAndDevelopment,trailingResearchAndDevelopment,annualSellingGeneralAndAdministration,trailingSellingGeneralAndAdministration,annualSellingAndMarketingExpense,trailingSellingAndMarketingExpense,annualGeneralAndAdministrativeExpense,trailingGeneralAndAdministrativeExpense,annualOtherGandA,trailingOtherGandA,annualInsuranceAndClaims,trailingInsuranceAndClaims,annualRentAndLandingFees,trailingRentAndLandingFees,annualSalariesAndWages,trailingSalariesAndWages,annualGrossProfit,trailingGrossProfit,annualCostOfRevenue,trailingCostOfRevenue,annualTotalRevenue,trailingTotalRevenue,annualExciseTaxes,trailingExciseTaxes,annualOperatingRevenue,trailingOperatingRevenue&merge=false&period1=493590046&period2=1675874638&corsDomain=finance.yahoo.com

[deeleeramone]

Here's how you get all the VanGuard stuff:

https://investor.vanguard.com/investment-products/etfs/profile/api/VTI/profile
etf_id = pd.read_json('https://investor.vanguard.com/investment-products/etfs/profile/api/VTI/profile').loc['associatedFundIds']['fundProfile']['etfFundId']
https://investor.vanguard.com/investment-products/etfs/profile/api/VTI/delayedPrice
https://investor.vanguard.com/investment-products/etfs/profile/api/0970/price-history/1M
https://investor.vanguard.com/investment-products/etfs/profile/api/0970/price-history/1Y
https://investor.vanguard.com/investment-products/etfs/profile/api/VTI/content
https://investor.vanguard.com/investment-products/etfs/profile/api/VTI/classification
https://investor.vanguard.com/investment-products/etfs/profile/api/VTI/characteristic
https://investor.vanguard.com/investment-products/etfs/profile/api/VTI/distribution
https://investor.vanguard.com/investment-products/etfs/profile/api/VTI/performance
https://investor.vanguard.com/investment-products/etfs/profile/api/VTI/price
https://investor.vanguard.com/investment-products/etfs/profile/api/VTI/expense
https://investor.vanguard.com/investment-products/etfs/profile/api/VTI/fee
https://investor.vanguard.com/investment-products/etfs/profile/api/0970/premium-discount/CURR
https://investor.vanguard.com/investment-products/etfs/profile/api/0970/premiumDiscountSummary
https://investor.vanguard.com/investment-products/etfs/profile/api/0970/fundDailyAnalytics
https://investor.vanguard.com/investment-products/etfs/profile/api/VTI/portfolio-holding/bond
https://investor.vanguard.com/investment-products/etfs/profile/api/VTI/portfolio-holding/stock
https://investor.vanguard.com/investment-products/etfs/profile/api/VTI/portfolio-holding/short-term-reserve
https://investor.vanguard.com/investment-products/etfs/profile/api/VTI/portfolio-holding/currency
https://investor.vanguard.com/investment-products/etfs/profile/api/VTI/portfolio-holding/derivative
https://investor.vanguard.com/investment-products/etfs/profile/api/VTI/portfolio-holding/commodity
https://investor.vanguard.com/investment-products/etfs/profile/api/VTI/portfolio-holding/money-market
https://investor.vanguard.com/investment-products/etfs/profile/api/VTI/diversification

Replace VTI with f"{ticker}" and 0970 with f"{etf_id}"

[deeleeramone]

Here's another resource for US-listed ETFs:

https://www.etfcentral.com/nyse/most_recent.json
https://www.etfcentral.com/nyse/funds/movers_shakers.json
https://www.etfcentral.com/nyse/segments/all_segments.json
https://www.etfcentral.com/nyse/segments/stocks-us-large-cap.json
https://www.etfcentral.com/nyse/segments/commodity-energy.json
https://www.etfcentral.com/funds/SPY.json
https://www.etfcentral.com/nyse/similar/SPY.json
https://www.etfcentral.com/snapshots/SPY/2022/stamp.json
https://www.etfcentral.com/snapshots/SPY/2022/flow.json
https://www.etfcentral.com/snapshots/SPY/2022/nav.json
https://www.etfcentral.com/snapshots/SPY/2022/perf.json
https://www.etfcentral.com/holdings/SPY.json
https://www.etfcentral.com/funds/SPY/daily.json

[deeleeramone]

stocks/etf/weights (actually broken) --> https://financialmodelingprep.com/api/v3/etf-sector-weightings/SPY?apikey=KEY.

Any other ETF from FMP - at the free level - does not return results.

For example, https://financialmodelingprep.com/api/v3/etf-holdings/QQQ?apikey=

[deeleeramone]

On another note, it seems unnecessary to delete functions. The goal should be to remove the blockers which break the terminal as a result of a single point of failure and that helper functions do not cheat to get some basic metadata from yFinance instead of the --source which was requested by the user.

Despite the error message being printed, this is the actual DataFrame being returned from yFinance.

[deeleeramone]

Need to add etf/pir to the list.

2023 Feb 08, 23:00 (🦋) /etf/ $ load spy

Top holdings found:
   Apple Inc.
   Microsoft Corporation
   Amazon.com, Inc.
   Alphabet Inc.
   Berkshire Hathaway Inc.

2023 Feb 08, 23:00 (🦋) /etf/ $ weights

Error: yfinance failed to decrypt Yahoo data response

No data was found for that ETF

2023 Feb 08, 23:00 (🦋) /etf/ $ pir

Not able to collect data for SPY
Created ETF report as ETF_report_20230208_200121 in folder /Users/danglewood/OpenBBUserData/exports 

[JerBouma]

The point of the PR will just be to delete those that currently do not work or find a replacement where possible.

So anything I can replace with an actual supported official API, I'll do and nothing that points to a website or result into scraping a webpage that can change over time again. Especially since it will be against their ToS or/and will be unusable in the OpenBB Terminal Pro solution.

[JerBouma]

stocks/etf/weights (actually broken) --> https://financialmodelingprep.com/api/v3/etf-sector-weightings/SPY?apikey=KEY.

Any other ETF from FMP - at the free level - does not return results.

For example, https://financialmodelingprep.com/api/v3/etf-holdings/QQQ?apikey=

This is false, some are indeed behind a Premium endpoint but that doesn't mean all are.

https://financialmodelingprep.com/api/v3/etf-sector-weightings/SPY?apikey= https://financialmodelingprep.com/api/v3/etf-sector-weightings/VOO?apikey= https://financialmodelingprep.com/api/v3/etf-sector-weightings/VTI?apikey= https://financialmodelingprep.com/api/v3/etf-sector-weightings/ARKK?apikey= https://financialmodelingprep.com/api/v3/etf-sector-weightings/IVV?apikey= https://financialmodelingprep.com/api/v3/etf-sector-weightings/EEM?apikey=

[andrewkenreich]

What about that other lib that needs YFinance to work? :P

[JerBouma]

Which one?

[jmaslek]

It's something about a database in the finance world

[JerBouma]

Doesn't rely on yFinance though, just updating which I already confirmed is not going to happen anymore :)

[andrewkenreich]

ok well we should get rid of that too then - or we just asking for trouble later

[JerBouma]

It needs to be removed at some point yes but that would mean we are deleting the entire stocks/sia menu as well. That's a bit odd especially since the database isn't providing severely outdated information.

[andrewkenreich]

Yes but eventually we are going to have people asking "why can't I find X ticker?" And our answer shouldn't be - "well we aren't updating that database anymore"

[JerBouma]

Correct but we would need to find a replacement then first that doesn't only cover the American exchanges but any exchange. In essence the only tickers that are missing are the ones that IPO recently which is only a very small percentage of the total amount of companies available as there are 155.000 tickers in there. And even then, I can just manually add them in.

So really, before we have people consistently spamming us that they can't find a company they are looking for and it becomes impossible for me to update it, you are looking at several years. I am all for when we find something free that matches it and does update, of course.

E.g. I realised Meta was still called Facebook so a quick CTRL + H fixed that.

[andrewkenreich]

Maybe we can work on something once some Dev time is freed up later this year then

[deeleeramone]

Quality > Quantity. What does anyone need obsolete market cap data for?

The SIA menu has lots of issues, and the stats presented have no way to be verified. For example, there are definitely more than six companies in the Financial Sector. The names of those companies can't be displayed. What purpose is this data serving?

There isn't anything unique about a list of exchange-traded products. Every single exchange in the world maintains a list of symbols, and it is updated daily. There are myriads of reasons why a symbol is updated, added, or removed. IPOs don't even crack the top ten.

https://www.statista.com/statistics/1277216/nyse-nasdaq-comparison-number-listed-companies/

While the NYSE has generally had around 1,800 to 1,850 listed domestic companies, October 2020 saw a spike up to 2,363. In total, as of September 2022, the NYSE had a combined total of 2,578 listed domestic and international companies, while the Nasdaq had a much higher 3,788.

Here's "companies per Industry based on Country (and Market Cap)" - with no market cap.

According to these statistics the Russell 2000 or Wilshire 5000 probably shouldn't exist.

https://www.investopedia.com/terms/f/financial_sector.asp

Financial Sector Makeup

As mentioned above, the financial sector is made up of many different industries ranging from banks, investment houses, insurance companies, real estate brokers, consumer finance companies, mortgage lenders, and [real estate investment trusts](https://www.investopedia.com/terms/r/reit.asp) (REITs).

The financial sector is one of the largest portions of the [S&P 500](https://www.investopedia.com/terms/s/sp500.asp). The largest companies within the financial sector are some of the most recognizable banking institutions in the world, including the following:

    JPMorgan Chase (JPM)
    Wells Fargo (WFC)
    Bank of America (BAC)
    Citigroup (C)

While these large companies dominate the sector, there are other, smaller companies that participate in the sector as well. Insurers are also a major industry within the financial sector, being made up of such companies as American International Group (AIG) and Chubb (CB).

https://www.theice.com/products