[Bug] Error out when backtesting

[WyattLiu]

Describe the bug In Pred, once specifies backtesting date such as lstm -e 2021-03-23, it has KeyError:1980

To Reproduce load AMD pred AMD lstm -e 2021-03-23

Screenshots

(✨) (pred)> help

Daily Stock: AMD

Prediction Techniques:
   help        show this prediction techniques menu again
   q           quit this menu, and shows back to main menu
   quit        quit to abandon program

   sma         simple moving average
   ets         exponential smoothing (e.g. Holt-Winters)
   knn         k-Nearest Neighbors
   linear      linear regression (polynomial 1)
   quadratic   quadratic regression (polynomial 2)
   cubic       cubic regression (polynomial 3)
   regression  regression (other polynomial)
   arima       autoregressive integrated moving average
   prophet     Facebook's prophet prediction
   mlp         MultiLayer Perceptron
   rnn         Recurrent Neural Network
   lstm        Long-Short Term Memory

(✨) (pred)> lstm -e 2021-03-23
1980
Traceback (most recent call last):
  File "/home/wyatt/GamestonkTerminal/gamestonk_terminal/prediction_techniques/neural_networks.py", line 872, in lstm
    df_stock, df_future = _setup_backtesting(df_stock, ns_parser)
  File "/home/wyatt/GamestonkTerminal/gamestonk_terminal/prediction_techniques/neural_networks.py", line 290, in _setup_backtesting
    if ns_parser.s_end_date < get_next_stock_market_days(
  File "/home/wyatt/GamestonkTerminal/gamestonk_terminal/helper_funcs.py", line 211, in get_next_stock_market_days
    holidays = holidays + us_market_holidays(year)
  File "/home/wyatt/GamestonkTerminal/gamestonk_terminal/helper_funcs.py", line 119, in us_market_holidays
    validHolidays.append(datetime.strptime(goodFridays[year], "%Y-%m-%d").date())
KeyError: 1980

Desktop (please complete the following information):

• OS:Ubuntu 18.04.5 LTS
• Python 3.8.8

Additional context N/A

[jmaslek]

If you load the stock with a more recent start date, maybe load -t AMD -s 2015-01-01 does the pred work?

Looks like the key error is from a us holiday that year...

[jmaslek]

You also will need to set the end date to I believe 5 days previous to today.

@DidierRLopes can confirm that. And also we may need to relook at that PR involving dates (or just default maybe a couple years not all time under load)

[WyattLiu]

confirming load with start date works -> at least meaningful debug msg is here:

(✨) (pred)> lstm -e 2021-03-22
Backtesting not allowed, since End Date + Prediction days is in the future
Traceback (most recent call last):
  File "/home/wyatt/GamestonkTerminal/gamestonk_terminal/prediction_techniques/neural_networks.py", line 872, in lstm
    df_stock, df_future = _setup_backtesting(df_stock, ns_parser)
  File "/home/wyatt/GamestonkTerminal/gamestonk_terminal/prediction_techniques/neural_networks.py", line 303, in _setup_backtesting
    raise Exception(
Exception: Backtesting not allowed, since End Date + Prediction days is in the future

Confirm back dial 5 market open days works.

Thank you, I hope this helps confused people like me.

May I ask why default load AMD without start date will not work? Should we have a default start date loaded like 1000 market days ago

[jmaslek]

It seems that the default doesn't work due to some kind of holiday calendar. The key error seems to indicate the holiday error was in 1980.

We should probably set a default start load date. Also if you are training a model such as lstm, more recent data should be used since market conditions change over time.

Edit: confirmed that the holiday helper function has Good Friday's only back through 2010 (also needs dates to be updated) so I bet that any stock with data loaded before 2010 would raise this issue

[DidierRLopes]

@jmaslek is 100% correct. The PR he has created should fix the issue, thank you for reporting it, and thanks to @jmaslek for reacting so quickly.

I'll close this now on that basis :)