search

AI4Finance-Foundation / FinRL

FinRL: Financial Reinforcement Learning. 🔥
https://ai4finance.org
MIT License
10.2k stars 2.45k forks source link

Google colab Stock_NeurIPS2018_SB3.ipynb StockTradingEnv.reset() got an unexpected keyword argument 'seed' #1013

Open eugene933 opened 1 year ago

eugene933 commented 1 year ago

In running the following code in google colab for Stock_NeurIPS2018_SB3.ipynb

envtrain, = e_train_gym.get_sb_env() I get the following error. Seems related to gym version. Gynasium 0.28.1 is installed.

TypeError Traceback (most recent call last) in <cell line: 1>() ----> 1 envtrain, = e_train_gym.get_sb_env() 2 print(type(env_train))

1 frames /usr/local/lib/python3.10/site-packages/stable_baselines3/common/vec_env/dummy_vec_env.py in reset(self) 74 def reset(self) -> VecEnvObs: 75 for env_idx in range(self.num_envs): ---> 76 obs, self.reset_infos[env_idx] = self.envs[env_idx].reset(seed=self._seeds[env_idx]) 77 self._save_obs(env_idx, obs) 78 # Seeds are only used once

TypeError: StockTradingEnv.reset() got an unexpected keyword argument 'seed'

--

!pip show gymnasium Name: gymnasium Version: 0.28.1 Summary: A standard API for reinforcement learning and a diverse set of reference environments (formerly Gym). Home-page: Author: Author-email: Farama Foundation [contact@farama.org](mailto:contact@farama.org) License: MIT License Location: /usr/local/lib/python3.10/site-packages Requires: cloudpickle, farama-notifications, jax-jumpy, numpy, typing-extensions Required-by: Shimmy, stable-baselines3

eugene933 commented 1 year ago

Same problem in FinRL_Ensemble_StockTrading_ICAIF_2020.ipynb on google colab

df_summary = ensemble_agent.run_ensemble_strategy(A2C_model_kwargs, PPO_model_kwargs, DDPG_model_kwargs, timesteps_dict)

============Start Ensemble Strategy============

turbulence_threshold: 201.7176868788971 ======Model training from: 2010-01-01 to 2021-10-04 ======A2C Training======== {'n_steps': 5, 'ent_coef': 0.005, 'learning_rate': 0.0007} Using cuda device

TypeError Traceback (most recent call last) in <cell line: 1>() ----> 1 df_summary = ensemble_agent.run_ensemble_strategy(A2C_model_kwargs, 2 PPO_model_kwargs, 3 DDPG_model_kwargs, 4 timesteps_dict)

5 frames /usr/local/lib/python3.10/site-packages/stable_baselines3/common/vec_env/dummy_vec_env.py in reset(self) 74 def reset(self) -> VecEnvObs: 75 for env_idx in range(self.num_envs): ---> 76 obs, self.reset_infos[env_idx] = self.envs[env_idx].reset(seed=self._seeds[env_idx]) 77 self._save_obs(env_idx, obs) 78 # Seeds are only used once

TypeError: StockTradingEnv.reset() got an unexpected keyword argument 'seed'

shql-tgr commented 1 year ago

if you copy the dummy_vec_env.py, remove the seed=self._seeds[env_idx] inside the reset(self) function, and then implement this modified version inside the trading environment, it works. Of course it is not a "real" solution, since it gets around the problem instead of fixing it, but it makes it work

ildefons commented 1 year ago

The last stable version of stable3_baseline (2.0.0a9) is buggy. Just install the previous working version: pip install stable-baselines3==2.0.0a5

YangletLiu commented 1 year ago

The last stable version of stable3_baseline (2.0.0a9) is buggy. Just install the previous working version: pip install stable-baselines3==2.0.0a5

Would you like to submit a PR to fix it?

cindyhli2012 commented 1 year ago

The last stable version of stable3_baseline (2.0.0a9) is buggy. Just install the previous working version: pip install stable-baselines3==2.0.0a5

This works for me.

eugene933 commented 1 year ago

The last stable version of stable3_baseline (2.0.0a9) is buggy. Just install the previous working version: pip install stable-baselines3==2.0.0a5

This doesn't work either. Slightly different error message

envtrain, = e_train_gym.get_sb_env() print(type(env_train))

TypeError Traceback (most recent call last) in <cell line: 1>() ----> 1 envtrain, = e_train_gym.get_sb_env() 2 print(type(env_train))

1 frames /usr/local/lib/python3.10/site-packages/stable_baselines3/common/vec_env/dummy_vec_env.py in reset(self) 74 # Avoid circular import 75 from stable_baselines3.common.utils import compat_gym_seed ---> 76 77 if seed is None: 78 seed = np.random.randint(0, 2**32 - 1)

TypeError: StockTradingEnv.reset() got an unexpected keyword argument 'seed'

eugene933 commented 1 year ago

Similar error for the most recent stable_baseline3 version 2.0.0a11

!pip install git+https://github.com/DLR-RM/stable-baselines3

TypeError Traceback (most recent call last) in <cell line: 1>() ----> 1 envtrain, = e_train_gym.get_sb_env() 2 print(type(env_train))

1 frames /usr/local/lib/python3.10/site-packages/stable_baselines3/common/vec_env/dummy_vec_env.py in reset(self) 74 def reset(self) -> VecEnvObs: 75 for env_idx in range(self.num_envs): ---> 76 obs, self.reset_infos[env_idx] = self.envs[env_idx].reset(seed=self._seeds[env_idx]) 77 self._save_obs(env_idx, obs) 78 # Seeds are only used once

TypeError: StockTradingEnv.reset() got an unexpected keyword argument 'seed'

ildefons commented 1 year ago

The last stable version of stable3_baseline (2.0.0a9) is buggy. Just install the previous working version: pip install stable-baselines3==2.0.0a5

Would you like to submit a PR to fix it?

IMO best would be to add a quick note in the installation instructions web page just after the regular project pip install. Eventually, with newer versions of the stable3_baseline package we will figure out whether this is a bug or a new feature. If this is indeed a feature, then a PR will be needed to update the FinRL env.

ildefons commented 1 year ago

The last stable version of stable3_baseline (2.0.0a9) is buggy. Just install the previous working version: pip install stable-baselines3==2.0.0a5

This doesn't work either. Slightly different error message

envtrain, = e_train_gym.get_sb_env() print(type(env_train))

TypeError Traceback (most recent call last) in <cell line: 1>() ----> 1 envtrain, = e_train_gym.get_sb_env() 2 print(type(env_train))

1 frames /usr/local/lib/python3.10/site-packages/stable_baselines3/common/vec_env/dummy_vec_env.py in reset(self) 74 # Avoid circular import 75 from stable_baselines3.common.utils import compat_gym_seed ---> 76 77 if seed is None: 78 seed = np.random.randint(0, 2**32 - 1)

TypeError: StockTradingEnv.reset() got an unexpected keyword argument 'seed'

Could you check which stable3_baseline release do you have by doing "pip list" inside your conda environment?

sateeh2010 commented 1 year ago

Same issue and I have following version: stable-baselines3 2.0.0a5

eugene933 commented 1 year ago

pip list This is the output. 2.0.0a5 for stable-baselines3 Thanks in advance.

Package Version

absl-py 1.4.0 aiodns 3.0.0 aiohttp 3.8.1 aiohttp-cors 0.7.0 aiosignal 1.3.1 ale-py 0.8.1 alpaca-trade-api 3.0.0 appdirs 1.4.4 asttokens 2.2.1 async-timeout 4.0.2 attrs 23.1.0 AutoROM 0.6.1 AutoROM.accept-rom-license 0.6.1 backcall 0.2.0 beautifulsoup4 4.12.2 blessed 1.20.0 box2d-py 2.3.5 brotlipy 0.7.0 cachetools 5.3.1 ccxt 3.1.13 certifi 2022.12.7 cffi 1.15.1 charset-normalizer 2.1.1 click 8.1.3 cloudpickle 2.2.1 cmake 3.26.3 colorama 0.4.6 colorful 0.5.5 conda 23.1.0 conda-package-handling 2.0.2 conda_package_streaming 0.7.0 contourpy 1.0.7 cryptography 40.0.1 cvxpy 1.3.1 cycler 0.11.0 decorator 5.1.1 deprecation 2.1.0 distlib 0.3.6 ecos 2.0.12 elegantrl 0.3.6 empyrical 0.5.5 exchange-calendars 4.2.8 executing 1.2.0 Farama-Notifications 0.0.4 filelock 3.12.0 finrl 0.3.6 fonttools 4.39.4 frozendict 2.3.8 frozenlist 1.3.3 google-api-core 2.11.0 google-auth 2.19.0 google-auth-oauthlib 1.0.0 googleapis-common-protos 1.59.0 gpustat 1.1 greenlet 2.0.2 grpcio 1.51.3 gym 0.26.2 gym-notices 0.0.8 gymnasium 0.28.1 html5lib 1.1 idna 3.4 importlib-resources 5.12.0 ipython 8.13.2 jax-jumpy 1.0.0 jedi 0.18.2 Jinja2 3.1.2 joblib 1.2.0 jqdatasdk 1.8.11 jsonschema 4.17.3 kiwisolver 1.4.4 korean-lunar-calendar 0.3.1 libmambapy 1.4.1 lit 16.0.5 lxml 4.9.2 mamba 1.4.1 Markdown 3.4.3 markdown-it-py 2.2.0 MarkupSafe 2.1.2 matplotlib 3.7.1 matplotlib-inline 0.1.6 mdurl 0.1.2 mpmath 1.3.0 msgpack 1.0.3 multidict 6.0.4 multitasking 0.0.11 networkx 3.1 numpy 1.24.3 nvidia-cublas-cu11 11.10.3.66 nvidia-cuda-cupti-cu11 11.7.101 nvidia-cuda-nvrtc-cu11 11.7.99 nvidia-cuda-runtime-cu11 11.7.99 nvidia-cudnn-cu11 8.5.0.96 nvidia-cufft-cu11 10.9.0.58 nvidia-curand-cu11 10.2.10.91 nvidia-cusolver-cu11 11.4.0.1 nvidia-cusparse-cu11 11.7.4.91 nvidia-ml-py 11.525.112 nvidia-nccl-cu11 2.14.3 nvidia-nvtx-cu11 11.7.91 oauthlib 3.2.2 opencensus 0.11.2 opencensus-context 0.1.3 opencv-python 4.7.0.72 osqp 0.6.3 packaging 23.1 pandas 2.0.2 pandas-datareader 0.10.0 parso 0.8.3 pexpect 4.8.0 pickleshare 0.7.5 Pillow 9.5.0 pip 23.0.1 platformdirs 3.5.1 pluggy 1.0.0 ply 3.11 prometheus-client 0.17.0 prompt-toolkit 3.0.38 protobuf 3.20.3 psutil 5.9.5 psycopg2-binary 2.9.6 ptyprocess 0.7.0 pure-eval 0.2.2 py-spy 0.3.14 pyasn1 0.5.0 pyasn1-modules 0.3.0 pycares 4.3.0 pycosat 0.6.4 pycparser 2.21 pydantic 1.10.8 pyfolio 0.9.2 pygame 2.1.0 Pygments 2.15.1 pyluach 2.2.0 PyMySQL 1.0.3 pyOpenSSL 23.1.1 pyparsing 3.0.9 pyportfolioopt 1.5.5 pyrsistent 0.19.3 PySocks 1.7.1 python-dateutil 2.8.2 pytz 2023.3 PyYAML 6.0 qdldl 0.1.7 ray 2.4.0 requests 2.28.2 requests-oauthlib 1.3.1 rich 13.3.5 rsa 4.9 ruamel.yaml 0.17.21 ruamel.yaml.clib 0.2.7 scikit-learn 1.2.2 scipy 1.10.1 scs 3.2.3 seaborn 0.12.2 setuptools 65.6.3 Shimmy 0.2.1 six 1.16.0 smart-open 6.3.0 soupsieve 2.4.1 SQLAlchemy 1.4.48 stable-baselines3 2.0.0a5 stack-data 0.6.2 stockstats 0.5.2 swig 4.1.1 sympy 1.12 tabulate 0.9.0 tensorboard 2.13.0 tensorboard-data-server 0.7.0 tensorboardX 2.6 threadpoolctl 3.1.0 thriftpy2 0.4.16 toolz 0.12.0 torch 2.0.1 tqdm 4.65.0 traitlets 5.9.0 triton 2.0.0 typing_extensions 4.6.2 tzdata 2023.3 urllib3 1.26.15 virtualenv 20.21.0 wcwidth 0.2.6 webencodings 0.5.1 websocket-client 1.5.2 websockets 10.4 Werkzeug 2.3.4 wheel 0.40.0 wrds 3.1.6 yarl 1.9.2 yfinance 0.2.18 zstandard 0.19.0

ildefons commented 1 year ago

Start a new setup from scratch: 1) create a new conda environment 2) activate conda environment 3) pip install FinRL project 4) pip install stable-baselines3==2.0.0a5

pip list This is the output. 2.0.0a5 for stable-baselines3 Thanks in advance.

Aashish-1008 commented 1 year ago

Start a new setup from scratch:

  1. create a new conda environment
  2. activate conda environment
  3. pip install FinRL project
  4. pip install stable-baselines3==2.0.0a5

pip list This is the output. 2.0.0a5 for stable-baselines3 Thanks in advance.

It fixes the main issue. thanks for that.

I am running the FinRL_Ensemble_StockTrading_ICAIF_2020.

Getting below error, please find below logs

`======PPO Validation from: 2021-10-04 to 2022-01-03 PPO Sharpe Ratio: 0.1777555768777347 ======DDPG Training======== {'buffer_size': 10000, 'learning_rate': 0.0005, 'batch_size': 64} Using cpu device Logging to tensorboard_log/ddpg/ddpg_126_2 day: 2957, episode: 10 begin_total_asset: 1000000.00 end_total_asset: 4007726.44 total_reward: 3007726.44 total_cost: 1320.17 total_trades: 49302 Sharpe: 0.730

| time/ | | | episodes | 4 | | fps | 50 | | time_elapsed | 232 | | total_timesteps | 11832 | | train/ | | | actor_loss | 25.2 | | critic_loss | 325 | | learning_rate | 0.0005 | | n_updates | 8874 | | reward | 4.2382293 |

======DDPG Validation from: 2021-10-04 to 2022-01-03 ======Best Model Retraining from: 2010-01-01 to 2022-01-03 ======Trading from: 2022-01-03 to 2022-04-04

ValueError Traceback (most recent call last) Cell In[21], line 1 ----> 1 df_summary = ensemble_agent.run_ensemble_strategy(A2C_model_kwargs, 2 PPO_model_kwargs, 3 DDPG_model_kwargs, 4 timesteps_dict)

File ~/anaconda3/lib/python3.10/site-packages/finrl/agents/stablebaselines3/models.py:668, in DRLEnsembleAgent.run_ensemble_strategy(self, A2C_model_kwargs, PPO_model_kwargs, DDPG_model_kwargs, timesteps_dict) 661 print( 662 "======Trading from: ", 663 self.unique_trade_date[i - self.rebalance_window], 664 "to ", 665 self.unique_trade_date[i], 666 ) 667 # print("Used Model: ", model_ensemble) --> 668 last_state_ensemble = self.DRL_prediction( 669 model=model_ensemble, 670 name="ensemble", 671 last_state=last_state_ensemble, 672 iter_num=i, 673 turbulence_threshold=turbulence_threshold, 674 initial=initial, 675 ) 676 ############## Trading ends ############## 678 end = time.time()

File ~/anaconda3/lib/python3.10/site-packages/finrl/agents/stablebaselines3/models.py:321, in DRLEnsembleAgent.DRL_prediction(self, model, name, last_state, iter_num, turbulence_threshold, initial) 317 if i == (len(trade_data.index.unique()) - 2): 318 # print(env_test.render()) 319 last_state = trade_env.render() --> 321 df_last_state = pd.DataFrame({"last_state": last_state}) 322 df_last_state.to_csv(f"results/laststate{name}_{i}.csv", index=False) 323 return last_state

File ~/anaconda3/lib/python3.10/site-packages/pandas/core/frame.py:664, in DataFrame.init(self, data, index, columns, dtype, copy) 658 mgr = self._init_mgr( 659 data, axes={"index": index, "columns": columns}, dtype=dtype, copy=copy 660 ) 662 elif isinstance(data, dict): 663 # GH#38939 de facto copy defaults to False only in non-dict cases --> 664 mgr = dict_to_mgr(data, index, columns, dtype=dtype, copy=copy, typ=manager) 665 elif isinstance(data, ma.MaskedArray): 666 import numpy.ma.mrecords as mrecords

File ~/anaconda3/lib/python3.10/site-packages/pandas/core/internals/construction.py:493, in dict_to_mgr(data, index, columns, dtype, typ, copy) 489 else: 490 # dtype check to exclude e.g. range objects, scalars 491 arrays = [x.copy() if hasattr(x, "dtype") else x for x in arrays] --> 493 return arrays_to_mgr(arrays, columns, index, dtype=dtype, typ=typ, consolidate=copy)

File ~/anaconda3/lib/python3.10/site-packages/pandas/core/internals/construction.py:118, in arrays_to_mgr(arrays, columns, index, dtype, verify_integrity, typ, consolidate) 115 if verify_integrity: 116 # figure out the index, if necessary 117 if index is None: --> 118 index = _extract_index(arrays) 119 else: 120 index = ensure_index(index)

File ~/anaconda3/lib/python3.10/site-packages/pandas/core/internals/construction.py:656, in _extract_index(data) 653 raise ValueError("Per-column arrays must each be 1-dimensional") 655 if not indexes and not raw_lengths: --> 656 raise ValueError("If using all scalar values, you must pass an index") 658 elif have_series: 659 index = union_indexes(indexes)

ValueError: If using all scalar values, you must pass an index

`