Changes to StockTradingEnv Cause Hyperparameter Optimization Notebook to Fail

[DLWCMD]

My notebook, _FinRL_HyperparameterTuningOptuna.ipynb, (in Tutorials/4-Optimization) is failing because of the changes recently made to the StockTradingEnv class, specifically the introduction of "initial_list" to _envstocktrading.py. After an initial failure, I created an initial list based on reading the new code, but it failed as well. See below.

This new code was introduced through the standard installation command I have been using since I first began working on my notebook:

!pip install git+https://github.com/AI4Finance-LLC/FinRL-Library.git

In the near term, is there a previous release of the FinRL library available that does not include the initial_list concept? If not, I need further instructions on using the _initiallist.

Thanks. Please let me know if you need more information.

[I 2022-03-08 20:33:10,205] A new study created in memory with name: ddpg_study
Callback threshold 1e-05,             trial_number 5,             patience 15
{'buffer_size': 10000, 'learning_rate': 0.15271325981748027, 'batch_size': 64}
Using cuda device
[W 2022-03-08 20:33:22,933] Trial 0 failed because of the following error: TypeError("'float' object is not subscriptable")
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/optuna/study/_optimize.py", line 213, in _run_trial
    value_or_values = func(trial)
  File "[<ipython-input-21-0d1eff1ddb52>](https://12lizvrkplh-496ff2e9c6d22116-0-colab.googleusercontent.com/outputframe.html?vrz=colab-20220304-060046-RC00_432410246#)", line 26, in objective
    total_timesteps=total_timesteps)
  File "/usr/local/lib/python3.7/dist-packages/finrl/drl_agents/stablebaselines3/models.py", line 103, in train_model
    callback=TensorboardCallback(),
  File "/usr/local/lib/python3.7/dist-packages/stable_baselines3/ddpg/ddpg.py", line 139, in learn
    reset_num_timesteps=reset_num_timesteps,
  File "/usr/local/lib/python3.7/dist-packages/stable_baselines3/td3/td3.py", line 214, in learn
    reset_num_timesteps=reset_num_timesteps,
  File "/usr/local/lib/python3.7/dist-packages/stable_baselines3/common/off_policy_algorithm.py", line 361, in learn
    log_interval=log_interval,
  File "/usr/local/lib/python3.7/dist-packages/stable_baselines3/common/off_policy_algorithm.py", line 587, in collect_rollouts
    new_obs, rewards, dones, infos = env.step(actions)
  File "/usr/local/lib/python3.7/dist-packages/stable_baselines3/common/vec_env/base_vec_env.py", line 162, in step
    return self.step_wait()
  File "/usr/local/lib/python3.7/dist-packages/stable_baselines3/common/vec_env/dummy_vec_env.py", line 44, in step_wait
    self.actions[env_idx]
  File "[/usr/local/lib/python3.7/dist-packages/finrl/finrl_meta/env_stock_trading/env_stocktrading.py](https://12lizvrkplh-496ff2e9c6d22116-0-colab.googleusercontent.com/outputframe.html?vrz=colab-20220304-060046-RC00_432410246#)", line 298, in step
    actions[index] = self._sell_stock(index, actions[index]) * (-1)
  File "/usr/local/lib/python3.7/dist-packages/finrl/finrl_meta/env_stock_trading/env_stocktrading.py", line 149, in _sell_stock
    sell_num_shares = _do_sell_normal()
  File "/usr/local/lib/python3.7/dist-packages/finrl/finrl_meta/env_stock_trading/env_stocktrading.py", line 104, in _do_sell_normal
    * (1 - self.sell_cost_pct[index])
TypeError: 'float' object is not subscriptable
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-21-0d1eff1ddb52> in <module>()
     51                                    trial_number=lc_trial_number)
     52 #You can increase the n_trials for a better search space scanning
---> 53 study.optimize(objective, n_trials=n_trials,catch=(ValueError,),callbacks=[logging_callback])

15 frames
/usr/local/lib/python3.7/dist-packages/finrl/finrl_meta/env_stock_trading/env_stocktrading.py in _do_sell_normal()
    102                         self.state[index + 1]
    103                         * sell_num_shares
--> 104                         * (1 - self.sell_cost_pct[index])
    105                     )
    106                     # update balance

TypeError: 'float' object is not subscriptable

[Athe-kunal]

In your env_kwargs, instead of initial_amount, pass initial_list because inside the init method initial amount is set to the first value only. What changes have you done?

[DLWCMD]

Below is the information I shared with Athe-kunal yesterday. I tested both a initial_list with only dollar amount and an extended list with both cash and numbers of shares for each of the 29 stocks in the portfolio. Both failed.

The attached file provides details. Initial_List_Failures.txt

[DLWCMD]

Astarag,

Thanks for your response, to which I am replying in this email, which came directly to me, not via Slack. Let me know if you want me to also reply on Slack.

I have tried two approaches using initial list, both of which have failed: Only a dollar amount. Thus, initial_list[0] holds this value, consistent with the new code. Dollar amount followed by entries for each stock in portfolio, for a total length of 30

Here is a code snippet:

stock_dimension=29

initial_amt = 1000000

Case (1)

initial_list = [initial_amt]

Case(2)

initialshares = [10 for in range(stock_dimension)]

initial_list.extend(initial_shares)

I will re-run the two cases and share error information with you.

Thanks.

David L. Wilt

3272 Bayou Road

Longboat Key FL 34228

@.***

540-420-0844

From: Astarag Mohapatra @.> Reply-To: AI4Finance-Foundation/FinRL @.> Date: Wednesday, March 9, 2022 at 7:46 AM To: AI4Finance-Foundation/FinRL @.> Cc: DLWCMD @.>, Author @.***> Subject: Re: [AI4Finance-Foundation/FinRL] Changes to StockTradingEnv Cause Hyperparameter Optimization Notebook to Fail (Issue #506)

In your env_kwargs, instead of initial_amount, pass initial_list because inside the init method initial amount is set to the first value only. What changes have you done?

— Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android. You are receiving this because you authored the thread.Message ID: @.***>