passing env to keras model.fit()

[traderpro446]

Is it possible to pass btGym environments to Keras-rl model.fit() https://github.com/keras-rl/keras-rl ?

The GYM environments can be easily passed to Keras-rl model.fit(env) to be treated as training data.

However, I get an error when I am trying it with btgym envs. ValueError: Error when checking input: expected lstm_1_input to have 3 dimensions, but got array with shape (1, 1)

[Kismuz]

@traderpro446 , would you please provide your exact setup and full error traceback? That's hardly possible to identify the problem origin by the single output line. Refer to issue opening template guidelines.

[traderpro446]

Thanks for the reply. There are two issues that I outline below.

First I tried to create an environment similar to the example in your "setting_up_environment_basic". I got this error:

[2018-07-08 14:40:07.028706] NOTICE: BTgymDataServer_0: Initial global_time set to: 2016-01-03 18:01:00 / stamp: 1451840460.0 [2018-07-08 14:40:07.123182] ERROR: BTgymAPIshell_0: State observation shape/range mismatch! Space set by env:

raw_state: Box(30, 4), low: 1.03522, high: 1.1616

Space returned by server:

metadata: first_row: array of shape: (), low: 0, high: 0

trial_num: array of shape: (), low: 0, high: 0

timestamp: array of shape: (), low: 1457947380.0, high: 1457947380.0

trial_type: array of shape: (), low: 0, high: 0

type: array of shape: (), low: 0, high: 0

sample_num: array of shape: (), low: 0, high: 0

raw_state: array of shape: (30, 4), low: 1.11002, high: 1.11152

Full response: {'metadata': {'first_row': array(0), 'trial_num': array(0), 'timestamp': array(1.45794738e+09), 'trial_type': array(0), 'type': array(0), 'sample_num': array(0)}, 'raw_state': array([[1.11048, 1.11048, 1.11033, 1.11034], [1.11035, 1.11053, 1.11022, 1.11022], [1.11023, 1.11034, 1.1101 , 1.11015], [1.11016, 1.1103 , 1.11002, 1.11019], [1.11019, 1.1107 , 1.11016, 1.1107 ], [1.11069, 1.11112, 1.11042, 1.11111], [1.11111, 1.11117, 1.11081, 1.11104], [1.11105, 1.11112, 1.11075, 1.11102], [1.11103, 1.11152, 1.11093, 1.11143], [1.11142, 1.11142, 1.1109 , 1.11096], [1.11095, 1.11114, 1.11083, 1.11093], [1.11093, 1.11107, 1.11075, 1.11078], [1.11077, 1.11079, 1.11061, 1.11066], [1.11066, 1.11073, 1.11055, 1.11065], [1.11066, 1.11067, 1.11031, 1.11055], [1.11055, 1.11055, 1.1102 , 1.11046], [1.11047, 1.11052, 1.11039, 1.1105 ], [1.11047, 1.11078, 1.11047, 1.11072], [1.11073, 1.11082, 1.11055, 1.11058], [1.11057, 1.11081, 1.11046, 1.11081], [1.11079, 1.11089, 1.11071, 1.11082], [1.11086, 1.11094, 1.11066, 1.11067], [1.11067, 1.11076, 1.11056, 1.11076], [1.11076, 1.11079, 1.11032, 1.11039], [1.11041, 1.11051, 1.11039, 1.1105 ], [1.11051, 1.11077, 1.1105 , 1.11057], [1.11057, 1.11096, 1.11057, 1.11092], [1.11095, 1.11098, 1.11082, 1.11095], [1.11096, 1.11099, 1.11069, 1.11087], [1.11086, 1.11097, 1.11086, 1.11092]])} Reward: 0.0 Done: False Info: {'drawdown': 0.0, 'broker_value': 100000.0, 'time': datetime.datetime(2016, 3, 14, 10, 23), 'max_drawdown': 0.0, 'brokermessage': '', 'step': 0, 'broker_cash': 100000.0, 'action': 'hold'}

Hint: Wrong Strategy.get_state() parameters? Traceback (most recent call last): File "/home/painkillerrr/Documents/AAA/Trading/Gym/btgym/btgym/envs/backtrader.py", line 674, in reset assert self.observation_space.contains(self.env_response[0]) AssertionError Traceback (most recent call last): File "/home/painkillerrr/Documents/AAA/Trading/Gym/btgym/btgym/envs/backtrader.py", line 674, in reset assert self.observation_space.contains(self.env_response[0]) AssertionError

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/painkillerrr/Documents/AAA/Trading/Gym/btgym/examples/TestFromVideo.py", line 154, in main() File "/home/painkillerrr/Documents/AAA/Trading/Gym/btgym/examples/TestFromVideo.py", line 134, in main cur_state = np.array(list(env.reset().items())[0][1]) File "/home/painkillerrr/Documents/AAA/Trading/Gym/btgym/btgym/envs/backtrader.py", line 701, in reset raise AssertionError(msg) AssertionError: State observation shape/range mismatch! Space set by env:

I am running this in an Anacanda environment with python 3.6. This error seems to come from the reset() function in "backtrader.py" under btgym/envs/. This line of code: assert self.observation_space.contains(self.env_response[0])

The solution that worked for me was to comment out the above line and add a simple "pass" command.

The second issue that I mentioned previously is after the btGym env is created and I am trying to train an agent. Assuming that the btGym env is similar to Gym env, I tried to pass it to Keras-RL. Keras-RL has this nice feature that if you pass a Gym environment, you can use its RL algorithms to train on the environment.

However, it seems that the step() function in btGym does not return the same state shape that a regular Gym env returns. For instance, Gym returns a numpy array as the state while btGym returns a dictionary! I guess there are some hacks for it but it would be less headache if we could treat btGym env similar to Gym envs.

[Kismuz]

@traderpro446,

1. State observation shape/range mismatch: see recent #56 - this issue will remain open for several days, see fix inside.
2. btGym returns a dictionary - yes indeed, since the environment is more complex than basic ones there is need for nested multimodal observation space. BtGYm uses two main modalities: 'external' - holds current price information and 'internal' - holds trading account/broker information. Each modality has to be processed separately an is essential for decision making. See also usage comments in code for gym.Dict space here: https://github.com/openai/gym/blob/master/gym/spaces/dict_space.py. If Keras RL can't handle gym.spaces.Dict - the only fix you can do is to make environment wrapper to do all preprocessing stuff and pass nice single tensor. In practice it is quite inconvenient though.

[traderpro446]

@Kismuz How can we calculate and add the indicators in addition to the OHLC prices? The returned state is a pre-defined window of past prices but for the indicators we are going to need the prices beyond the window.

[Kismuz]

@traderpro446, see:

31,

35 (esp. four last posts).

Take a look at advanced strategies source code: https://github.com/Kismuz/btgym/blob/master/btgym/research/strategy_gen_4.py

Docs: https://kismuz.github.io/btgym/