Sell action when there isn't any holding assets

[ekrembk]

First of all, thank you for this project, I really believe it has a great potential and also let me tell that I'm nothing but a newbie so excuse my question if it is a dumb one.

When I configure and start running, during some epocs it keeps giving SELL actions although there is not any holding assets. 2 questions:

• Why is the reward decreases when it sells even though there isn't anything to sell
• Can we change that in a way that the action list is based on current holdings so it doesn't try to buy when there isn't any money left or sell when there isn't anything to sell

Thanks again

[Kismuz]

@ekrembk ,

Can we change that in a way that the action list is based on current holdings so it doesn't try to buy when there isn't any money left or sell when there isn't anything to sell

RL model-free setting assumes no prior knowledge or bias on action distribution; the key is to let agent figure dynamics out by itself. If it tries to perform action that is illegal in given [account] state, backtrader simulation engine will simply reject order, see #36.
Though it is possible to introduce penalty term to reward function for performing illegal actions (in our case when broker returns response 'order failed') and force agent to be 'polite an don't knock closed door several times in a row', in practice I found it is not necessary: agents quickly learns skewed action distribution(in our case 'hold' action has higher probability than others); moreover introducing such bias severely hurts exploration.

Why is the reward decreases when it sells even though there isn't anything to sell

it depends if any position is opened at the time. It also can spike at the end of the episode because all positions get forcefully closed several steps before episode end (several steps required for agent to receive final reward)

[ekrembk]

Thanks a lot for the detailed response!