How to implement a continuous action space?

[stabilus]

say I wanted to train actions to be in a range between 0 and 1, whereby the number represents the percentage of the networth I should be invested in the asset. The resulting action is then to buy or sell the difference.

[AminHP]

To have a continuous action space, we need to rewrite all the three environments. Personally, I like this idea and I am up to implement continuous_trading_env, continuous_forex_env, and continuous_stocks_env. But, I'm not sure if I have enough time for that at this moment.

I was thinking about having a range between -1 and +1 for action values. The range -1 to 0 indicates a sell signal while the range 0 to +1 indicates a buy signal. Also, the absolute value of that can represent the percentage of the networth and multiply self._total_profit for the next trade. What do you think?

[stabilus]

It would probably be simpler if you just used 0-1 and let the algo decide whether that means buying or selling. i.e. if you're invested 50%, an action of 0.3 would imply selling the position from a percentage of 50% down to 30%.

[xicocaio]

Hello @AminHP ,

First of all, great work on this project as it is proving to be very useful to me.

I also prefer the [-1, +1] range for the continuous case as it usually the most common in the research community.

On a similar note, for the discrete case, how do you recommend extending the TradingEnv class to define my action values?

Thanks in advance =)

[AminHP]

Hi @xicocaio, I'm glad it's been useful to you :)

Thanks for the information. I will consider it.

I don't exactly understand your question. Do you want to extend the TradingEnv to support more than just two actions? For example, you want to have 4 discrete actions, right?

[xicocaio]

Hi,

My issue is that I want to extend TradingEnv, but I want to use other values. Or even add other actions, as such:

class Actions(Enum):
    Sell = -1
    Hold = 0
    Buy = 1

Then, because these actions are in the base class, I found it not very practical to extend them in a subclass.

So my question is about what do you think is the best way I could deal with that.

Tks!

[AminHP]

@xicocaio Yes you are right, this is a fundamental change and affects almost all parts of the trading environments (both logic and rendering). So, I think it is better to fork and rewrite those parts according to your custom actions.

[xicocaio]

Ok, I will do that, Thank you!

[luccosta]

class Actions(Enum):
    Sell = -1
    Hold = 0
    Buy = 1

@xicocaio I was using this format, but my model just holds. Something similar happened with you?

[xicocaio]

@luccosta Well, it depends on how do you want your env and model to work.

In my case, I had to follow a specific design of a specific scientific paper, and so I had to fork this repo to make adjustments here (changes are at the feat-moody_env branch).

In summary, when adopting continuous actions, you have to take into account what @AminHP said:

I was thinking about having a range between -1 and +1 for action values. The range -1 to 0 indicates a sell signal while the range 0 to +1 indicates a buy signal. Also, the absolute value of that can represent the percentage of the networth and multiply self._total_profit for the next trade.

Also, you have to think about how these changes impact how you calculate the return (additive or multiplicative) as described in the mentioned paper.

[AminHP]

I just released a new project that supports the required feature.

https://github.com/AminHP/gym-mtsim

[kmafutah]

Thanks will be trying it out this weekend and will provide feedback. Once again thanks

Rgds KM

Sent from my iPhone

On 7 Sep 2021, at 21:50, Mohammad Amin Haghpanah @.***> wrote:

 I just released a new project that supports the required feature.

https://github.com/AminHP/gym-mtsim

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe.